[Qemu-devel] [PATCH v2 0/5] Add common QEMU control functionality to qemu-iotests

[Qemu-devel] [PATCH v2 0/5] Add common QEMU control functionality to qemu-iotests

[Jeff Cody]

Changes from v1:

    Patch 1: * Fixed commit message, clarified comments (Thanks Benoît)
             * Changed 'shift' line to be POSIX-friendly, instead of
               relying on bashism (Thanks Eric)
             * Added ability to repeat qmp or hmp commands an arbitrary
               number of times
    Patch 3: New patch, for live migration

Original Description:

This adds some common functionality to control QEMU for qemu-iotests.

Additionally, test 085 is updated to use this new functionality.

Some minor fixups along the way, to clear up spaced pathname issues, 
for common.rc, test 019, and test 086.


Jeff Cody (5):
  block: qemu-iotests - add common.qemu, for bash-controlled qemu tests
  block: qemu-iotests - update 085 to use common.qemu
  block: qemu-iotests - test for live migration
  block: qemu-iotests - fix image cleanup when using spaced pathnames
  block: qemu-iotests: make test 019 and 086 work with spaced pathnames

 tests/qemu-iotests/019         |   2 +-
 tests/qemu-iotests/085         |  73 +++------------
 tests/qemu-iotests/086         |   8 +-
 tests/qemu-iotests/089         |  97 ++++++++++++++++++++
 tests/qemu-iotests/089.out     |  20 +++++
 tests/qemu-iotests/common.qemu | 195 +++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/common.rc   |   4 +-
 tests/qemu-iotests/group       |   1 +
 8 files changed, 332 insertions(+), 68 deletions(-)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out
 create mode 100644 tests/qemu-iotests/common.qemu

-- 
1.8.3.1

[Qemu-devel] [PATCH v2 1/5] block: qemu-iotests - add common.qemu, for bash-controlled qemu tests

[Jeff Cody]

This creates some common functions for bash language qemu-iotests
to control, and communicate with, a running QEMU process.

4 functions are introduced:

    1. _launch_qemu()
        This launches the QEMU process(es), and sets up the file
        descriptors and fifos for communication.  You can choose to
        launch each QEMU process listening for either QMP or HMP
        monitor.  You can call this function multiple times, and
        save the handle returned from each.  The returned handle is
        in $QEMU_HANDLE.  You must copy this value.

Commands 2 and 3 use the handle received from _launch_qemu(), to talk
to the appropriate process.

    2. _send_qemu_cmd()
        Sends a command string, specified by $2, to QEMU.  If $2 is
        non-NULL, _send_qemu_cmd() will wait to receive $2 as a
        required result string from QEMU.  Failure to receive $3 will
        cause the test to fail.  The command can optionally be retried
        $qemu_cmd_repeat number of times.

    3. _timed_wait_for()
        Waits for a response, for up to a default of 10 seconds.  If
        $2 is not seen in that time (anywhere in the response), then
        the test fails.  Primarily used by _send_qemu_cmd, but could
        be useful standalone, as well.  To prevent automatic exit
        (and therefore test failure), set $qemu_wait_no_error to a
        non-NULL value.  If $silent is a non-NULL value, then output
        to stdout will be suppressed.

    4. _cleanup_qemu()
        Kills the running QEMU processes, and removes the fifos.

Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 tests/qemu-iotests/common.qemu | 195 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 195 insertions(+)
 create mode 100644 tests/qemu-iotests/common.qemu

diff --git a/tests/qemu-iotests/common.qemu b/tests/qemu-iotests/common.qemu
new file mode 100644
index 0000000..12c42f1
--- /dev/null
+++ b/tests/qemu-iotests/common.qemu
@@ -0,0 +1,195 @@
+#!/bin/bash
+#
+# This allows for launching of multiple QEMU instances, with independent
+# communication possible to each instance.
+#
+# Each instance can choose, at launch, to use either the QMP or the
+# HMP (monitor) interface.
+#
+# All instances are cleaned up via _cleanup_qemu, including killing the
+# running qemu instance.
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+QEMU_COMM_TIMEOUT=10
+
+QEMU_FIFO_IN="${TEST_DIR}/qmp-in-$$"
+QEMU_FIFO_OUT="${TEST_DIR}/qmp-out-$$"
+
+QEMU_PID=
+_QEMU_HANDLE=0
+QEMU_HANDLE=0
+
+# If bash version is >= 4.1, these will be overwritten and dynamic
+# file descriptor values assigned.
+_out_fd=3
+_in_fd=4
+
+# Wait for expected QMP response from QEMU.  Will time out
+# after 10 seconds, which counts as failure.
+#
+# Override QEMU_COMM_TIMEOUT for a timeout different than the
+# default 10 seconds
+#
+# $1: The handle to use
+# $2+ All remaining arguments comprise the string to search for
+#    in the response.
+#
+# If $silent is set to anything but an empty string, then
+# response is not echoed out.
+function _timed_wait_for()
+{
+    local h=${1}
+    shift
+
+    QEMU_STATUS[$h]=0
+    while read -t ${QEMU_COMM_TIMEOUT} resp <&${QEMU_OUT[$h]}
+    do
+        if [ -z "${silent}" ]; then
+            echo "${resp}" | _filter_testdir | _filter_qemu \
+                           | _filter_qemu_io | _filter_qmp
+        fi
+        grep -q "${*}" < <(echo ${resp})
+        if [ $? -eq 0 ]; then
+            return
+        fi
+    done
+    QEMU_STATUS[$h]=-1
+    if [ -z "${qemu_wait_no_error}" ]; then
+        echo "Timeout waiting for ${*} on handle ${h}"
+        exit 1  # Timeout means the test failed
+    fi
+}
+
+
+# Sends QMP or HMP command to QEMU, and waits for the expected response
+#
+# $1:       QEMU handle to use
+# $2:       String of the QMP command to send
+# ${@: -1}  (Last string passed)
+#             String that the QEMU response should contain. If it is a null
+#             string, do not wait for a response
+#
+# Set qemu_cmd_repeat to the number of times to repeat the cmd
+# until either timeout, or a response.  If it is not set, or <=0,
+# then the command is only sent once.
+#
+function _send_qemu_cmd()
+{
+    local h=${1}
+    local count=1
+    local cmd=
+    local use_error=
+    shift
+
+    if [ ${qemu_cmd_repeat} -gt 0 ] 2>/dev/null; then
+        count=${qemu_cmd_repeat}
+        use_error="no"
+    fi
+    # This array element extraction is done to accomodate pathnames with spaces
+    cmd=${@: 1:${#@}-1}
+    shift $(($# - 1))
+
+    while [ ${count} -gt 0 ]
+    do
+        echo "${cmd}" >&${QEMU_IN[${h}]}
+        if [ -n "${1}" ]; then
+            qemu_wait_no_error=${use_error} _timed_wait_for ${h} "${1}"
+            if [ ${QEMU_STATUS[$h]} -eq 0 ]; then
+                return
+            fi
+        fi
+        let count--;
+    done
+    if [ ${QEMU_STATUS[$h]} -ne 0 ]; then
+        echo "Timeout waiting for ${1} on handle ${h}"
+        exit 1 #Timeout means the test failed
+    fi
+}
+
+
+# Launch a QEMU process.
+#
+# Input parameters:
+# $qemu_comm_method: set this variable to 'monitor' (case insensitive)
+#                    to use the QEMU HMP monitor for communication.
+#                    Otherwise, the default of QMP is used.
+# Returns:
+# $QEMU_HANDLE: set to a handle value to communicate with this QEMU instance.
+#
+function _launch_qemu()
+{
+    local comm=
+    local fifo_out=
+    local fifo_in=
+
+    if (shopt -s nocasematch; [[ "${qemu_comm_method}" == "monitor" ]])
+    then
+        comm="-monitor stdio"
+    else
+        local qemu_comm_method="qmp"
+        comm="-monitor none -qmp stdio"
+    fi
+
+    fifo_out=${QEMU_FIFO_OUT}_${_QEMU_HANDLE}
+    fifo_in=${QEMU_FIFO_IN}_${_QEMU_HANDLE}
+    mkfifo "${fifo_out}"
+    mkfifo "${fifo_in}"
+
+    "${QEMU}" -nographic -serial none ${comm} "${@}" 2>&1 \
+                                                     >"${fifo_out}" \
+                                                     <"${fifo_in}" &
+    QEMU_PID[${_QEMU_HANDLE}]=$!
+
+    if [ "${BASH_VERSINFO[0]}" -ge "4" ] && [ "${BASH_VERSINFO[1]}" -ge "1" ]
+    then
+        # bash >= 4.1 required for automatic fd
+        exec {_out_fd}<"${fifo_out}"
+        exec {_in_fd}>"${fifo_in}"
+    else
+        let _out_fd++
+        let _in_fd++
+        eval "exec ${_out_fd}<'${fifo_out}'"
+        eval "exec ${_in_fd}>'${fifo_in}'"
+    fi
+
+    QEMU_OUT[${_QEMU_HANDLE}]=${_out_fd}
+    QEMU_IN[${_QEMU_HANDLE}]=${_in_fd}
+    QEMU_STATUS[${_QEMU_HANDLE}]=0
+
+    if [ "${qemu_comm_method}" == "qmp" ]
+    then
+        # Don't print response, since it has version information in it
+        silent=yes _timed_wait_for ${_QEMU_HANDLE} "capabilities"
+    fi
+    QEMU_HANDLE=${_QEMU_HANDLE}
+    let _QEMU_HANDLE++
+}
+
+
+# Silenty kills the QEMU process
+function _cleanup_qemu()
+{
+    # QEMU_PID[], QEMU_IN[], QEMU_OUT[] all use same indices
+    for i in "${!QEMU_OUT[@]}"
+    do
+        kill -KILL ${QEMU_PID[$i]}
+        wait ${QEMU_PID[$i]} 2>/dev/null # silent kill
+        rm -f "${QEMU_FIFO_IN}_${i}" "${QEMU_FIFO_OUT}_${i}"
+    done
+}
+
-- 
1.8.3.1

[Qemu-devel] [PATCH v2 2/5] block: qemu-iotests - update 085 to use common.qemu

[Jeff Cody]

The new functionality of common.qemu implements the QEMU control
and communication functionality that was originally in test 085.

This removes that now-duplicate functionality, and uses the
common.qemu functions.

Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 tests/qemu-iotests/085 | 73 +++++++++-----------------------------------------
 1 file changed, 12 insertions(+), 61 deletions(-)

diff --git a/tests/qemu-iotests/085 b/tests/qemu-iotests/085
index 33c8dc4..56cd6f8 100755
--- a/tests/qemu-iotests/085
+++ b/tests/qemu-iotests/085
@@ -30,10 +30,6 @@ echo "QA output created by $seq"
 
 here=`pwd`
 status=1	# failure is the default!
-qemu_pid=
-
-QMP_IN="${TEST_DIR}/qmp-in-$$"
-QMP_OUT="${TEST_DIR}/qmp-out-$$"
 
 snapshot_virt0="snapshot-v0.qcow2"
 snapshot_virt1="snapshot-v1.qcow2"
@@ -42,10 +38,7 @@ MAX_SNAPSHOTS=10
 
 _cleanup()
 {
-    kill -KILL ${qemu_pid}
-    wait ${qemu_pid} 2>/dev/null  # silent kill
-
-    rm -f "${QMP_IN}" "${QMP_OUT}"
+    _cleanup_qemu
     for i in $(seq 1 ${MAX_SNAPSHOTS})
     do
         rm -f "${TEST_DIR}/${i}-${snapshot_virt0}"
@@ -59,43 +52,12 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
+. ./common.qemu
 
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
 
-# Wait for expected QMP response from QEMU.  Will time out
-# after 10 seconds, which counts as failure.
-#
-# $1 is the string to expect
-#
-# If $silent is set to anything but an empty string, then
-# response is not echoed out.
-function timed_wait_for()
-{
-    while read -t 10 resp <&5
-    do
-        if [ "${silent}" == "" ]; then
-            echo "${resp}" | _filter_testdir | _filter_qemu
-        fi
-        grep -q "${1}" < <(echo ${resp})
-        if [ $? -eq 0 ]; then
-            return
-        fi
-    done
-    echo "Timeout waiting for ${1}"
-    exit 1  # Timeout means the test failed
-}
-
-# Sends QMP command to QEMU, and waits for the expected response
-#
-# ${1}:  String of the QMP command to send
-# ${2}:  String that the QEMU response should contain
-function send_qmp_cmd()
-{
-    echo "${1}" >&6
-    timed_wait_for "${2}"
-}
 
 # ${1}: unique identifier for the snapshot filename
 function create_single_snapshot()
@@ -104,7 +66,7 @@ function create_single_snapshot()
                       'arguments': { 'device': 'virtio0',
                                      'snapshot-file':'"${TEST_DIR}/${1}-${snapshot_virt0}"',
                                      'format': 'qcow2' } }"
-    send_qmp_cmd "${cmd}" "return"
+    _send_qemu_cmd $h "${cmd}" "return"
 }
 
 # ${1}: unique identifier for the snapshot filename
@@ -120,14 +82,11 @@ function create_group_snapshot()
                        'snapshot-file': '"${TEST_DIR}/${1}-${snapshot_virt1}"' } } ]
              } }"
 
-    send_qmp_cmd "${cmd}" "return"
+    _send_qemu_cmd $h "${cmd}" "return"
 }
 
 size=128M
 
-mkfifo "${QMP_IN}"
-mkfifo "${QMP_OUT}"
-
 _make_test_img $size
 mv "${TEST_IMG}" "${TEST_IMG}.orig"
 _make_test_img $size
@@ -136,23 +95,15 @@ echo
 echo === Running QEMU ===
 echo
 
-"${QEMU}" -nographic -monitor none -serial none -qmp stdio\
-          -drive file="${TEST_IMG}.orig",if=virtio\
-          -drive file="${TEST_IMG}",if=virtio 2>&1 >"${QMP_OUT}" <"${QMP_IN}"&
-qemu_pid=$!
-
-# redirect fifos to file descriptors, to keep from blocking
-exec 5<"${QMP_OUT}"
-exec 6>"${QMP_IN}"
-
-# Don't print response, since it has version information in it
-silent=yes timed_wait_for "capabilities"
+qemu_comm_method="qmp"
+_launch_qemu -drive file="${TEST_IMG}.orig",if=virtio -drive file="${TEST_IMG}",if=virtio
+h=$QEMU_HANDLE
 
 echo
 echo === Sending capabilities ===
 echo
 
-send_qmp_cmd "{ 'execute': 'qmp_capabilities' }" "return"
+_send_qemu_cmd $h "{ 'execute': 'qmp_capabilities' }" "return"
 
 echo
 echo === Create a single snapshot on virtio0 ===
@@ -165,16 +116,16 @@ echo
 echo === Invalid command - missing device and nodename ===
 echo
 
-send_qmp_cmd "{ 'execute': 'blockdev-snapshot-sync',
-                      'arguments': { 'snapshot-file':'"${TEST_DIR}"/1-${snapshot_virt0}',
+_send_qemu_cmd $h "{ 'execute': 'blockdev-snapshot-sync',
+                         'arguments': { 'snapshot-file':'"${TEST_DIR}/1-${snapshot_virt0}"',
                                      'format': 'qcow2' } }" "error"
 
 echo
 echo === Invalid command - missing snapshot-file ===
 echo
 
-send_qmp_cmd "{ 'execute': 'blockdev-snapshot-sync',
-                      'arguments': { 'device': 'virtio0',
+_send_qemu_cmd $h "{ 'execute': 'blockdev-snapshot-sync',
+                         'arguments': { 'device': 'virtio0',
                                      'format': 'qcow2' } }" "error"
 echo
 echo
-- 
1.8.3.1

[Qemu-devel] [PATCH v2 3/5] block: qemu-iotests - test for live migration

[Jeff Cody]

This is an initial, simple live migration test from one
running VM to another, using monitor commands.

This is also an example on using the new common.qemu functions
for controlling multiple running qemu instances, for tests that
need a live qemu vm.

Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 tests/qemu-iotests/089     | 97 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/089.out | 20 ++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 118 insertions(+)
 create mode 100755 tests/qemu-iotests/089
 create mode 100644 tests/qemu-iotests/089.out

diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
new file mode 100755
index 0000000..22a7cf1
--- /dev/null
+++ b/tests/qemu-iotests/089
@@ -0,0 +1,97 @@
+#!/bin/bash
+#
+# Live migration test
+#
+# Performs a migration from one VM to another via monitor commands
+#
+# Copyright (C) 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=jcody@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1	# failure is the default!
+
+MIG_FIFO="${TEST_DIR}/migrate"
+
+_cleanup()
+{
+    rm -f "${MIG_FIFO}"
+    _cleanup_qemu
+	_cleanup_test_img
+
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+size=1G
+
+IMGOPTS="cluster_size=512" _make_test_img $size
+
+mkfifo "${MIG_FIFO}"
+
+echo
+echo === Starting QEMU VM1 ===
+echo
+
+qemu_comm_method="monitor"
+_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk
+h1=$QEMU_HANDLE
+
+echo
+echo === Starting QEMU VM2 ===
+echo
+_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk \
+             -incoming "exec: cat '${MIG_FIFO}'"
+h2=$QEMU_HANDLE
+
+echo
+echo === VM 1: Migrate from VM1 to VM2  ===
+echo
+
+silent=yes
+_send_qemu_cmd $h1 'qemu-io disk "write 0 4M"' "(qemu)"
+echo "vm1: qemu-io disk write complete"
+_send_qemu_cmd $h1 "migrate \"exec: cat > '${MIG_FIFO}'\"" "(qemu)"
+echo "vm1: live migration started"
+qemu_cmd_repeat=20 _send_qemu_cmd $h1 "info migrate" "completed"
+echo "vm1: live migration completed"
+
+echo
+echo === VM 2: Post-migration, write to disk, verify running ===
+echo
+
+_send_qemu_cmd $h2 'qemu-io disk "write 4M 1M"' "(qemu)"
+echo "vm2: qemu-io disk write complete"
+qemu_cmd_repeat=20 _send_qemu_cmd $h2 "info status" "running"
+echo "vm2: qemu process running successfully"
+
+
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
new file mode 100644
index 0000000..4e9e6c9
--- /dev/null
+++ b/tests/qemu-iotests/089.out
@@ -0,0 +1,20 @@
+QA output created by 089
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
+
+=== Starting QEMU VM1 ===
+
+
+=== Starting QEMU VM2 ===
+
+
+=== VM 1: Migrate from VM1 to VM2 ===
+
+vm1: qemu-io disk write complete
+vm1: live migration started
+vm1: live migration completed
+
+=== VM 2: Post-migration, write to disk, verify running ===
+
+vm2: qemu-io disk write complete
+vm2: qemu process running successfully
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 864643d..73e6b5d 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -95,3 +95,4 @@
 086 rw auto quick
 087 rw auto
 088 rw auto
+089 rw auto
-- 
1.8.3.1

[Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Jeff Cody]

The _rm_test_img() function in common.rc did not quote the image
file, which left droppings in the scratch directory (and performed
a potentially unsafe rm -f).

This adds the necessary quotes.

Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 tests/qemu-iotests/common.rc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 7f00883..195c564 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -178,10 +178,10 @@ _rm_test_img()
     local img=$1
     if [ "$IMGFMT" = "vmdk" ]; then
         # Remove all the extents for vmdk
-        $QEMU_IMG info $img 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
+        "$QEMU_IMG" info "$img" 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
             | xargs -I {} rm -f "{}"
     fi
-    rm -f $img
+    rm -f "$img"
 }
 
 _cleanup_test_img()
-- 
1.8.3.1

[Qemu-devel] [PATCH v2 5/5] block: qemu-iotests: make test 019 and 086 work with spaced pathnames

[Jeff Cody]

Both tests 019 and 086 need proper quotations to work with pathnames
that contain spaces.

Signed-off-by: Jeff Cody <jcody@redhat.com>
---
 tests/qemu-iotests/019 | 2 +-
 tests/qemu-iotests/086 | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index e67445c..f5ecbf5 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -96,7 +96,7 @@ mv "$TEST_IMG" "$TEST_IMG.orig"
 for backing_option in "-B " "-o backing_file="; do
 
     echo
-    echo Testing conversion with $backing_option$TEST_IMG.base | _filter_testdir | _filter_imgfmt
+    echo Testing conversion with $backing_option"$TEST_IMG.base" | _filter_testdir | _filter_imgfmt
     echo
     $QEMU_IMG convert -O $IMGFMT $backing_option"$TEST_IMG.base" "$TEST_IMG.orig" "$TEST_IMG"
 
diff --git a/tests/qemu-iotests/086 b/tests/qemu-iotests/086
index 48fe85b..d9a80cf 100755
--- a/tests/qemu-iotests/086
+++ b/tests/qemu-iotests/086
@@ -51,10 +51,10 @@ function run_qemu_img()
 size=128M
 
 _make_test_img $size
-$QEMU_IO -c 'write 0 1M' $TEST_IMG | _filter_qemu_io
-$QEMU_IO -c 'write 2M 1M' $TEST_IMG | _filter_qemu_io
-$QEMU_IO -c 'write 4M 1M' $TEST_IMG | _filter_qemu_io
-$QEMU_IO -c 'write 32M 1M' $TEST_IMG | _filter_qemu_io
+$QEMU_IO -c 'write 0 1M' "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c 'write 2M 1M' "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c 'write 4M 1M' "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c 'write 32M 1M' "$TEST_IMG" | _filter_qemu_io
 
 $QEMU_IMG convert -p -O $IMGFMT -f $IMGFMT "$TEST_IMG" "$TEST_IMG".base  2>&1 |\
     _filter_testdir | sed -e 's/\r/\n/g'
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH v2 1/5] block: qemu-iotests - add common.qemu, for bash-controlled qemu tests

[Fam Zheng]

On Wed, 04/09 22:41, Jeff Cody wrote:
> This creates some common functions for bash language qemu-iotests
> to control, and communicate with, a running QEMU process.
> 
> 4 functions are introduced:
> 
>     1. _launch_qemu()
>         This launches the QEMU process(es), and sets up the file
>         descriptors and fifos for communication.  You can choose to
>         launch each QEMU process listening for either QMP or HMP
>         monitor.  You can call this function multiple times, and
>         save the handle returned from each.  The returned handle is
>         in $QEMU_HANDLE.  You must copy this value.
> 
> Commands 2 and 3 use the handle received from _launch_qemu(), to talk
> to the appropriate process.
> 
>     2. _send_qemu_cmd()
>         Sends a command string, specified by $2, to QEMU.  If $2 is
>         non-NULL, _send_qemu_cmd() will wait to receive $2 as a

Do you mean $3 in this sentence?

>         required result string from QEMU.  Failure to receive $3 will
>         cause the test to fail.  The command can optionally be retried
>         $qemu_cmd_repeat number of times.
> 
>     3. _timed_wait_for()
>         Waits for a response, for up to a default of 10 seconds.  If
>         $2 is not seen in that time (anywhere in the response), then
>         the test fails.  Primarily used by _send_qemu_cmd, but could
>         be useful standalone, as well.  To prevent automatic exit
>         (and therefore test failure), set $qemu_wait_no_error to a
>         non-NULL value.  If $silent is a non-NULL value, then output
>         to stdout will be suppressed.
> 
>     4. _cleanup_qemu()
>         Kills the running QEMU processes, and removes the fifos.
> 
> Signed-off-by: Jeff Cody <jcody@redhat.com>
> ---
>  tests/qemu-iotests/common.qemu | 195 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 195 insertions(+)
>  create mode 100644 tests/qemu-iotests/common.qemu
> 
> diff --git a/tests/qemu-iotests/common.qemu b/tests/qemu-iotests/common.qemu
> new file mode 100644
> index 0000000..12c42f1
> --- /dev/null
> +++ b/tests/qemu-iotests/common.qemu
> @@ -0,0 +1,195 @@
> +#!/bin/bash
> +#
> +# This allows for launching of multiple QEMU instances, with independent
> +# communication possible to each instance.
> +#
> +# Each instance can choose, at launch, to use either the QMP or the
> +# HMP (monitor) interface.
> +#
> +# All instances are cleaned up via _cleanup_qemu, including killing the
> +# running qemu instance.
> +#
> +# Copyright (C) 2014 Red Hat, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +QEMU_COMM_TIMEOUT=10
> +
> +QEMU_FIFO_IN="${TEST_DIR}/qmp-in-$$"
> +QEMU_FIFO_OUT="${TEST_DIR}/qmp-out-$$"
> +
> +QEMU_PID=
> +_QEMU_HANDLE=0
> +QEMU_HANDLE=0
> +
> +# If bash version is >= 4.1, these will be overwritten and dynamic
> +# file descriptor values assigned.
> +_out_fd=3
> +_in_fd=4
> +
> +# Wait for expected QMP response from QEMU.  Will time out
> +# after 10 seconds, which counts as failure.
> +#
> +# Override QEMU_COMM_TIMEOUT for a timeout different than the
> +# default 10 seconds
> +#
> +# $1: The handle to use
> +# $2+ All remaining arguments comprise the string to search for
> +#    in the response.
> +#
> +# If $silent is set to anything but an empty string, then
> +# response is not echoed out.
> +function _timed_wait_for()
> +{
> +    local h=${1}
> +    shift
> +
> +    QEMU_STATUS[$h]=0
> +    while read -t ${QEMU_COMM_TIMEOUT} resp <&${QEMU_OUT[$h]}
> +    do
> +        if [ -z "${silent}" ]; then
> +            echo "${resp}" | _filter_testdir | _filter_qemu \
> +                           | _filter_qemu_io | _filter_qmp
> +        fi
> +        grep -q "${*}" < <(echo ${resp})
> +        if [ $? -eq 0 ]; then
> +            return
> +        fi
> +    done
> +    QEMU_STATUS[$h]=-1
> +    if [ -z "${qemu_wait_no_error}" ]; then
> +        echo "Timeout waiting for ${*} on handle ${h}"
> +        exit 1  # Timeout means the test failed
> +    fi
> +}
> +
> +
> +# Sends QMP or HMP command to QEMU, and waits for the expected response
> +#
> +# $1:       QEMU handle to use
> +# $2:       String of the QMP command to send
> +# ${@: -1}  (Last string passed)
> +#             String that the QEMU response should contain. If it is a null
> +#             string, do not wait for a response
> +#
> +# Set qemu_cmd_repeat to the number of times to repeat the cmd
> +# until either timeout, or a response.  If it is not set, or <=0,
> +# then the command is only sent once.
> +#
> +function _send_qemu_cmd()
> +{
> +    local h=${1}
> +    local count=1
> +    local cmd=
> +    local use_error=
> +    shift
> +
> +    if [ ${qemu_cmd_repeat} -gt 0 ] 2>/dev/null; then
> +        count=${qemu_cmd_repeat}
> +        use_error="no"
> +    fi
> +    # This array element extraction is done to accomodate pathnames with spaces
> +    cmd=${@: 1:${#@}-1}
> +    shift $(($# - 1))
> +
> +    while [ ${count} -gt 0 ]
> +    do
> +        echo "${cmd}" >&${QEMU_IN[${h}]}
> +        if [ -n "${1}" ]; then
> +            qemu_wait_no_error=${use_error} _timed_wait_for ${h} "${1}"
> +            if [ ${QEMU_STATUS[$h]} -eq 0 ]; then
> +                return
> +            fi
> +        fi
> +        let count--;
> +    done
> +    if [ ${QEMU_STATUS[$h]} -ne 0 ]; then
> +        echo "Timeout waiting for ${1} on handle ${h}"
> +        exit 1 #Timeout means the test failed
> +    fi
> +}
> +
> +
> +# Launch a QEMU process.
> +#
> +# Input parameters:
> +# $qemu_comm_method: set this variable to 'monitor' (case insensitive)
> +#                    to use the QEMU HMP monitor for communication.
> +#                    Otherwise, the default of QMP is used.
> +# Returns:
> +# $QEMU_HANDLE: set to a handle value to communicate with this QEMU instance.
> +#
> +function _launch_qemu()
> +{
> +    local comm=
> +    local fifo_out=
> +    local fifo_in=
> +
> +    if (shopt -s nocasematch; [[ "${qemu_comm_method}" == "monitor" ]])
> +    then
> +        comm="-monitor stdio"
> +    else
> +        local qemu_comm_method="qmp"
> +        comm="-monitor none -qmp stdio"
> +    fi
> +
> +    fifo_out=${QEMU_FIFO_OUT}_${_QEMU_HANDLE}
> +    fifo_in=${QEMU_FIFO_IN}_${_QEMU_HANDLE}
> +    mkfifo "${fifo_out}"
> +    mkfifo "${fifo_in}"
> +
> +    "${QEMU}" -nographic -serial none ${comm} "${@}" 2>&1 \
> +                                                     >"${fifo_out}" \
> +                                                     <"${fifo_in}" &

Maybe add "-machine accel=qtest"?

Thanks,
Fam

> +    QEMU_PID[${_QEMU_HANDLE}]=$!
> +
> +    if [ "${BASH_VERSINFO[0]}" -ge "4" ] && [ "${BASH_VERSINFO[1]}" -ge "1" ]
> +    then
> +        # bash >= 4.1 required for automatic fd
> +        exec {_out_fd}<"${fifo_out}"
> +        exec {_in_fd}>"${fifo_in}"
> +    else
> +        let _out_fd++
> +        let _in_fd++
> +        eval "exec ${_out_fd}<'${fifo_out}'"
> +        eval "exec ${_in_fd}>'${fifo_in}'"
> +    fi
> +
> +    QEMU_OUT[${_QEMU_HANDLE}]=${_out_fd}
> +    QEMU_IN[${_QEMU_HANDLE}]=${_in_fd}
> +    QEMU_STATUS[${_QEMU_HANDLE}]=0
> +
> +    if [ "${qemu_comm_method}" == "qmp" ]
> +    then
> +        # Don't print response, since it has version information in it
> +        silent=yes _timed_wait_for ${_QEMU_HANDLE} "capabilities"
> +    fi
> +    QEMU_HANDLE=${_QEMU_HANDLE}
> +    let _QEMU_HANDLE++
> +}
> +
> +
> +# Silenty kills the QEMU process
> +function _cleanup_qemu()
> +{
> +    # QEMU_PID[], QEMU_IN[], QEMU_OUT[] all use same indices
> +    for i in "${!QEMU_OUT[@]}"
> +    do
> +        kill -KILL ${QEMU_PID[$i]}
> +        wait ${QEMU_PID[$i]} 2>/dev/null # silent kill
> +        rm -f "${QEMU_FIFO_IN}_${i}" "${QEMU_FIFO_OUT}_${i}"
> +    done
> +}
> +
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH v2 2/5] block: qemu-iotests - update 085 to use common.qemu

[Fam Zheng]

On Wed, 04/09 22:41, Jeff Cody wrote:
> The new functionality of common.qemu implements the QEMU control
> and communication functionality that was originally in test 085.
> 
> This removes that now-duplicate functionality, and uses the
> common.qemu functions.
> 

Just a note.

A quick grep shows 067, 071, 081 and 087 are also bash cases with QEMU process.
Not necessarily in this series but they are also candidates too convert.

Thanks,
Fam

Re: [Qemu-devel] [PATCH v2 3/5] block: qemu-iotests - test for live migration

[Fam Zheng]

On Wed, 04/09 22:41, Jeff Cody wrote:
> This is an initial, simple live migration test from one
> running VM to another, using monitor commands.
> 
> This is also an example on using the new common.qemu functions
> for controlling multiple running qemu instances, for tests that
> need a live qemu vm.
> 
> Signed-off-by: Jeff Cody <jcody@redhat.com>
> ---
>  tests/qemu-iotests/089     | 97 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/089.out | 20 ++++++++++

I used 089 in my last image fleecing series, (originally 083 but already used).
So one of us need to shift the case number.

Thanks,
Fam

>  tests/qemu-iotests/group   |  1 +
>  3 files changed, 118 insertions(+)
>  create mode 100755 tests/qemu-iotests/089
>  create mode 100644 tests/qemu-iotests/089.out
> 
> diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
> new file mode 100755
> index 0000000..22a7cf1
> --- /dev/null
> +++ b/tests/qemu-iotests/089
> @@ -0,0 +1,97 @@
> +#!/bin/bash
> +#
> +# Live migration test
> +#
> +# Performs a migration from one VM to another via monitor commands
> +#
> +# Copyright (C) 2014 Red Hat, Inc.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +# creator
> +owner=jcody@redhat.com
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +status=1	# failure is the default!
> +
> +MIG_FIFO="${TEST_DIR}/migrate"
> +
> +_cleanup()
> +{
> +    rm -f "${MIG_FIFO}"
> +    _cleanup_qemu
> +	_cleanup_test_img
> +
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.qemu
> +
> +_supported_fmt qcow2
> +_supported_proto file
> +_supported_os Linux
> +
> +size=1G
> +
> +IMGOPTS="cluster_size=512" _make_test_img $size
> +
> +mkfifo "${MIG_FIFO}"
> +
> +echo
> +echo === Starting QEMU VM1 ===
> +echo
> +
> +qemu_comm_method="monitor"
> +_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk
> +h1=$QEMU_HANDLE
> +
> +echo
> +echo === Starting QEMU VM2 ===
> +echo
> +_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk \
> +             -incoming "exec: cat '${MIG_FIFO}'"
> +h2=$QEMU_HANDLE
> +
> +echo
> +echo === VM 1: Migrate from VM1 to VM2  ===
> +echo
> +
> +silent=yes
> +_send_qemu_cmd $h1 'qemu-io disk "write 0 4M"' "(qemu)"
> +echo "vm1: qemu-io disk write complete"
> +_send_qemu_cmd $h1 "migrate \"exec: cat > '${MIG_FIFO}'\"" "(qemu)"
> +echo "vm1: live migration started"
> +qemu_cmd_repeat=20 _send_qemu_cmd $h1 "info migrate" "completed"
> +echo "vm1: live migration completed"
> +
> +echo
> +echo === VM 2: Post-migration, write to disk, verify running ===
> +echo
> +
> +_send_qemu_cmd $h2 'qemu-io disk "write 4M 1M"' "(qemu)"
> +echo "vm2: qemu-io disk write complete"
> +qemu_cmd_repeat=20 _send_qemu_cmd $h2 "info status" "running"
> +echo "vm2: qemu process running successfully"
> +
> +
> +echo "*** done"
> +rm -f $seq.full
> +status=0
> diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
> new file mode 100644
> index 0000000..4e9e6c9
> --- /dev/null
> +++ b/tests/qemu-iotests/089.out
> @@ -0,0 +1,20 @@
> +QA output created by 089
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
> +
> +=== Starting QEMU VM1 ===
> +
> +
> +=== Starting QEMU VM2 ===
> +
> +
> +=== VM 1: Migrate from VM1 to VM2 ===
> +
> +vm1: qemu-io disk write complete
> +vm1: live migration started
> +vm1: live migration completed
> +
> +=== VM 2: Post-migration, write to disk, verify running ===
> +
> +vm2: qemu-io disk write complete
> +vm2: qemu process running successfully
> +*** done
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 864643d..73e6b5d 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -95,3 +95,4 @@
>  086 rw auto quick
>  087 rw auto
>  088 rw auto
> +089 rw auto
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH v2 5/5] block: qemu-iotests: make test 019 and 086 work with spaced pathnames

[Fam Zheng]

On Wed, 04/09 22:41, Jeff Cody wrote:
> Both tests 019 and 086 need proper quotations to work with pathnames
> that contain spaces.
> 
> Signed-off-by: Jeff Cody <jcody@redhat.com>
> ---
>  tests/qemu-iotests/019 | 2 +-
>  tests/qemu-iotests/086 | 8 ++++----
>  2 files changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
> index e67445c..f5ecbf5 100755
> --- a/tests/qemu-iotests/019
> +++ b/tests/qemu-iotests/019
> @@ -96,7 +96,7 @@ mv "$TEST_IMG" "$TEST_IMG.orig"
>  for backing_option in "-B " "-o backing_file="; do
>  
>      echo
> -    echo Testing conversion with $backing_option$TEST_IMG.base | _filter_testdir | _filter_imgfmt
> +    echo Testing conversion with $backing_option"$TEST_IMG.base" | _filter_testdir | _filter_imgfmt
>      echo
>      $QEMU_IMG convert -O $IMGFMT $backing_option"$TEST_IMG.base" "$TEST_IMG.orig" "$TEST_IMG"
>  
> diff --git a/tests/qemu-iotests/086 b/tests/qemu-iotests/086
> index 48fe85b..d9a80cf 100755
> --- a/tests/qemu-iotests/086
> +++ b/tests/qemu-iotests/086
> @@ -51,10 +51,10 @@ function run_qemu_img()
>  size=128M
>  
>  _make_test_img $size
> -$QEMU_IO -c 'write 0 1M' $TEST_IMG | _filter_qemu_io
> -$QEMU_IO -c 'write 2M 1M' $TEST_IMG | _filter_qemu_io
> -$QEMU_IO -c 'write 4M 1M' $TEST_IMG | _filter_qemu_io
> -$QEMU_IO -c 'write 32M 1M' $TEST_IMG | _filter_qemu_io
> +$QEMU_IO -c 'write 0 1M' "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c 'write 2M 1M' "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c 'write 4M 1M' "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c 'write 32M 1M' "$TEST_IMG" | _filter_qemu_io
>  
>  $QEMU_IMG convert -p -O $IMGFMT -f $IMGFMT "$TEST_IMG" "$TEST_IMG".base  2>&1 |\
>      _filter_testdir | sed -e 's/\r/\n/g'
> -- 
> 1.8.3.1
> 
> 

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Fam Zheng]

On Wed, 04/09 22:41, Jeff Cody wrote:
> The _rm_test_img() function in common.rc did not quote the image
> file, which left droppings in the scratch directory (and performed
> a potentially unsafe rm -f).
> 
> This adds the necessary quotes.
> 
> Signed-off-by: Jeff Cody <jcody@redhat.com>
> ---
>  tests/qemu-iotests/common.rc | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
> index 7f00883..195c564 100644
> --- a/tests/qemu-iotests/common.rc
> +++ b/tests/qemu-iotests/common.rc
> @@ -178,10 +178,10 @@ _rm_test_img()
>      local img=$1

Since we are quoting $img, should we quote $1 as well?

Fam

>      if [ "$IMGFMT" = "vmdk" ]; then
>          # Remove all the extents for vmdk
> -        $QEMU_IMG info $img 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
> +        "$QEMU_IMG" info "$img" 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
>              | xargs -I {} rm -f "{}"
>      fi
> -    rm -f $img
> +    rm -f "$img"
>  }
>  
>  _cleanup_test_img()
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH v2 3/5] block: qemu-iotests - test for live migration

[Jeff Cody]

On Thu, Apr 10, 2014 at 02:16:46PM +0800, Fam Zheng wrote:
> On Wed, 04/09 22:41, Jeff Cody wrote:
> > This is an initial, simple live migration test from one
> > running VM to another, using monitor commands.
> > 
> > This is also an example on using the new common.qemu functions
> > for controlling multiple running qemu instances, for tests that
> > need a live qemu vm.
> > 
> > Signed-off-by: Jeff Cody <jcody@redhat.com>
> > ---
> >  tests/qemu-iotests/089     | 97 ++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/qemu-iotests/089.out | 20 ++++++++++
> 
> I used 089 in my last image fleecing series, (originally 083 but already used).
> So one of us need to shift the case number.
>

I'll bump mine to 090, your series is more complicated and has been
around longer.

> 
> >  tests/qemu-iotests/group   |  1 +
> >  3 files changed, 118 insertions(+)
> >  create mode 100755 tests/qemu-iotests/089
> >  create mode 100644 tests/qemu-iotests/089.out
> > 
> > diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
> > new file mode 100755
> > index 0000000..22a7cf1
> > --- /dev/null
> > +++ b/tests/qemu-iotests/089
> > @@ -0,0 +1,97 @@
> > +#!/bin/bash
> > +#
> > +# Live migration test
> > +#
> > +# Performs a migration from one VM to another via monitor commands
> > +#
> > +# Copyright (C) 2014 Red Hat, Inc.
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > +#
> > +
> > +# creator
> > +owner=jcody@redhat.com
> > +
> > +seq=`basename $0`
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +status=1	# failure is the default!
> > +
> > +MIG_FIFO="${TEST_DIR}/migrate"
> > +
> > +_cleanup()
> > +{
> > +    rm -f "${MIG_FIFO}"
> > +    _cleanup_qemu
> > +	_cleanup_test_img
> > +
> > +}
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common.rc
> > +. ./common.filter
> > +. ./common.qemu
> > +
> > +_supported_fmt qcow2
> > +_supported_proto file
> > +_supported_os Linux
> > +
> > +size=1G
> > +
> > +IMGOPTS="cluster_size=512" _make_test_img $size
> > +
> > +mkfifo "${MIG_FIFO}"
> > +
> > +echo
> > +echo === Starting QEMU VM1 ===
> > +echo
> > +
> > +qemu_comm_method="monitor"
> > +_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk
> > +h1=$QEMU_HANDLE
> > +
> > +echo
> > +echo === Starting QEMU VM2 ===
> > +echo
> > +_launch_qemu -drive file="${TEST_IMG}",cache=none,id=disk \
> > +             -incoming "exec: cat '${MIG_FIFO}'"
> > +h2=$QEMU_HANDLE
> > +
> > +echo
> > +echo === VM 1: Migrate from VM1 to VM2  ===
> > +echo
> > +
> > +silent=yes
> > +_send_qemu_cmd $h1 'qemu-io disk "write 0 4M"' "(qemu)"
> > +echo "vm1: qemu-io disk write complete"
> > +_send_qemu_cmd $h1 "migrate \"exec: cat > '${MIG_FIFO}'\"" "(qemu)"
> > +echo "vm1: live migration started"
> > +qemu_cmd_repeat=20 _send_qemu_cmd $h1 "info migrate" "completed"
> > +echo "vm1: live migration completed"
> > +
> > +echo
> > +echo === VM 2: Post-migration, write to disk, verify running ===
> > +echo
> > +
> > +_send_qemu_cmd $h2 'qemu-io disk "write 4M 1M"' "(qemu)"
> > +echo "vm2: qemu-io disk write complete"
> > +qemu_cmd_repeat=20 _send_qemu_cmd $h2 "info status" "running"
> > +echo "vm2: qemu process running successfully"
> > +
> > +
> > +echo "*** done"
> > +rm -f $seq.full
> > +status=0
> > diff --git a/tests/qemu-iotests/089.out b/tests/qemu-iotests/089.out
> > new file mode 100644
> > index 0000000..4e9e6c9
> > --- /dev/null
> > +++ b/tests/qemu-iotests/089.out
> > @@ -0,0 +1,20 @@
> > +QA output created by 089
> > +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
> > +
> > +=== Starting QEMU VM1 ===
> > +
> > +
> > +=== Starting QEMU VM2 ===
> > +
> > +
> > +=== VM 1: Migrate from VM1 to VM2 ===
> > +
> > +vm1: qemu-io disk write complete
> > +vm1: live migration started
> > +vm1: live migration completed
> > +
> > +=== VM 2: Post-migration, write to disk, verify running ===
> > +
> > +vm2: qemu-io disk write complete
> > +vm2: qemu process running successfully
> > +*** done
> > diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> > index 864643d..73e6b5d 100644
> > --- a/tests/qemu-iotests/group
> > +++ b/tests/qemu-iotests/group
> > @@ -95,3 +95,4 @@
> >  086 rw auto quick
> >  087 rw auto
> >  088 rw auto
> > +089 rw auto
> > -- 
> > 1.8.3.1
> > 
> > 

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Jeff Cody]

On Thu, Apr 10, 2014 at 03:53:57PM +0800, Fam Zheng wrote:
> On Wed, 04/09 22:41, Jeff Cody wrote:
> > The _rm_test_img() function in common.rc did not quote the image
> > file, which left droppings in the scratch directory (and performed
> > a potentially unsafe rm -f).
> > 
> > This adds the necessary quotes.
> > 
> > Signed-off-by: Jeff Cody <jcody@redhat.com>
> > ---
> >  tests/qemu-iotests/common.rc | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
> > index 7f00883..195c564 100644
> > --- a/tests/qemu-iotests/common.rc
> > +++ b/tests/qemu-iotests/common.rc
> > @@ -178,10 +178,10 @@ _rm_test_img()
> >      local img=$1
> 
> Since we are quoting $img, should we quote $1 as well?
>

I believe not, because variable assignment won't undergo all the shell
expansions.  Notably, in variable assignment word splitting is not
performed on the parameter expansion on the argument immediately to the
right of the '='.  Quote removal, however, will still be performed.  So
img=$1 and img="$1" are identical once processed.

> 
> >      if [ "$IMGFMT" = "vmdk" ]; then
> >          # Remove all the extents for vmdk
> > -        $QEMU_IMG info $img 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
> > +        "$QEMU_IMG" info "$img" 2>/dev/null | grep 'filename:' | cut -f 2 -d: \
> >              | xargs -I {} rm -f "{}"
> >      fi
> > -    rm -f $img
> > +    rm -f "$img"
> >  }
> >  
> >  _cleanup_test_img()
> > -- 
> > 1.8.3.1
> > 
> > 

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

On 04/10/2014 06:53 AM, Jeff Cody wrote:

>>> +++ b/tests/qemu-iotests/common.rc
>>> @@ -178,10 +178,10 @@ _rm_test_img()
>>>      local img=$1
>>
>> Since we are quoting $img, should we quote $1 as well?
>>
> 
> I believe not, because variable assignment won't undergo all the shell
> expansions.  Notably, in variable assignment word splitting is not
> performed on the parameter expansion on the argument immediately to the
> right of the '='.  Quote removal, however, will still be performed.  So
> img=$1 and img="$1" are identical once processed.

Ooh, tricky.

You are correct that in isolation:

img=$1
img="$1"

are semantically identical, no matter what $1 contains, across ALL
shells.  However, that's not the code you wrote above.

local img=$1

is not POSIX (yet - although there has been some effort in the POSIX
working group to standardize some form of local variables while still
allowing for the fact that bash and ksh implemented scoping of local
variables differently).  But 'local' is similar to 'export'; and observe
the difference when using 'export' between dash and bash:

$ dash -c 'set "a  b"; export a=$1 b="$1"; echo "$a.$b"'
a.a  b
$ bash -c 'set "a  b"; export a=$1 b="$1"; echo "$a.$b"'
a  b.a  b

Here, the shell word a=$1 is semantically NOT a raw assignment, but
rather an argument to 'export', and arguments DO undergo word splitting
in the current wording of POSIX.  There was a recent bug report stating
that the dash behavior (which is strictly POSIX) is undesirable, and
that the bash/ksh behavior of export, while not strictly compliant with
the POSIX 2008 wording, is nicer; so the next version of POSIX will be
amended to add a definition of a 'declaration utility' which can
evaluate (some) arguments in assignment context.  'export' is one such
declaration utility, 'local' (if it gets standardized) would be another:

http://austingroupbugs.net/view.php?id=351

But even with the notion of an assignment-context argument added to a
future version of POSIX, the reality is that given the present standard,
it's safer to either use "" to ensure no word splitting:

local img="$1"

or to rewrite things across two statements to avoid relying on whether
assignment-context arguments work the way you want:

local img
img=$1

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

On 04/10/2014 08:43 AM, Eric Blake wrote:
> On 04/10/2014 06:53 AM, Jeff Cody wrote:
> 
>>>> +++ b/tests/qemu-iotests/common.rc
>>>> @@ -178,10 +178,10 @@ _rm_test_img()
>>>>      local img=$1
>>>
>>> Since we are quoting $img, should we quote $1 as well?
>>>

> 
> http://austingroupbugs.net/view.php?id=351
> 
> But even with the notion of an assignment-context argument added to a
> future version of POSIX, the reality is that given the present standard,
> it's safer to either use "" to ensure no word splitting:

Well, if you were trying to be portable to multiple shells, then it
would matter.  But as this script is explicitly being run under
/bin/bash, and as bash already has support for declaration utilities
where local is one such utility, your script as written is safe without
"" in the arguments to local.  So I'm fine whether you choose to change
it in a respin or to leave it as written in this version.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Jeff Cody]

On Thu, Apr 10, 2014 at 08:48:10AM -0600, Eric Blake wrote:
> On 04/10/2014 08:43 AM, Eric Blake wrote:
> > On 04/10/2014 06:53 AM, Jeff Cody wrote:
> > 
> >>>> +++ b/tests/qemu-iotests/common.rc
> >>>> @@ -178,10 +178,10 @@ _rm_test_img()
> >>>>      local img=$1
> >>>
> >>> Since we are quoting $img, should we quote $1 as well?
> >>>
> 
> > 
> > http://austingroupbugs.net/view.php?id=351
> > 
> > But even with the notion of an assignment-context argument added to a
> > future version of POSIX, the reality is that given the present standard,
> > it's safer to either use "" to ensure no word splitting:
> 
> Well, if you were trying to be portable to multiple shells, then it
> would matter.  But as this script is explicitly being run under
> /bin/bash, and as bash already has support for declaration utilities
> where local is one such utility, your script as written is safe without
> "" in the arguments to local.  So I'm fine whether you choose to change
> it in a respin or to leave it as written in this version.
>

Hi Eric,

Thanks - I consulted specifically with just the bash documentation, so
you are right, this script (and likely most of qemu-iotests) is
bash-only.

That particular line is context as well, and not an actual change - so
while it may be a good idea to quote it to make the scripts closer to
posix-only, my guess is there are quite a few similar lines throughout
all the qemu-iotests scripts.

Given that, if we address that it would probably make sense to do that
in a bash->posix conversion series for all the scripts (likely a low
priority, however).

Jeff

Re: [Qemu-devel] [PATCH v2 1/5] block: qemu-iotests - add common.qemu, for bash-controlled qemu tests

[Jeff Cody]

On Thu, Apr 10, 2014 at 01:27:37PM +0800, Fam Zheng wrote:
> On Wed, 04/09 22:41, Jeff Cody wrote:
> > This creates some common functions for bash language qemu-iotests
> > to control, and communicate with, a running QEMU process.
> > 
> > 4 functions are introduced:
> > 
> >     1. _launch_qemu()
> >         This launches the QEMU process(es), and sets up the file
> >         descriptors and fifos for communication.  You can choose to
> >         launch each QEMU process listening for either QMP or HMP
> >         monitor.  You can call this function multiple times, and
> >         save the handle returned from each.  The returned handle is
> >         in $QEMU_HANDLE.  You must copy this value.
> > 
> > Commands 2 and 3 use the handle received from _launch_qemu(), to talk
> > to the appropriate process.
> > 
> >     2. _send_qemu_cmd()
> >         Sends a command string, specified by $2, to QEMU.  If $2 is
> >         non-NULL, _send_qemu_cmd() will wait to receive $2 as a
> 
> Do you mean $3 in this sentence?
>

Oops, yes - thanks.

> >         required result string from QEMU.  Failure to receive $3 will
> >         cause the test to fail.  The command can optionally be retried
> >         $qemu_cmd_repeat number of times.
> > 
> >     3. _timed_wait_for()
> >         Waits for a response, for up to a default of 10 seconds.  If
> >         $2 is not seen in that time (anywhere in the response), then
> >         the test fails.  Primarily used by _send_qemu_cmd, but could
> >         be useful standalone, as well.  To prevent automatic exit
> >         (and therefore test failure), set $qemu_wait_no_error to a
> >         non-NULL value.  If $silent is a non-NULL value, then output
> >         to stdout will be suppressed.
> > 
> >     4. _cleanup_qemu()
> >         Kills the running QEMU processes, and removes the fifos.
> > 
> > Signed-off-by: Jeff Cody <jcody@redhat.com>
> > ---
> >  tests/qemu-iotests/common.qemu | 195 +++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 195 insertions(+)
> >  create mode 100644 tests/qemu-iotests/common.qemu
> > 
> > diff --git a/tests/qemu-iotests/common.qemu b/tests/qemu-iotests/common.qemu
> > new file mode 100644
> > index 0000000..12c42f1
> > --- /dev/null
> > +++ b/tests/qemu-iotests/common.qemu
> > @@ -0,0 +1,195 @@
> > +#!/bin/bash
> > +#
> > +# This allows for launching of multiple QEMU instances, with independent
> > +# communication possible to each instance.
> > +#
> > +# Each instance can choose, at launch, to use either the QMP or the
> > +# HMP (monitor) interface.
> > +#
> > +# All instances are cleaned up via _cleanup_qemu, including killing the
> > +# running qemu instance.
> > +#
> > +# Copyright (C) 2014 Red Hat, Inc.
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> > +#
> > +
> > +QEMU_COMM_TIMEOUT=10
> > +
> > +QEMU_FIFO_IN="${TEST_DIR}/qmp-in-$$"
> > +QEMU_FIFO_OUT="${TEST_DIR}/qmp-out-$$"
> > +
> > +QEMU_PID=
> > +_QEMU_HANDLE=0
> > +QEMU_HANDLE=0
> > +
> > +# If bash version is >= 4.1, these will be overwritten and dynamic
> > +# file descriptor values assigned.
> > +_out_fd=3
> > +_in_fd=4
> > +
> > +# Wait for expected QMP response from QEMU.  Will time out
> > +# after 10 seconds, which counts as failure.
> > +#
> > +# Override QEMU_COMM_TIMEOUT for a timeout different than the
> > +# default 10 seconds
> > +#
> > +# $1: The handle to use
> > +# $2+ All remaining arguments comprise the string to search for
> > +#    in the response.
> > +#
> > +# If $silent is set to anything but an empty string, then
> > +# response is not echoed out.
> > +function _timed_wait_for()
> > +{
> > +    local h=${1}
> > +    shift
> > +
> > +    QEMU_STATUS[$h]=0
> > +    while read -t ${QEMU_COMM_TIMEOUT} resp <&${QEMU_OUT[$h]}
> > +    do
> > +        if [ -z "${silent}" ]; then
> > +            echo "${resp}" | _filter_testdir | _filter_qemu \
> > +                           | _filter_qemu_io | _filter_qmp
> > +        fi
> > +        grep -q "${*}" < <(echo ${resp})
> > +        if [ $? -eq 0 ]; then
> > +            return
> > +        fi
> > +    done
> > +    QEMU_STATUS[$h]=-1
> > +    if [ -z "${qemu_wait_no_error}" ]; then
> > +        echo "Timeout waiting for ${*} on handle ${h}"
> > +        exit 1  # Timeout means the test failed
> > +    fi
> > +}
> > +
> > +
> > +# Sends QMP or HMP command to QEMU, and waits for the expected response
> > +#
> > +# $1:       QEMU handle to use
> > +# $2:       String of the QMP command to send
> > +# ${@: -1}  (Last string passed)
> > +#             String that the QEMU response should contain. If it is a null
> > +#             string, do not wait for a response
> > +#
> > +# Set qemu_cmd_repeat to the number of times to repeat the cmd
> > +# until either timeout, or a response.  If it is not set, or <=0,
> > +# then the command is only sent once.
> > +#
> > +function _send_qemu_cmd()
> > +{
> > +    local h=${1}
> > +    local count=1
> > +    local cmd=
> > +    local use_error=
> > +    shift
> > +
> > +    if [ ${qemu_cmd_repeat} -gt 0 ] 2>/dev/null; then
> > +        count=${qemu_cmd_repeat}
> > +        use_error="no"
> > +    fi
> > +    # This array element extraction is done to accomodate pathnames with spaces
> > +    cmd=${@: 1:${#@}-1}
> > +    shift $(($# - 1))
> > +
> > +    while [ ${count} -gt 0 ]
> > +    do
> > +        echo "${cmd}" >&${QEMU_IN[${h}]}
> > +        if [ -n "${1}" ]; then
> > +            qemu_wait_no_error=${use_error} _timed_wait_for ${h} "${1}"
> > +            if [ ${QEMU_STATUS[$h]} -eq 0 ]; then
> > +                return
> > +            fi
> > +        fi
> > +        let count--;
> > +    done
> > +    if [ ${QEMU_STATUS[$h]} -ne 0 ]; then
> > +        echo "Timeout waiting for ${1} on handle ${h}"
> > +        exit 1 #Timeout means the test failed
> > +    fi
> > +}
> > +
> > +
> > +# Launch a QEMU process.
> > +#
> > +# Input parameters:
> > +# $qemu_comm_method: set this variable to 'monitor' (case insensitive)
> > +#                    to use the QEMU HMP monitor for communication.
> > +#                    Otherwise, the default of QMP is used.
> > +# Returns:
> > +# $QEMU_HANDLE: set to a handle value to communicate with this QEMU instance.
> > +#
> > +function _launch_qemu()
> > +{
> > +    local comm=
> > +    local fifo_out=
> > +    local fifo_in=
> > +
> > +    if (shopt -s nocasematch; [[ "${qemu_comm_method}" == "monitor" ]])
> > +    then
> > +        comm="-monitor stdio"
> > +    else
> > +        local qemu_comm_method="qmp"
> > +        comm="-monitor none -qmp stdio"
> > +    fi
> > +
> > +    fifo_out=${QEMU_FIFO_OUT}_${_QEMU_HANDLE}
> > +    fifo_in=${QEMU_FIFO_IN}_${_QEMU_HANDLE}
> > +    mkfifo "${fifo_out}"
> > +    mkfifo "${fifo_in}"
> > +
> > +    "${QEMU}" -nographic -serial none ${comm} "${@}" 2>&1 \
> > +                                                     >"${fifo_out}" \
> > +                                                     <"${fifo_in}" &
> 
> Maybe add "-machine accel=qtest"?
>

OK

> 
> > +    QEMU_PID[${_QEMU_HANDLE}]=$!
> > +
> > +    if [ "${BASH_VERSINFO[0]}" -ge "4" ] && [ "${BASH_VERSINFO[1]}" -ge "1" ]
> > +    then
> > +        # bash >= 4.1 required for automatic fd
> > +        exec {_out_fd}<"${fifo_out}"
> > +        exec {_in_fd}>"${fifo_in}"
> > +    else
> > +        let _out_fd++
> > +        let _in_fd++
> > +        eval "exec ${_out_fd}<'${fifo_out}'"
> > +        eval "exec ${_in_fd}>'${fifo_in}'"
> > +    fi
> > +
> > +    QEMU_OUT[${_QEMU_HANDLE}]=${_out_fd}
> > +    QEMU_IN[${_QEMU_HANDLE}]=${_in_fd}
> > +    QEMU_STATUS[${_QEMU_HANDLE}]=0
> > +
> > +    if [ "${qemu_comm_method}" == "qmp" ]
> > +    then
> > +        # Don't print response, since it has version information in it
> > +        silent=yes _timed_wait_for ${_QEMU_HANDLE} "capabilities"
> > +    fi
> > +    QEMU_HANDLE=${_QEMU_HANDLE}
> > +    let _QEMU_HANDLE++
> > +}
> > +
> > +
> > +# Silenty kills the QEMU process
> > +function _cleanup_qemu()
> > +{
> > +    # QEMU_PID[], QEMU_IN[], QEMU_OUT[] all use same indices
> > +    for i in "${!QEMU_OUT[@]}"
> > +    do
> > +        kill -KILL ${QEMU_PID[$i]}
> > +        wait ${QEMU_PID[$i]} 2>/dev/null # silent kill
> > +        rm -f "${QEMU_FIFO_IN}_${i}" "${QEMU_FIFO_OUT}_${i}"
> > +    done
> > +}
> > +
> > -- 
> > 1.8.3.1
> > 
> > 

Re: [Qemu-devel] [PATCH v2 2/5] block: qemu-iotests - update 085 to use common.qemu

[Jeff Cody]

On Thu, Apr 10, 2014 at 02:10:48PM +0800, Fam Zheng wrote:
> On Wed, 04/09 22:41, Jeff Cody wrote:
> > The new functionality of common.qemu implements the QEMU control
> > and communication functionality that was originally in test 085.
> > 
> > This removes that now-duplicate functionality, and uses the
> > common.qemu functions.
> > 
> 
> Just a note.
> 
> A quick grep shows 067, 071, 081 and 087 are also bash cases with QEMU process.
> Not necessarily in this series but they are also candidates too convert.
>

Good idea - like you said, probably in another series though

Re: [Qemu-devel] [PATCH v2 4/5] block: qemu-iotests - fix image cleanup when using spaced pathnames

[Fam Zheng]

On Thu, 04/10 14:09, Jeff Cody wrote:
> On Thu, Apr 10, 2014 at 08:48:10AM -0600, Eric Blake wrote:
> > On 04/10/2014 08:43 AM, Eric Blake wrote:
> > > On 04/10/2014 06:53 AM, Jeff Cody wrote:
> > > 
> > >>>> +++ b/tests/qemu-iotests/common.rc
> > >>>> @@ -178,10 +178,10 @@ _rm_test_img()
> > >>>>      local img=$1
> > >>>
> > >>> Since we are quoting $img, should we quote $1 as well?
> > >>>
> > 
> > > 
> > > http://austingroupbugs.net/view.php?id=351
> > > 
> > > But even with the notion of an assignment-context argument added to a
> > > future version of POSIX, the reality is that given the present standard,
> > > it's safer to either use "" to ensure no word splitting:
> > 
> > Well, if you were trying to be portable to multiple shells, then it
> > would matter.  But as this script is explicitly being run under
> > /bin/bash, and as bash already has support for declaration utilities
> > where local is one such utility, your script as written is safe without
> > "" in the arguments to local.  So I'm fine whether you choose to change
> > it in a respin or to leave it as written in this version.

Thanks for the thorough explanation, Eric!

> 
> Hi Eric,
> 
> Thanks - I consulted specifically with just the bash documentation, so
> you are right, this script (and likely most of qemu-iotests) is
> bash-only.
> 
> That particular line is context as well, and not an actual change - so
> while it may be a good idea to quote it to make the scripts closer to
> posix-only, my guess is there are quite a few similar lines throughout
> all the qemu-iotests scripts.
> 
> Given that, if we address that it would probably make sense to do that
> in a bash->posix conversion series for all the scripts (likely a low
> priority, however).
> 

OK :)

Thanks,
Fam