[Qemu-devel] [PATCH v4 0/7] Allow Valgrind checking all QEMU processes

[Qemu-devel] [PATCH v4 0/7] Allow Valgrind checking all QEMU processes

[Andrey Shinkevich]

In the current implementation of the QEMU bash iotests, only qemu-io
processes may be run under the Valgrind, which is a useful tool for
finding memory usage issues. Let's allow the common.rc bash script
runing all the QEMU processes, such as qemu-kvm, qemu-img, qemu-ndb
and qemu-vxhs, under the Valgrind tool.

v4:
  01: The patch "iotests: Set read-zeroes on in null block driver for Valgrind"
      was extended with new cases and issued as a separate series.
  02: The new patch "block/nbd: NBDReply is used being uninitialized" was
      added to resolve the failure of the iotest 083 run under Valgrind.

v3:
  01: The new function _casenotrun() was added to the common.rc bash
      script to notify the user of test cases dropped for some reason.
      Suggested by Kevin.
      Particularly, the notification about the nonexistent TMPDIR in
      the test 051 was added (noticed by Vladimir).
  02: The timeout in some test cases was extended for Valgrind because
      it differs when running on the ramdisk.
  03: Due to the common.nbd script has been changed with the commit
      b28f582c, the patch "iotests: amend QEMU NBD process synchronization"
      is actual no more. Note that QEMU_NBD is launched in the bash nested
      shell in the _qemu_nbd_wrapper() as it was before in common.rc.
  04: The patch "iotests: new file to suppress Valgrind errors" was dropped
      due to my superficial understanding of the work of the function
      blk_pread_unthrottled(). Special thanks to Kevin who shed the light
      on the null block driver involved. Now, the parameter 'read-zeroes=on'
      is passed to the null block driver to initialize the buffer in the
      function guess_disk_lchs() that the Valgrind was complaining to.

v2:
  01: The patch 2/7 of v1 was merged into the patch 1/7, suggested by Daniel.
  02: Another patch 7/7 was added to introduce the Valgrind error suppression
      file into the QEMU project.
  Discussed in the email thread with the message ID:
  <1560276131-683243-1-git-send-email-andrey.shinkevich@virtuozzo.com>

Andrey Shinkevich (7):
  iotests: allow Valgrind checking all QEMU processes
  iotests: exclude killed processes from running under Valgrind
  iotests: Add casenotrun report to bash tests
  iotests: Valgrind fails with nonexistent directory
  iotests: extended timeout under Valgrind
  iotests: extend sleeping time under Valgrind
  block/nbd: NBDReply is used being uninitialized

 block/nbd.c                  |  2 +-
 tests/qemu-iotests/028       |  6 +++-
 tests/qemu-iotests/039       |  5 +++
 tests/qemu-iotests/039.out   | 30 +++--------------
 tests/qemu-iotests/051       |  4 +++
 tests/qemu-iotests/061       |  2 ++
 tests/qemu-iotests/061.out   | 12 ++-----
 tests/qemu-iotests/137       |  1 +
 tests/qemu-iotests/137.out   |  6 +---
 tests/qemu-iotests/183       |  9 +++++-
 tests/qemu-iotests/192       |  6 +++-
 tests/qemu-iotests/247       |  6 +++-
 tests/qemu-iotests/common.rc | 76 +++++++++++++++++++++++++++++++++-----------
 13 files changed, 102 insertions(+), 63 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH v4 1/7] iotests: allow Valgrind checking all QEMU processes

[Andrey Shinkevich]

With the '-valgrind' option, let all the QEMU processes be run under
the Valgrind tool. The Valgrind own parameters may be set with its
environment variable VALGRIND_OPTS, e.g.
VALGRIND_OPTS="--leak-check=yes" ./check -qcow2 -valgrind <test#>
or they may be listed in the Valgrind checked file ./.valgrindrc or
~/.valgrindrc like
--memcheck:leak-check=no
--memcheck:track-origins=yes
When QEMU-IO process is being killed, the shell report refers to the
text of the command in _qemu_io_wrapper(), which was modified with this
patch. So, the benchmark output for the tests 039, 061 and 137 is to be
changed also.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
---
 tests/qemu-iotests/039.out   | 30 ++++---------------
 tests/qemu-iotests/061.out   | 12 ++------
 tests/qemu-iotests/137.out   |  6 +---
 tests/qemu-iotests/common.rc | 69 ++++++++++++++++++++++++++++++++------------
 4 files changed, 59 insertions(+), 58 deletions(-)

diff --git a/tests/qemu-iotests/039.out b/tests/qemu-iotests/039.out
index 724d7b2..972c6c0 100644
--- a/tests/qemu-iotests/039.out
+++ b/tests/qemu-iotests/039.out
@@ -11,11 +11,7 @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x1
 ERROR cluster 5 refcount=0 reference=1
 ERROR OFLAG_COPIED data cluster: l2_entry=8000000000050000 refcount=0
@@ -50,11 +46,7 @@ read 512/512 bytes at offset 0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x1
 ERROR cluster 5 refcount=0 reference=1
 Rebuilding refcount structure
@@ -68,11 +60,7 @@ incompatible_features     0x0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x0
 No errors were found on the image.
 
@@ -91,11 +79,7 @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x1
 ERROR cluster 5 refcount=0 reference=1
 ERROR OFLAG_COPIED data cluster: l2_entry=8000000000050000 refcount=0
@@ -105,11 +89,7 @@ Data may be corrupted, or further writes to the image may corrupt it.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x0
 No errors were found on the image.
 *** done
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 1aa7d37..8cb57eb 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -118,11 +118,7 @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 magic                     0x514649fb
 version                   3
 backing_file_offset       0x0
@@ -280,11 +276,7 @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 magic                     0x514649fb
 version                   3
 backing_file_offset       0x0
diff --git a/tests/qemu-iotests/137.out b/tests/qemu-iotests/137.out
index 22d59df..7fed5e6 100644
--- a/tests/qemu-iotests/137.out
+++ b/tests/qemu-iotests/137.out
@@ -35,11 +35,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: Unsupported value 'blubb' for qcow2 option 'overlap-check'. Allowed are any of the following: none, constant, cached, all
 wrote 512/512 bytes at offset 0
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-./common.rc: Killed                  ( if [ "${VALGRIND_QEMU}" == "y" ]; then
-    exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-else
-    exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@";
-fi )
+./common.rc: Killed                  ( _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@" )
 incompatible_features     0x0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 wrote 65536/65536 bytes at offset 0
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 5502c3d..6e461a1 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -60,19 +60,52 @@ if ! . ./common.config
     exit 1
 fi
 
+_qemu_proc_wrapper()
+{
+    local VALGRIND_LOGFILE="$1"
+    shift
+    if [ "${VALGRIND_QEMU}" == "y" ]; then
+        exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$@"
+    else
+        exec "$@"
+    fi
+}
+
+_qemu_proc_valgrind_log()
+{
+    local VALGRIND_LOGFILE="$1"
+    local RETVAL="$2"
+    if [ "${VALGRIND_QEMU}" == "y" ]; then
+        if [ $RETVAL == 99 ]; then
+            cat "${VALGRIND_LOGFILE}"
+        fi
+        rm -f "${VALGRIND_LOGFILE}"
+    fi
+}
+
 _qemu_wrapper()
 {
+    local VALGRIND_LOGFILE="${TEST_DIR}"/$$.valgrind
     (
         if [ -n "${QEMU_NEED_PID}" ]; then
             echo $BASHPID > "${QEMU_TEST_DIR}/qemu-${_QEMU_HANDLE}.pid"
         fi
-        exec "$QEMU_PROG" $QEMU_OPTIONS "$@"
+        _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_PROG" $QEMU_OPTIONS "$@"
     )
+    RETVAL=$?
+    _qemu_proc_valgrind_log "${VALGRIND_LOGFILE}" $RETVAL
+    return $RETVAL
 }
 
 _qemu_img_wrapper()
 {
-    (exec "$QEMU_IMG_PROG" $QEMU_IMG_OPTIONS "$@")
+    local VALGRIND_LOGFILE="${TEST_DIR}"/$$.valgrind
+    (
+        _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IMG_PROG" $QEMU_IMG_OPTIONS "$@"
+    )
+    RETVAL=$?
+    _qemu_proc_valgrind_log "${VALGRIND_LOGFILE}" $RETVAL
+    return $RETVAL
 }
 
 _qemu_io_wrapper()
@@ -85,36 +118,36 @@ _qemu_io_wrapper()
             QEMU_IO_ARGS="--object secret,id=keysec0,data=$IMGKEYSECRET $QEMU_IO_ARGS"
         fi
     fi
-    local RETVAL
     (
-        if [ "${VALGRIND_QEMU}" == "y" ]; then
-            exec valgrind --log-file="${VALGRIND_LOGFILE}" --error-exitcode=99 "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@"
-        else
-            exec "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@"
-        fi
+        _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_IO_PROG" $QEMU_IO_ARGS "$@"
     )
     RETVAL=$?
-    if [ "${VALGRIND_QEMU}" == "y" ]; then
-        if [ $RETVAL == 99 ]; then
-            cat "${VALGRIND_LOGFILE}"
-        fi
-        rm -f "${VALGRIND_LOGFILE}"
-    fi
-    (exit $RETVAL)
+    _qemu_proc_valgrind_log "${VALGRIND_LOGFILE}" $RETVAL
+    return $RETVAL
 }
 
 _qemu_nbd_wrapper()
 {
-    "$QEMU_NBD_PROG" --pid-file="${QEMU_TEST_DIR}/qemu-nbd.pid" \
-                     $QEMU_NBD_OPTIONS "$@"
+    local VALGRIND_LOGFILE="${TEST_DIR}"/$$.valgrind
+    (
+        _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_NBD_PROG" \
+            --pid-file="${QEMU_TEST_DIR}/qemu-nbd.pid" $QEMU_NBD_OPTIONS "$@"
+    )
+    RETVAL=$?
+    _qemu_proc_valgrind_log "${VALGRIND_LOGFILE}" $RETVAL
+    return $RETVAL
 }
 
 _qemu_vxhs_wrapper()
 {
+    local VALGRIND_LOGFILE="${TEST_DIR}"/$$.valgrind
     (
         echo $BASHPID > "${TEST_DIR}/qemu-vxhs.pid"
-        exec "$QEMU_VXHS_PROG" $QEMU_VXHS_OPTIONS "$@"
+        _qemu_proc_wrapper "${VALGRIND_LOGFILE}" "$QEMU_VXHS_PROG" $QEMU_VXHS_OPTIONS "$@"
     )
+    RETVAL=$?
+    _qemu_proc_valgrind_log "${VALGRIND_LOGFILE}" $RETVAL
+    return $RETVAL
 }
 
 export QEMU=_qemu_wrapper
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 2/7] iotests: exclude killed processes from running under Valgrind

[Andrey Shinkevich]

 The Valgrind tool fails to manage its termination when QEMU raises the
 signal SIGKILL in the multi-threaded process. The bug has been
 reported to the Valgrind maintainers and was registered as Bug 409141.
 Let's exclude such test cases from running under the Valgrind until
 new release of it because checking for the memory issues is covered by
 other test cases.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
---
 tests/qemu-iotests/039 | 5 +++++
 tests/qemu-iotests/061 | 2 ++
 tests/qemu-iotests/137 | 1 +
 3 files changed, 8 insertions(+)

diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index 0d4e963..95115e2 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -65,6 +65,7 @@ echo "== Creating a dirty image file =="
 IMGOPTS="compat=1.1,lazy_refcounts=on"
 _make_test_img $size
 
+VALGRIND_QEMU="" \
 $QEMU_IO -c "write -P 0x5a 0 512" \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 \
     | _filter_qemu_io
@@ -100,6 +101,7 @@ echo "== Opening a dirty image read/write should repair it =="
 IMGOPTS="compat=1.1,lazy_refcounts=on"
 _make_test_img $size
 
+VALGRIND_QEMU="" \
 $QEMU_IO -c "write -P 0x5a 0 512" \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 \
     | _filter_qemu_io
@@ -118,6 +120,7 @@ echo "== Creating an image file with lazy_refcounts=off =="
 IMGOPTS="compat=1.1,lazy_refcounts=off"
 _make_test_img $size
 
+VALGRIND_QEMU="" \
 $QEMU_IO -c "write -P 0x5a 0 512" \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 \
     | _filter_qemu_io
@@ -151,6 +154,7 @@ echo "== Changing lazy_refcounts setting at runtime =="
 IMGOPTS="compat=1.1,lazy_refcounts=off"
 _make_test_img $size
 
+VALGRIND_QEMU="" \
 $QEMU_IO -c "reopen -o lazy-refcounts=on" \
          -c "write -P 0x5a 0 512" \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 \
@@ -163,6 +167,7 @@ _check_test_img
 IMGOPTS="compat=1.1,lazy_refcounts=on"
 _make_test_img $size
 
+VALGRIND_QEMU="" \
 $QEMU_IO -c "reopen -o lazy-refcounts=off" \
          -c "write -P 0x5a 0 512" \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 \
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index d7dbd7e..5d0724c 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -73,6 +73,7 @@ echo
 echo "=== Testing dirty version downgrade ==="
 echo
 IMGOPTS="compat=1.1,lazy_refcounts=on" _make_test_img 64M
+VALGRIND_QEMU="" \
 $QEMU_IO -c "write -P 0x2a 0 128k" -c flush \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 | _filter_qemu_io
 $PYTHON qcow2.py "$TEST_IMG" dump-header
@@ -107,6 +108,7 @@ echo
 echo "=== Testing dirty lazy_refcounts=off ==="
 echo
 IMGOPTS="compat=1.1,lazy_refcounts=on" _make_test_img 64M
+VALGRIND_QEMU="" \
 $QEMU_IO -c "write -P 0x2a 0 128k" -c flush \
          -c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 | _filter_qemu_io
 $PYTHON qcow2.py "$TEST_IMG" dump-header
diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137
index 0c3d2a1..a442fc8 100755
--- a/tests/qemu-iotests/137
+++ b/tests/qemu-iotests/137
@@ -130,6 +130,7 @@ echo
 
 # Whether lazy-refcounts was actually enabled can easily be tested: Check if
 # the dirty bit is set after a crash
+VALGRIND_QEMU="" \
 $QEMU_IO \
     -c "reopen -o lazy-refcounts=on,overlap-check=blubb" \
     -c "write -P 0x5a 0 512" \
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 3/7] iotests: Add casenotrun report to bash tests

[Andrey Shinkevich]

The new function _casenotrun() is to be invoked if a test case cannot
be run for some reason. The user will be notified by a message passed
to the function.

Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
---
 tests/qemu-iotests/common.rc | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 6e461a1..1089050 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -428,6 +428,13 @@ _notrun()
     exit
 }
 
+# bail out, setting up .casenotrun file
+#
+_casenotrun()
+{
+    echo "    [case not run] $*" >>"$OUTPUT_DIR/$seq.casenotrun"
+}
+
 # just plain bail out
 #
 _fail()
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 4/7] iotests: Valgrind fails with nonexistent directory

[Andrey Shinkevich]

The Valgrind uses the exported variable TMPDIR and fails if the
directory does not exist. Let us exclude such a test case from
being run under the Valgrind and notify the user of it.

Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
---
 tests/qemu-iotests/051 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/051 b/tests/qemu-iotests/051
index ce942a5..f8141ca 100755
--- a/tests/qemu-iotests/051
+++ b/tests/qemu-iotests/051
@@ -377,6 +377,10 @@ printf %b "qemu-io $device_id \"write -P 0x33 0 4k\"\ncommit $device_id\n" |
 $QEMU_IO -c "read -P 0x33 0 4k" "$TEST_IMG" | _filter_qemu_io
 
 # Using snapshot=on with a non-existent TMPDIR
+if [ "${VALGRIND_QEMU}" == "y" ]; then
+    _casenotrun "Valgrind needs a valid TMPDIR for itself"
+fi
+VALGRIND_QEMU="" \
 TMPDIR=/nonexistent run_qemu -drive driver=null-co,snapshot=on
 
 # Using snapshot=on together with read-only=on
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 5/7] iotests: extended timeout under Valgrind

[Andrey Shinkevich]

As the iotests run longer under the Valgrind, the QEMU_COMM_TIMEOUT is
to be increased in the test cases 028, 183 and 192 when running under
the Valgrind.

Suggested-by: Roman Kagan <rkagan@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/028 | 6 +++++-
 tests/qemu-iotests/183 | 9 ++++++++-
 tests/qemu-iotests/192 | 6 +++++-
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
index 01f4959..71301ec 100755
--- a/tests/qemu-iotests/028
+++ b/tests/qemu-iotests/028
@@ -110,7 +110,11 @@ echo
 qemu_comm_method="monitor"
 _launch_qemu -drive file="${TEST_IMG}",cache=${CACHEMODE},id=disk
 h=$QEMU_HANDLE
-QEMU_COMM_TIMEOUT=1
+if [ "${VALGRIND_QEMU}" == "y" ]; then
+    QEMU_COMM_TIMEOUT=7
+else
+    QEMU_COMM_TIMEOUT=1
+fi
 
 # Silence output since it contains the disk image path and QEMU's readline
 # character echoing makes it very hard to filter the output. Plus, there
diff --git a/tests/qemu-iotests/183 b/tests/qemu-iotests/183
index fbe5a99..04fb344 100755
--- a/tests/qemu-iotests/183
+++ b/tests/qemu-iotests/183
@@ -94,8 +94,15 @@ if echo "$reply" | grep "compiled without old-style" > /dev/null; then
     _notrun "migrate -b support not compiled in"
 fi
 
-QEMU_COMM_TIMEOUT=0.1 qemu_cmd_repeat=50 silent=yes \
+timeout_comm=$QEMU_COMM_TIMEOUT
+if [ "${VALGRIND_QEMU}" == "y" ]; then
+    QEMU_COMM_TIMEOUT=4
+else
+    QEMU_COMM_TIMEOUT=0.1
+fi
+qemu_cmd_repeat=50 silent=yes \
     _send_qemu_cmd $src "{ 'execute': 'query-migrate' }" '"status": "completed"'
+QEMU_COMM_TIMEOUT=$timeout_comm
 _send_qemu_cmd $src "{ 'execute': 'query-status' }" "return"
 
 echo
diff --git a/tests/qemu-iotests/192 b/tests/qemu-iotests/192
index 6193257..0344322 100755
--- a/tests/qemu-iotests/192
+++ b/tests/qemu-iotests/192
@@ -60,7 +60,11 @@ fi
 qemu_comm_method="monitor"
 _launch_qemu -drive $DRIVE_ARG -incoming defer
 h=$QEMU_HANDLE
-QEMU_COMM_TIMEOUT=1
+if [ "${VALGRIND_QEMU}" == "y" ]; then
+    QEMU_COMM_TIMEOUT=7
+else
+    QEMU_COMM_TIMEOUT=1
+fi
 
 _send_qemu_cmd $h "nbd_server_start unix:$TEST_DIR/nbd" "(qemu)"
 _send_qemu_cmd $h "nbd_server_add -w drive0" "(qemu)"
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 6/7] iotests: extend sleeping time under Valgrind

[Andrey Shinkevich]

To synchronize the time when QEMU is running longer under the Valgrind,
increase the sleeping time in the test 247.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/247 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/247 b/tests/qemu-iotests/247
index 546a794..c853b73 100755
--- a/tests/qemu-iotests/247
+++ b/tests/qemu-iotests/247
@@ -57,7 +57,11 @@ TEST_IMG="$TEST_IMG.4" _make_test_img $size
 {"execute":"block-commit",
  "arguments":{"device":"format-4", "top-node": "format-2", "base-node":"format-0", "job-id":"job0"}}
 EOF
-sleep 1
+if [ "${VALGRIND_QEMU}" == "y" ]; then
+    sleep 10
+else
+    sleep 1
+fi
 echo '{"execute":"quit"}'
 ) | $QEMU -qmp stdio -nographic -nodefaults \
     -blockdev file,node-name=file-0,filename=$TEST_IMG.0,auto-read-only=on \
-- 
1.8.3.1

[Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Andrey Shinkevich]

In case nbd_co_receive_one_chunk() fails in
nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
the check nbd_reply_is_simple() without being initialized. The iotest
083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
The alternative solution is to swap the operands in the condition:
'if (s->quit || nbd_reply_is_simple(reply))'

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
---
 block/nbd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/nbd.c b/block/nbd.c
index 81edabb..8480ad4 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -786,7 +786,7 @@ static int nbd_co_receive_cmdread_reply(BDRVNBDState *s, uint64_t handle,
                                         int *request_ret, Error **errp)
 {
     NBDReplyChunkIter iter;
-    NBDReply reply;
+    NBDReply reply = {};
     void *payload = NULL;
     Error *local_err = NULL;
 
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1426 bytes --]

On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
> In case nbd_co_receive_one_chunk() fails in
> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
> the check nbd_reply_is_simple() without being initialized. The iotest
> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
> The alternative solution is to swap the operands in the condition:
> 'if (s->quit || nbd_reply_is_simple(reply))'
> 
> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
> ---
>  block/nbd.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Huh. Very similar to
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
affects a different function. I can queue this one through my NBD tree
to get both in my rc2 pull request.

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/block/nbd.c b/block/nbd.c
> index 81edabb..8480ad4 100644
> --- a/block/nbd.c
> +++ b/block/nbd.c
> @@ -786,7 +786,7 @@ static int nbd_co_receive_cmdread_reply(BDRVNBDState *s, uint64_t handle,
>                                          int *request_ret, Error **errp)
>  {
>      NBDReplyChunkIter iter;
> -    NBDReply reply;
> +    NBDReply reply = {};
>      void *payload = NULL;
>      Error *local_err = NULL;
>  
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Andrey Shinkevich]

On 19/07/2019 17:34, Eric Blake wrote:
> On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
>> In case nbd_co_receive_one_chunk() fails in
>> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
>> the check nbd_reply_is_simple() without being initialized. The iotest
>> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
>> The alternative solution is to swap the operands in the condition:
>> 'if (s->quit || nbd_reply_is_simple(reply))'
>>
>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>> ---
>>   block/nbd.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> Huh. Very similar to
> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
> affects a different function. I can queue this one through my NBD tree
> to get both in my rc2 pull request.
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>
> 

Many thanks,
Andrey

>>
>> diff --git a/block/nbd.c b/block/nbd.c
>> index 81edabb..8480ad4 100644
>> --- a/block/nbd.c
>> +++ b/block/nbd.c
>> @@ -786,7 +786,7 @@ static int nbd_co_receive_cmdread_reply(BDRVNBDState *s, uint64_t handle,
>>                                           int *request_ret, Error **errp)
>>   {
>>       NBDReplyChunkIter iter;
>> -    NBDReply reply;
>> +    NBDReply reply = {};
>>       void *payload = NULL;
>>       Error *local_err = NULL;
>>   
>>
> 

-- 
With the best regards,
Andrey Shinkevich

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1660 bytes --]

On 7/19/19 9:34 AM, Eric Blake wrote:
> On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
>> In case nbd_co_receive_one_chunk() fails in
>> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
>> the check nbd_reply_is_simple() without being initialized. The iotest
>> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
>> The alternative solution is to swap the operands in the condition:
>> 'if (s->quit || nbd_reply_is_simple(reply))'
>>
>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>> ---
>>  block/nbd.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> Huh. Very similar to
> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
> affects a different function. I can queue this one through my NBD tree
> to get both in my rc2 pull request.
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

Actually, since this is the second patch on the same topic, I'm
wondering if it's better to use the following one-liner to fix BOTH
issues and without relying on a gcc extension:

diff --git i/block/nbd.c w/block/nbd.c
index 8d565cc624ec..f751a8e633e5 100644
--- i/block/nbd.c
+++ w/block/nbd.c
@@ -640,6 +640,7 @@ static coroutine_fn int nbd_co_receive_one_chunk(
                                           request_ret, qiov, payload,
errp);

     if (ret < 0) {
+        memset(reply, 0, sizeof *reply);
         s->quit = true;
     } else {
         /* For assert at loop start in nbd_connection_entry */

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Andrey Shinkevich]

On 19/07/2019 17:44, Eric Blake wrote:
> On 7/19/19 9:34 AM, Eric Blake wrote:
>> On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
>>> In case nbd_co_receive_one_chunk() fails in
>>> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
>>> the check nbd_reply_is_simple() without being initialized. The iotest
>>> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
>>> The alternative solution is to swap the operands in the condition:
>>> 'if (s->quit || nbd_reply_is_simple(reply))'
>>>
>>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>>> ---
>>>   block/nbd.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> Huh. Very similar to
>> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
>> affects a different function. I can queue this one through my NBD tree
>> to get both in my rc2 pull request.
>>
>> Reviewed-by: Eric Blake <eblake@redhat.com>
> 
> Actually, since this is the second patch on the same topic, I'm
> wondering if it's better to use the following one-liner to fix BOTH
> issues and without relying on a gcc extension:
> 
> diff --git i/block/nbd.c w/block/nbd.c
> index 8d565cc624ec..f751a8e633e5 100644
> --- i/block/nbd.c
> +++ w/block/nbd.c
> @@ -640,6 +640,7 @@ static coroutine_fn int nbd_co_receive_one_chunk(
>                                             request_ret, qiov, payload,
> errp);
> 
>       if (ret < 0) {
> +        memset(reply, 0, sizeof *reply);

The call to memset() consumes more CPU time. I don't know how frequent 
the "ret < 0" path is. The initialization ' = {0}' is cheaper.
Is it safe to swap the operands in the condition in 
nbd_reply_chunk_iter_receive():
'if (s->quit || nbd_reply_is_simple(reply))' ?

Andrey

>           s->quit = true;
>       } else {
>           /* For assert at loop start in nbd_connection_entry */
> 

-- 
With the best regards,
Andrey Shinkevich

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 3803 bytes --]

On 7/19/19 10:00 AM, Andrey Shinkevich wrote:
> 
> 
> On 19/07/2019 17:44, Eric Blake wrote:
>> On 7/19/19 9:34 AM, Eric Blake wrote:
>>> On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
>>>> In case nbd_co_receive_one_chunk() fails in
>>>> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
>>>> the check nbd_reply_is_simple() without being initialized. The iotest
>>>> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
>>>> The alternative solution is to swap the operands in the condition:
>>>> 'if (s->quit || nbd_reply_is_simple(reply))'
>>>>
>>>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>>>> ---
>>>>   block/nbd.c | 2 +-
>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> Huh. Very similar to
>>> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
>>> affects a different function. I can queue this one through my NBD tree
>>> to get both in my rc2 pull request.
>>>
>>> Reviewed-by: Eric Blake <eblake@redhat.com>
>>
>> Actually, since this is the second patch on the same topic, I'm
>> wondering if it's better to use the following one-liner to fix BOTH
>> issues and without relying on a gcc extension:
>>
>> diff --git i/block/nbd.c w/block/nbd.c
>> index 8d565cc624ec..f751a8e633e5 100644
>> --- i/block/nbd.c
>> +++ w/block/nbd.c
>> @@ -640,6 +640,7 @@ static coroutine_fn int nbd_co_receive_one_chunk(
>>                                             request_ret, qiov, payload,
>> errp);
>>
>>       if (ret < 0) {
>> +        memset(reply, 0, sizeof *reply);
> 
> The call to memset() consumes more CPU time. I don't know how frequent 
> the "ret < 0" path is. The initialization ' = {0}' is cheaper.

Wrong.  The 'ret < 0' path only happens on error, while the '= {0}' path
happens on ALL cases including success (if you'll look at the generated
assembly code, gcc should emit the same assembly sequence for
zero-initialization of the struct, whether that is a call to memset() or
just inline assignments of zeros based on the small size of the struct,
whether or not your code uses memset or '={}').  You don't need to
optimize the error case, because on error, your NBD connection is dead,
and you have worse problems.  Pre-initialization that is going to be
overwritten on success is worse (although probably immeasurably, because
it is likely in the noise) than exactly one initialization on any
control flow path.

> Is it safe to swap the operands in the condition in 
> nbd_reply_chunk_iter_receive():
> 'if (s->quit || nbd_reply_is_simple(reply))' ?

Yes, swapping the conditional appears to fix the only use of reply where
it is used uninitialized, at least in the current state of the code (but
it took me longer to audit that).  So if we're going for a one-line fix
for both observations of the problem, changing the conditional instead
of a memset on error is also acceptable for now (and maybe makes the
error case slightly faster, but that's not a big deal, because the error
case already means the NBD connection has bigger problems) - but who
knows what future uses of reply might creep in to an unsuspecting patch
writer that doesn't see the (in)action at a distance?  Which way is more
maintainable, proving that the low-level code always initializes the
variable (easy, since that can be checked at a single function), or
proving that all high-level uses may pass in an uninitialized variable
that is still left uninit on error but that they only use the variable
on success (harder, since now you have to audit every single caller to
see how they use reply on failure)?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v4 7/7] block/nbd: NBDReply is used being uninitialized

[Andrey Shinkevich]

On 19/07/2019 18:15, Eric Blake wrote:
> On 7/19/19 10:00 AM, Andrey Shinkevich wrote:
>>
>>
>> On 19/07/2019 17:44, Eric Blake wrote:
>>> On 7/19/19 9:34 AM, Eric Blake wrote:
>>>> On 7/19/19 4:40 AM, Andrey Shinkevich wrote:
>>>>> In case nbd_co_receive_one_chunk() fails in
>>>>> nbd_reply_chunk_iter_receive(), 'NBDReply reply' parameter is used in
>>>>> the check nbd_reply_is_simple() without being initialized. The iotest
>>>>> 083 does not pass under the Valgrind: $./check -nbd -valgrind 083.
>>>>> The alternative solution is to swap the operands in the condition:
>>>>> 'if (s->quit || nbd_reply_is_simple(reply))'
>>>>>
>>>>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>>>>> ---
>>>>>    block/nbd.c | 2 +-
>>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> Huh. Very similar to
>>>> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg03712.html, but
>>>> affects a different function. I can queue this one through my NBD tree
>>>> to get both in my rc2 pull request.
>>>>
>>>> Reviewed-by: Eric Blake <eblake@redhat.com>
>>>
>>> Actually, since this is the second patch on the same topic, I'm
>>> wondering if it's better to use the following one-liner to fix BOTH
>>> issues and without relying on a gcc extension:
>>>
>>> diff --git i/block/nbd.c w/block/nbd.c
>>> index 8d565cc624ec..f751a8e633e5 100644
>>> --- i/block/nbd.c
>>> +++ w/block/nbd.c
>>> @@ -640,6 +640,7 @@ static coroutine_fn int nbd_co_receive_one_chunk(
>>>                                              request_ret, qiov, payload,
>>> errp);
>>>
>>>        if (ret < 0) {
>>> +        memset(reply, 0, sizeof *reply);
>>
>> The call to memset() consumes more CPU time. I don't know how frequent
>> the "ret < 0" path is. The initialization ' = {0}' is cheaper.
> 
> Wrong.  The 'ret < 0' path only happens on error, while the '= {0}' path
> happens on ALL cases including success (if you'll look at the generated
> assembly code, gcc should emit the same assembly sequence for
> zero-initialization of the struct, whether that is a call to memset() or
> just inline assignments of zeros based on the small size of the struct,
> whether or not your code uses memset or '={}').  You don't need to
> optimize the error case, because on error, your NBD connection is dead,
> and you have worse problems.  Pre-initialization that is going to be
> overwritten on success is worse (although probably immeasurably, because
> it is likely in the noise) than exactly one initialization on any
> control flow path.
> 
>> Is it safe to swap the operands in the condition in
>> nbd_reply_chunk_iter_receive():
>> 'if (s->quit || nbd_reply_is_simple(reply))' ?
> 
> Yes, swapping the conditional appears to fix the only use of reply where
> it is used uninitialized, at least in the current state of the code (but
> it took me longer to audit that).  So if we're going for a one-line fix
> for both observations of the problem, changing the conditional instead
> of a memset on error is also acceptable for now (and maybe makes the
> error case slightly faster, but that's not a big deal, because the error
> case already means the NBD connection has bigger problems) - but who
> knows what future uses of reply might creep in to an unsuspecting patch
> writer that doesn't see the (in)action at a distance?  Which way is more
> maintainable, proving that the low-level code always initializes the
> variable (easy, since that can be checked at a single function), or
> proving that all high-level uses may pass in an uninitialized variable
> that is still left uninit on error but that they only use the variable
> on success (harder, since now you have to audit every single caller to
> see how they use reply on failure)?
> 

Sounds reasonable. Clear now. So, I will detach this patch from the 
series in the next v5 version.

Andrey
-- 
With the best regards,
Andrey Shinkevich