[PATCH v3 0/4] Improve integration of iotests in the meson test harness

[PATCH v3 0/4] Improve integration of iotests in the meson test harness

[Thomas Huth]

Though "make check-block" is currently already run via the meson test
runner, it still looks like an oddball in the output of "make check". It
would be nicer if the iotests would show up like the other tests suites.

My original plan was to add each iotests individually from meson.build,
but I did not get that done reliably yet [*], so here's now a cut-down
version to improve the situation at least a little bit: The first three
patches are preparation for the clean-up (long-term goal is to get rid
of check-block.sh, though we're not quite there yet), and the final
patch adds the iotests not as separate test target in the meson test
harness anymore. This way, we can now finally get the output of failed
tests on the console again (unless you're running meson test in verbose
mode, where meson only puts this to the log file - for incomprehensible
reasons), so this should hopefully help to diagnose problems with the
iotests in most cases more easily.

[*] See v2 here:
    https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg01942.html

Thomas Huth (4):
  tests/qemu-iotests: Rework the checks and spots using GNU sed
  tests/qemu-iotests/meson.build: Improve the indentation
  tests/qemu-iotests: Move the bash and sanitizer checks to meson.build
  tests: Do not treat the iotests as separate meson test target anymore

 meson.build                      |  6 +--
 scripts/mtest2make.py            |  4 --
 tests/Makefile.include           |  9 +---
 tests/check-block.sh             | 38 -----------------
 tests/qemu-iotests/271           |  2 +-
 tests/qemu-iotests/common.filter | 65 ++++++++++++++--------------
 tests/qemu-iotests/common.rc     | 45 ++++++++++----------
 tests/qemu-iotests/meson.build   | 73 ++++++++++++++++++++------------
 8 files changed, 106 insertions(+), 136 deletions(-)

-- 
2.27.0

[PATCH v3 1/4] tests/qemu-iotests: Rework the checks and spots using GNU sed

[Thomas Huth]

Instead of failing the iotests if GNU sed is not available (or skipping
them completely in the check-block.sh script), it would be better to
simply skip the bash-based tests that rely on GNU sed, so that the other
tests could still be run. Thus we now explicitely use "gsed" (either as
direct program or as a wrapper around "sed" if it's the GNU version)
in the spots that rely on the GNU sed behavior. Statements that use the
"-r" parameter of sed have been switched to use "-E" instead, since this
switch is supported by all sed versions on our supported build hosts
(most also support "-r", but macOS' sed only supports "-E"). With all
these changes in place, we then can also remove the sed checks from the
check-block.sh script, so that "make check-block" can now be run on
systems without GNU sed, too.

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/check-block.sh             | 12 ------
 tests/qemu-iotests/271           |  2 +-
 tests/qemu-iotests/common.filter | 65 ++++++++++++++++----------------
 tests/qemu-iotests/common.rc     | 45 +++++++++++-----------
 4 files changed, 57 insertions(+), 67 deletions(-)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index 18f7433901..f59496396c 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -48,18 +48,6 @@ if LANG=C bash --version | grep -q 'GNU bash, version [123]' ; then
     skip "bash version too old ==> Not running the qemu-iotests."
 fi
 
-if ! (sed --version | grep 'GNU sed') > /dev/null 2>&1 ; then
-    if ! command -v gsed >/dev/null 2>&1; then
-        skip "GNU sed not available ==> Not running the qemu-iotests."
-    fi
-else
-    # Double-check that we're not using BusyBox' sed which says
-    # that "This is not GNU sed version 4.0" ...
-    if sed --version | grep -q 'not GNU sed' ; then
-        skip "BusyBox sed not supported ==> Not running the qemu-iotests."
-    fi
-fi
-
 cd tests/qemu-iotests
 
 # QEMU_CHECK_BLOCK_AUTO is used to disable some unstable sub-tests
diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271
index 2775b4d130..c7c2cadda0 100755
--- a/tests/qemu-iotests/271
+++ b/tests/qemu-iotests/271
@@ -896,7 +896,7 @@ _make_test_img -o extended_l2=on 1M
 # Second and third writes in _concurrent_io() are independent and may finish in
 # different order. So, filter offset out to match both possible variants.
 _concurrent_io     | $QEMU_IO | _filter_qemu_io | \
-    $SED -e 's/\(20480\|40960\)/OFFSET/'
+    sed -e 's/\(20480\|40960\)/OFFSET/'
 _concurrent_verify | $QEMU_IO | _filter_qemu_io
 
 # success, all done
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 75cc241580..f6e6b3bd04 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -21,44 +21,44 @@
 
 _filter_date()
 {
-    $SED -re 's/[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}/yyyy-mm-dd hh:mm:ss/'
+    sed -Ee 's/[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}/yyyy-mm-dd hh:mm:ss/'
 }
 
 _filter_vmstate_size()
 {
-    $SED -r -e 's/[0-9. ]{5} [KMGT]iB/     SIZE/' \
-            -e 's/[0-9. ]{5} B/   SIZE/'
+    sed -E -e 's/[0-9. ]{5} [KMGT]iB/     SIZE/' \
+           -e 's/[0-9. ]{5} B/   SIZE/'
 }
 
 _filter_generated_node_ids()
 {
-    $SED -re 's/\#block[0-9]{3,}/NODE_NAME/'
+    sed -Ee 's/\#block[0-9]{3,}/NODE_NAME/'
 }
 
 _filter_qom_path()
 {
-    $SED -e '/Attached to:/s/\device[[0-9]\+\]/device[N]/g'
+    gsed -e '/Attached to:/s/\device[[0-9]\+\]/device[N]/g'
 }
 
 # replace occurrences of the actual TEST_DIR value with TEST_DIR
 _filter_testdir()
 {
-    $SED -e "s#$TEST_DIR/#TEST_DIR/#g" \
-         -e "s#$SOCK_DIR/#SOCK_DIR/#g" \
-         -e "s#SOCK_DIR/fuse-#TEST_DIR/#g"
+    sed -e "s#$TEST_DIR/#TEST_DIR/#g" \
+        -e "s#$SOCK_DIR/#SOCK_DIR/#g" \
+        -e "s#SOCK_DIR/fuse-#TEST_DIR/#g"
 }
 
 # replace occurrences of the actual IMGFMT value with IMGFMT
 _filter_imgfmt()
 {
-    $SED -e "s#$IMGFMT#IMGFMT#g"
+    sed -e "s#$IMGFMT#IMGFMT#g"
 }
 
 # Replace error message when the format is not supported and delete
 # the output lines after the first one
 _filter_qemu_img_check()
 {
-    $SED -e '/allocated.*fragmented.*compressed clusters/d' \
+    gsed -e '/allocated.*fragmented.*compressed clusters/d' \
         -e 's/qemu-img: This image format does not support checks/No errors were found on the image./' \
         -e '/Image end offset: [0-9]\+/d'
 }
@@ -66,13 +66,14 @@ _filter_qemu_img_check()
 # Removes \r from messages
 _filter_win32()
 {
-    $SED -e 's/\r//g'
+    gsed -e 's/\r//g'
 }
 
 # sanitize qemu-io output
 _filter_qemu_io()
 {
-    _filter_win32 | $SED -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \
+    _filter_win32 | \
+    gsed -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \
         -e "s/: line [0-9][0-9]*:  *[0-9][0-9]*\( Aborted\| Killed\)/:\1/" \
         -e "s/qemu-io> //g"
 }
@@ -80,7 +81,7 @@ _filter_qemu_io()
 # replace occurrences of QEMU_PROG with "qemu"
 _filter_qemu()
 {
-    $SED -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \
+    gsed -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \
         -e 's#^QEMU [0-9]\+\.[0-9]\+\.[0-9]\+ monitor#QEMU X.Y.Z monitor#' \
         -e $'s#\r##' # QEMU monitor uses \r\n line endings
 }
@@ -89,7 +90,7 @@ _filter_qemu()
 _filter_qmp()
 {
     _filter_win32 | \
-    $SED -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \
+    gsed -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \
         -e 's#^{"QMP":.*}$#QMP_VERSION#' \
         -e '/^    "QMP": {\s*$/, /^    }\s*$/ c\' \
         -e '    QMP_VERSION'
@@ -98,32 +99,32 @@ _filter_qmp()
 # readline makes HMP command strings so long that git complains
 _filter_hmp()
 {
-    $SED -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \
+    gsed -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \
         -e $'s/\e\\[K//g'
 }
 
 # replace block job offset
 _filter_block_job_offset()
 {
-    $SED -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/'
+    gsed -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/'
 }
 
 # replace block job len
 _filter_block_job_len()
 {
-    $SED -e 's/, "len": [0-9]\+,/, "len": LEN,/g'
+    gsed -e 's/, "len": [0-9]\+,/, "len": LEN,/g'
 }
 
 # replace actual image size (depends on the host filesystem)
 _filter_actual_image_size()
 {
-    $SED -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g'
+    gsed -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g'
 }
 
 # Filename filters for qemu-img create
 _filter_img_create_filenames()
 {
-    $SED \
+    sed \
         -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
         -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \
         -e "s#$TEST_DIR#TEST_DIR#g" \
@@ -141,7 +142,7 @@ _do_filter_img_create()
     # precedes ", fmt=") and the options part ($options, which starts
     # with "fmt=")
     # (And just echo everything before the first "^Formatting")
-    readarray formatting_line < <($SED -e 's/, fmt=/\n/')
+    readarray formatting_line < <(gsed -e 's/, fmt=/\n/')
 
     filename_part=${formatting_line[0]}
     unset formatting_line[0]
@@ -168,11 +169,11 @@ _do_filter_img_create()
     options=$(
         echo "$options" \
         | tr '\n' '\0' \
-        | $SED -e 's/ \([a-z0-9_.-]*\)=/\n\1=/g' \
+        | gsed -e 's/ \([a-z0-9_.-]*\)=/\n\1=/g' \
         | grep -a -e '^fmt' -e '^size' -e '^backing' -e '^preallocation' \
                   -e '^encryption' "${grep_data_file[@]}" \
         | _filter_img_create_filenames \
-        | $SED \
+        | sed \
             -e 's/^\(fmt\)/0-\1/' \
             -e 's/^\(size\)/1-\1/' \
             -e 's/^\(backing\)/2-\1/' \
@@ -180,9 +181,9 @@ _do_filter_img_create()
             -e 's/^\(encryption\)/4-\1/' \
             -e 's/^\(preallocation\)/8-\1/' \
         | LC_ALL=C sort \
-        | $SED -e 's/^[0-9]-//' \
+        | sed -e 's/^[0-9]-//' \
         | tr '\n\0' ' \n' \
-        | $SED -e 's/^ *$//' -e 's/ *$//'
+        | sed -e 's/^ *$//' -e 's/ *$//'
     )
 
     if [ -n "$options" ]; then
@@ -208,7 +209,7 @@ _filter_img_create()
 
 _filter_img_create_size()
 {
-    $SED -e "s# size=[0-9]\\+# size=SIZE#g"
+    gsed -e "s# size=[0-9]\\+# size=SIZE#g"
 }
 
 _filter_img_info()
@@ -222,7 +223,7 @@ _filter_img_info()
 
     discard=0
     regex_json_spec_start='^ *"format-specific": \{'
-    $SED -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
+    gsed -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \
         -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \
         -e "s#$TEST_DIR#TEST_DIR#g" \
         -e "s#$SOCK_DIR#SOCK_DIR#g" \
@@ -284,7 +285,7 @@ _filter_qemu_img_map()
         data_file_filter=(-e "s#$data_file_pattern#\\1#")
     fi
 
-    $SED -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \
+    sed -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \
         -e 's/"offset": [0-9]\+/"offset": OFFSET/g' \
         -e 's/Mapped to *//' \
         "${data_file_filter[@]}" \
@@ -298,7 +299,7 @@ _filter_nbd()
     # receive callbacks sometimes, making them unreliable.
     #
     # Filter out the TCP port number since this changes between runs.
-    $SED -e '/nbd\/.*\.c:/d' \
+    sed -e '/nbd\/.*\.c:/d' \
         -e 's#127\.0\.0\.1:[0-9]*#127.0.0.1:PORT#g' \
         -e "s#?socket=$SOCK_DIR#?socket=SOCK_DIR#g" \
         -e 's#\(foo\|PORT/\?\|.sock\): Failed to .*$#\1#'
@@ -335,14 +336,14 @@ sys.stdout.write(result)'
 
 _filter_authz_check_tls()
 {
-    $SED -e 's/TLS x509 authz check for .* is denied/TLS x509 authz check for DISTINGUISHED-NAME is denied/'
+    sed -e 's/TLS x509 authz check for .* is denied/TLS x509 authz check for DISTINGUISHED-NAME is denied/'
 }
 
 _filter_qcow2_compression_type_bit()
 {
-    $SED -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
-         -e 's/\(incompatible_features.*\), 3\]/\1]/' \
-         -e 's/\(incompatible_features.*\), 3\(,.*\)/\1\2/'
+    gsed -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
+        -e 's/\(incompatible_features.*\), 3\]/\1]/' \
+        -e 's/\(incompatible_features.*\), 3\(,.*\)/\1\2/'
 }
 
 # make sure this script returns success
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 9885030b43..3bfd94c2e0 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -17,17 +17,28 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-SED=
-for sed in sed gsed; do
-    ($sed --version | grep 'GNU sed') > /dev/null 2>&1
-    if [ "$?" -eq 0 ]; then
-        SED=$sed
-        break
-    fi
-done
-if [ -z "$SED" ]; then
-    echo "$0: GNU sed not found"
-    exit 1
+# bail out, setting up .notrun file
+_notrun()
+{
+    echo "$*" >"$OUTPUT_DIR/$seq.notrun"
+    echo "$seq not run: $*"
+    status=0
+    exit
+}
+
+if ! command -v gsed >/dev/null 2>&1; then
+    if sed --version 2>&1 | grep -v 'not GNU sed' | grep 'GNU sed' > /dev/null;
+    then
+        gsed()
+        {
+            sed "$@"
+        }
+    else
+        gsed()
+        {
+            _notrun "GNU sed not available"
+        }
+    fi
 fi
 
 dd()
@@ -722,16 +733,6 @@ _img_info()
         done
 }
 
-# bail out, setting up .notrun file
-#
-_notrun()
-{
-    echo "$*" >"$OUTPUT_DIR/$seq.notrun"
-    echo "$seq not run: $*"
-    status=0
-    exit
-}
-
 # bail out, setting up .casenotrun file
 # The function _casenotrun() is used as a notifier. It is the
 # caller's responsibility to make skipped a particular test.
@@ -920,7 +921,7 @@ _require_working_luks()
     IMGFMT='luks' _rm_test_img "$file"
 
     if [ $status != 0 ]; then
-        reason=$(echo "$output" | grep "$file:" | $SED -e "s#.*$file: *##")
+        reason=$(echo "$output" | grep "$file:" | sed -e "s#.*$file: *##")
         if [ -z "$reason" ]; then
             reason="Failed to create a LUKS image"
         fi
-- 
2.27.0

[PATCH v3 2/4] tests/qemu-iotests/meson.build: Improve the indentation

[Thomas Huth]

By using subdir_done(), we can get rid of one level of indentation
in this file. This will make it easier to add more conditions to
skip the iotests in future patches.

Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/qemu-iotests/meson.build | 61 ++++++++++++++++++----------------
 1 file changed, 32 insertions(+), 29 deletions(-)

diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 9747bb68a5..92f09251a6 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -1,30 +1,33 @@
-if have_tools and targetos != 'windows' and not get_option('gprof')
-  qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
-  qemu_iotests_env = {'PYTHON': python.full_path()}
-  qemu_iotests_formats = {
-    'qcow2': 'quick',
-    'raw': 'slow',
-    'qed': 'thorough',
-    'vmdk': 'thorough',
-    'vpc': 'thorough'
-  }
-
-  foreach k, v : emulators
-    if k.startswith('qemu-system-')
-      qemu_iotests_binaries += v
-    endif
-  endforeach
-  foreach format, speed: qemu_iotests_formats
-    if speed == 'quick'
-      suites = 'block'
-    else
-      suites = ['block-' + speed, speed]
-    endif
-    test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format],
-         depends: qemu_iotests_binaries, env: qemu_iotests_env,
-         protocol: 'tap',
-         suite: suites,
-         timeout: 0,
-         is_parallel: false)
-  endforeach
+if not have_tools or targetos == 'windows' or get_option('gprof')
+  subdir_done()
 endif
+
+qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
+qemu_iotests_env = {'PYTHON': python.full_path()}
+qemu_iotests_formats = {
+  'qcow2': 'quick',
+  'raw': 'slow',
+  'qed': 'thorough',
+  'vmdk': 'thorough',
+  'vpc': 'thorough'
+}
+
+foreach k, v : emulators
+  if k.startswith('qemu-system-')
+    qemu_iotests_binaries += v
+  endif
+endforeach
+
+foreach format, speed: qemu_iotests_formats
+  if speed == 'quick'
+    suites = 'block'
+  else
+    suites = ['block-' + speed, speed]
+  endif
+  test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format],
+       depends: qemu_iotests_binaries, env: qemu_iotests_env,
+       protocol: 'tap',
+       suite: suites,
+       timeout: 0,
+       is_parallel: false)
+endforeach
-- 
2.27.0

[PATCH v3 3/4] tests/qemu-iotests: Move the bash and sanitizer checks to meson.build

[Thomas Huth]

We want to get rid of check-block.sh in the long run, so let's move
the checks for the bash version and sanitizers from check-block.sh
into the meson.build file instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/check-block.sh           | 26 --------------------------
 tests/qemu-iotests/meson.build | 14 ++++++++++++++
 2 files changed, 14 insertions(+), 26 deletions(-)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index f59496396c..5de2c1ba0b 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -18,36 +18,10 @@ skip() {
     exit 0
 }
 
-# Disable tests with any sanitizer except for specific ones
-SANITIZE_FLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null )
-ALLOWED_SANITIZE_FLAGS="safe-stack cfi-icall"
-#Remove all occurrencies of allowed Sanitize flags
-for j in ${ALLOWED_SANITIZE_FLAGS}; do
-    TMP_FLAGS=${SANITIZE_FLAGS}
-    SANITIZE_FLAGS=""
-    for i in ${TMP_FLAGS}; do
-        if ! echo ${i} | grep -q "${j}" 2>/dev/null; then
-            SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}"
-        fi
-    done
-done
-if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then
-    # Have a sanitize flag that is not allowed, stop
-    skip "Sanitizers are enabled ==> Not running the qemu-iotests."
-fi
-
 if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
     skip "No qemu-system binary available ==> Not running the qemu-iotests."
 fi
 
-if ! command -v bash >/dev/null 2>&1 ; then
-    skip "bash not available ==> Not running the qemu-iotests."
-fi
-
-if LANG=C bash --version | grep -q 'GNU bash, version [123]' ; then
-    skip "bash version too old ==> Not running the qemu-iotests."
-fi
-
 cd tests/qemu-iotests
 
 # QEMU_CHECK_BLOCK_AUTO is used to disable some unstable sub-tests
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 92f09251a6..323a4acb6a 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -2,6 +2,20 @@ if not have_tools or targetos == 'windows' or get_option('gprof')
   subdir_done()
 endif
 
+foreach cflag: config_host['QEMU_CFLAGS'].split()
+  if cflag.startswith('-fsanitize') and \
+     not cflag.contains('safe-stack') and not cflag.contains('cfi-icall')
+    message('Sanitizers are enabled ==> Disabled the qemu-iotests.')
+    subdir_done()
+  endif
+endforeach
+
+bash = find_program('bash', required: false, version: '>= 4.0')
+if not bash.found()
+  message('bash >= v4.0 not available ==> Disabled the qemu-iotests.')
+  subdir_done()
+endif
+
 qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
 qemu_iotests_env = {'PYTHON': python.full_path()}
 qemu_iotests_formats = {
-- 
2.27.0

[PATCH v3 4/4] tests: Do not treat the iotests as separate meson test target anymore

[Thomas Huth]

If there is a failing iotest, the output is currently not logged to
the console anymore. To get this working again, we need to run the
meson test runner with --print-errorlogs (and without --verbose).
We could update the "meson test" call in tests/Makefile.include,
but actually it's nicer and easier if we simply do not treat the
iotests as separate test target anymore and integrate them along
with the other test suites.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 meson.build            | 6 +++---
 scripts/mtest2make.py  | 4 ----
 tests/Makefile.include | 9 +--------
 3 files changed, 4 insertions(+), 15 deletions(-)

diff --git a/meson.build b/meson.build
index 8df40bfac4..556479c785 100644
--- a/meson.build
+++ b/meson.build
@@ -3,9 +3,9 @@ project('qemu', ['c'], meson_version: '>=0.59.3',
                           'b_staticpic=false', 'stdsplit=false'],
         version: files('VERSION'))
 
-add_test_setup('quick', exclude_suites: ['block', 'slow', 'thorough'], is_default: true)
-add_test_setup('slow', exclude_suites: ['block', 'thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow'])
-add_test_setup('thorough', exclude_suites: ['block'], env: ['G_TEST_SLOW=1', 'SPEED=thorough'])
+add_test_setup('quick', exclude_suites: ['slow', 'thorough'], is_default: true)
+add_test_setup('slow', exclude_suites: ['thorough'], env: ['G_TEST_SLOW=1', 'SPEED=slow'])
+add_test_setup('thorough', env: ['G_TEST_SLOW=1', 'SPEED=thorough'])
 
 not_found = dependency('', required: false)
 keyval = import('keyval')
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 4d542e8aaa..304634b71e 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -101,10 +101,6 @@ def emit_suite(name, suite, prefix):
 testsuites = defaultdict(Suite)
 for test in introspect['tests']:
     process_tests(test, targets, testsuites)
-# HACK: check-block is a separate target so that it runs with --verbose;
-# only write the dependencies
-emit_suite_deps('block', testsuites['block'], 'check')
-del testsuites['block']
 emit_prolog(testsuites, 'check')
 for name, suite in testsuites.items():
     emit_suite(name, suite, 'check')
diff --git a/tests/Makefile.include b/tests/Makefile.include
index e7153c8e91..b89018cdcc 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -147,16 +147,9 @@ check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
 # Consolidated targets
 
-.PHONY: check-block check check-clean get-vm-images
+.PHONY: check check-clean get-vm-images
 check:
 
-ifneq ($(.check-block.deps),)
-check: check-block
-check-block: run-ninja
-	$(if $(MAKE.n),,+)$(MESON) test $(MTESTARGS) $(.mtestargs) --verbose \
-		--logbase iotestslog $(call .speed.$(SPEED), block block-slow block-thorough)
-endif
-
 check-build: run-ninja
 
 check-clean:
-- 
2.27.0

Re: [PATCH v3 0/4] Improve integration of iotests in the meson test harness

[Paolo Bonzini]

On 2/23/22 10:38, Thomas Huth wrote:
> This way, we can now finally get the output of failed
> tests on the console again (unless you're running meson test in verbose
> mode, where meson only puts this to the log file - for incomprehensible
> reasons),

It's a bug (introduced by yours truly, which perhaps makes it less 
incomprehensible) and there's an open pull request to fix it.

Paolo

Re: [PATCH v3 0/4] Improve integration of iotests in the meson test harness

[Hanna Reitz]

On 23.02.22 10:38, Thomas Huth wrote:
> Though "make check-block" is currently already run via the meson test
> runner, it still looks like an oddball in the output of "make check". It
> would be nicer if the iotests would show up like the other tests suites.
>
> My original plan was to add each iotests individually from meson.build,
> but I did not get that done reliably yet [*], so here's now a cut-down
> version to improve the situation at least a little bit: The first three
> patches are preparation for the clean-up (long-term goal is to get rid
> of check-block.sh, though we're not quite there yet), and the final
> patch adds the iotests not as separate test target in the meson test
> harness anymore. This way, we can now finally get the output of failed
> tests on the console again (unless you're running meson test in verbose
> mode, where meson only puts this to the log file - for incomprehensible
> reasons), so this should hopefully help to diagnose problems with the
> iotests in most cases more easily.
>
> [*] See v2 here:
>      https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg01942.html
>
> Thomas Huth (4):
>    tests/qemu-iotests: Rework the checks and spots using GNU sed
>    tests/qemu-iotests/meson.build: Improve the indentation
>    tests/qemu-iotests: Move the bash and sanitizer checks to meson.build
>    tests: Do not treat the iotests as separate meson test target anymore

What’s the status of this series?  I wonder why you split it apart, mainly.

Patch 1 was already merged, and I took patch 4 today.  So what about 
patches 2 and 3?  They look sensible to me, but is this series still 
relevant and fresh, considering you sent new versions of patches 1 and 4?

(And are there any other iotests patches from you that flew under my radar?)

Hanna

Re: [PATCH v3 3/4] tests/qemu-iotests: Move the bash and sanitizer checks to meson.build

[Hanna Reitz]

On 23.02.22 10:38, Thomas Huth wrote:
> We want to get rid of check-block.sh in the long run, so let's move
> the checks for the bash version and sanitizers from check-block.sh
> into the meson.build file instead.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/check-block.sh           | 26 --------------------------
>   tests/qemu-iotests/meson.build | 14 ++++++++++++++
>   2 files changed, 14 insertions(+), 26 deletions(-)

FWIW

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

Re: [PATCH v3 0/4] Improve integration of iotests in the meson test harness

[Thomas Huth]

On 21/03/2022 17.14, Hanna Reitz wrote:
> On 23.02.22 10:38, Thomas Huth wrote:
>> Though "make check-block" is currently already run via the meson test
>> runner, it still looks like an oddball in the output of "make check". It
>> would be nicer if the iotests would show up like the other tests suites.
>>
>> My original plan was to add each iotests individually from meson.build,
>> but I did not get that done reliably yet [*], so here's now a cut-down
>> version to improve the situation at least a little bit: The first three
>> patches are preparation for the clean-up (long-term goal is to get rid
>> of check-block.sh, though we're not quite there yet), and the final
>> patch adds the iotests not as separate test target in the meson test
>> harness anymore. This way, we can now finally get the output of failed
>> tests on the console again (unless you're running meson test in verbose
>> mode, where meson only puts this to the log file - for incomprehensible
>> reasons), so this should hopefully help to diagnose problems with the
>> iotests in most cases more easily.
>>
>> [*] See v2 here:
>>      https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg01942.html
>>
>> Thomas Huth (4):
>>    tests/qemu-iotests: Rework the checks and spots using GNU sed
>>    tests/qemu-iotests/meson.build: Improve the indentation
>>    tests/qemu-iotests: Move the bash and sanitizer checks to meson.build
>>    tests: Do not treat the iotests as separate meson test target anymore
> 
> What’s the status of this series?  I wonder why you split it apart, mainly.

I've mainly split the fourth patch apart since Paolo mentioned that the 
commit message should mention the meson bug (IIRC), and  since QEMU was 
entering soft-freeze, thus I doubt that a patch like "Move the bash and 
sanitizer checks to meson.build" is still acceptable at this point in time. 
The meson.build clean-up is rather something for 7.1 instead.

> Patch 1 was already merged, and I took patch 4 today.  So what about patches 
> 2 and 3?  They look sensible to me, but is this series still relevant and 
> fresh, considering you sent new versions of patches 1 and 4?

If you think they are still ok for 7.0, you can certainly also pick the 2nd 
and 3rd patch ... otherwise I'll respin them later for 7.1.

> (And are there any other iotests patches from you that flew under my radar?)

There's my "Supply a test plan in TAP mode" patch on the list, but 
apparently you already found it :-)

Thanks!

   Thomas

Re: [PATCH v3 0/4] Improve integration of iotests in the meson test harness

[Hanna Reitz]

On 21.03.22 18:26, Thomas Huth wrote:
> On 21/03/2022 17.14, Hanna Reitz wrote:
>> On 23.02.22 10:38, Thomas Huth wrote:
>>> Though "make check-block" is currently already run via the meson test
>>> runner, it still looks like an oddball in the output of "make 
>>> check". It
>>> would be nicer if the iotests would show up like the other tests 
>>> suites.
>>>
>>> My original plan was to add each iotests individually from meson.build,
>>> but I did not get that done reliably yet [*], so here's now a cut-down
>>> version to improve the situation at least a little bit: The first three
>>> patches are preparation for the clean-up (long-term goal is to get rid
>>> of check-block.sh, though we're not quite there yet), and the final
>>> patch adds the iotests not as separate test target in the meson test
>>> harness anymore. This way, we can now finally get the output of failed
>>> tests on the console again (unless you're running meson test in verbose
>>> mode, where meson only puts this to the log file - for incomprehensible
>>> reasons), so this should hopefully help to diagnose problems with the
>>> iotests in most cases more easily.
>>>
>>> [*] See v2 here:
>>> https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg01942.html
>>>
>>> Thomas Huth (4):
>>>    tests/qemu-iotests: Rework the checks and spots using GNU sed
>>>    tests/qemu-iotests/meson.build: Improve the indentation
>>>    tests/qemu-iotests: Move the bash and sanitizer checks to 
>>> meson.build
>>>    tests: Do not treat the iotests as separate meson test target 
>>> anymore
>>
>> What’s the status of this series?  I wonder why you split it apart, 
>> mainly.
>
> I've mainly split the fourth patch apart since Paolo mentioned that 
> the commit message should mention the meson bug (IIRC), and since QEMU 
> was entering soft-freeze, thus I doubt that a patch like "Move the 
> bash and sanitizer checks to meson.build" is still acceptable at this 
> point in time. The meson.build clean-up is rather something for 7.1 
> instead.
>
>> Patch 1 was already merged, and I took patch 4 today.  So what about 
>> patches 2 and 3?  They look sensible to me, but is this series still 
>> relevant and fresh, considering you sent new versions of patches 1 
>> and 4?
>
> If you think they are still ok for 7.0, you can certainly also pick 
> the 2nd and 3rd patch ... otherwise I'll respin them later for 7.1.

That sounds like you don’t really need to respin, so I’ve taken them to 
my block-next branch for 7.1:

https://gitlab.com/hreitz/qemu/-/commits/block-next

Thanks!

Hanna