[Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

This patchset is a little off the beaten path, but:

Add the ability to skip tests that have already been run whose external
dependencies have remained unchanged since the last test run.

This is useful for, say, re-running a test set to see if a failure was
a one-off or is reproducible by just re-running the last command, which
will skip all the tests that completed, but hitting all the ones that
failed or were not run.

It's also useful for testing patchsets for publication against bisect
problems by only running tests after each patch that have changed anything
worth testing -- for instance, a patch that only changes documentation
is not going to change any of the binaries, so we can skip a large swath
of tests here.

The new feature is, of course, optional and can be invoked with -ts.

===
v3:
===

- Fixed commit message for patch 1 (s/grep/awk/)
- Added a pinch of dependency group documentation to patch 2

===
v2:
===

- Replaced grep script for awk in patch 1
- Added qemu dependency in patch 2

==
For convenience, this branch is available at:
https://github.com/jnsnow/qemu.git branch ch3ck
https://github.com/jnsnow/qemu/tree/ch3ck

This version is tagged ch3ck-v3:
https://github.com/jnsnow/qemu/releases/tag/ch3ck-v3
==

John Snow (4):
  iotests: fix exclusion option
  iotests: Add dependency info to groups list
  iotests: add timestamp skip feature
  iotests: clarify help text

 tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common | 15 ++++++++--
 tests/qemu-iotests/group  | 70 +++++++++++++++++++++++++----------------------
 3 files changed, 105 insertions(+), 36 deletions(-)

-- 
2.1.0

[Qemu-devel] [PATCH v3 1/4] iotests: fix exclusion option

[John Snow]

If you are running out-of-tree, the -x option to exclude
a certain iotest is broken.

Replace porcelain usage of ls with a sturdier awk command.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 1030aaf..84b6f16 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -83,7 +83,8 @@ s/ .*//p
     elif $xgroup
     then
         # arg after -x
-        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
+        # Populate $tmp.list with all tests
+        awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null
         group_list=`sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
 s/ .*//p
 }'`
-- 
2.1.0

[Qemu-devel] [PATCH v3 2/4] iotests: Add dependency info to groups list

[John Snow]

Add a few external dependency groups to the iotests groups list,
such as "qemu", "nbd", and "scm".

This will assist us in knowing which tests need to be re-run
when those dependencies are updated, or by helping us to avoid
tests that we know are broken for a certain file.

qemu-img and qemu-io are so ubiquitous that it's not worth
amending this information to the groups file.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/group | 70 +++++++++++++++++++++++++-----------------------
 1 file changed, 37 insertions(+), 33 deletions(-)

diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 0b817ca..745021d 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -3,9 +3,13 @@
 # Defines test groups
 # - do not start group names with a digit
 #
-
+# External dependency groups:
+# qemu     Relies on the qemu binary.
+# nbd      Relies on the qemu-nbd binary.
+# scm      Relies on the socket scm helper binary.
 #
-# test-group association ... one line per test
+# Format, one line per test:
+# nnn [GROUP]...
 #
 001 rw auto quick
 002 rw auto quick
@@ -13,7 +17,7 @@
 004 rw auto quick
 005 img auto quick
 # 006 was removed, do not reuse
-007 snapshot auto
+007 snapshot auto qemu
 008 rw auto quick
 009 rw auto quick
 010 rw auto quick
@@ -34,9 +38,9 @@
 025 rw auto quick
 026 rw blkdbg auto
 027 rw auto quick
-028 rw backing auto quick
+028 rw backing auto quick qemu
 029 rw auto quick
-030 rw auto backing
+030 rw auto backing qemu
 031 rw auto quick
 032 rw auto quick
 033 rw auto quick
@@ -46,38 +50,38 @@
 037 rw auto backing quick
 038 rw auto backing quick
 039 rw auto quick
-040 rw auto
-041 rw auto backing
+040 rw auto qemu
+041 rw auto backing qemu
 042 rw auto quick
 043 rw auto backing
 044 rw auto
-045 rw auto quick
+045 rw auto quick scm qemu
 046 rw auto aio quick
 047 rw auto quick
 048 img auto quick
 049 rw auto
 050 rw auto backing quick
-051 rw auto
+051 rw auto qemu
 052 rw auto backing quick
 053 rw auto quick
 054 rw auto quick
-055 rw auto
-056 rw auto backing
-057 rw auto
-058 rw auto quick
+055 rw auto qemu
+056 rw auto backing qemu
+057 rw auto qemu
+058 rw auto quick nbd
 059 rw auto quick
 060 rw auto quick
 061 rw auto
 062 rw auto quick
 063 rw auto quick
 064 rw auto quick
-065 rw auto quick
+065 rw auto quick qemu
 066 rw auto quick
-067 rw auto quick
-068 rw auto quick
+067 rw auto quick qemu
+068 rw auto quick qemu
 069 rw auto quick
 070 rw auto quick
-071 rw auto quick
+071 rw auto quick qemu
 072 rw auto quick
 073 rw auto quick
 074 rw auto quick
@@ -87,46 +91,46 @@
 078 rw auto quick
 079 rw auto
 080 rw auto
-081 rw auto quick
+081 rw auto quick qemu
 082 rw auto quick
-083 rw auto
+083 rw auto nbd
 084 img auto quick
-085 rw auto
+085 rw auto qemu
 086 rw auto quick
-087 rw auto quick
+087 rw auto quick qemu
 088 rw auto quick
 089 rw auto quick
 090 rw auto quick
-091 rw auto
+091 rw auto qemu
 092 rw auto quick
-093 auto
-094 rw auto quick
-095 rw auto quick
+093 auto qemu
+094 rw auto quick qemu nbd
+095 rw auto quick qemu
 097 rw auto backing
 098 rw auto backing quick
-099 rw auto quick
+099 rw auto quick qemu
 100 rw auto quick
 101 rw auto quick
-102 rw auto quick
+102 rw auto quick qemu
 103 rw auto quick
 104 rw auto
 105 rw auto quick
 107 rw auto quick
 108 rw auto quick
-109 rw auto
+109 rw auto qemu
 110 rw auto backing quick
 111 rw auto quick
 112 rw auto
-113 rw auto quick
+113 rw auto quick nbd
 114 rw auto quick
 115 rw auto
 116 rw auto quick
 121 rw auto
 122 rw auto
-123 rw auto quick
-124 rw auto backing
+123 rw auto quick nbd
+124 rw auto backing qemu
 128 rw auto quick
-129 rw auto quick
-130 rw auto quick
+129 rw auto quick qemu
+130 rw auto quick qemu
 131 rw auto quick
 134 rw auto quick
-- 
2.1.0

[Qemu-devel] [PATCH v3 3/4] iotests: add timestamp skip feature

[John Snow]

Like a makefile, try to skip tests if we know they have already been
executed using the current set of external dependencies.

If a user passes the -ts option to ./check, if a test or its output
or its dependencies (qemu, qemu-nbd, qemu-io, qemu-img,
socket-scm-helper) have been modified, the test will be skipped.

This is primarily to speed up running bisectability tests for larger
series where each interim patch may or may not change all of the
dependencies, so some tests can be skipped.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common |  6 +++++
 2 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 1fa6319..800c7b0 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -141,6 +141,51 @@ _timestamp()
     echo -n " [$now]"
 }
 
+_okfile()
+{
+    echo "ts/${IMGPROTO}/${IMGFMT}/${1}.ok"
+}
+
+# _hasdep <seq> <dep>
+_hasdep()
+{
+    grep -q "^${1}.*${2}" "${source_iotests}/group"
+}
+
+# _skippable <seq>
+_skippable()
+{
+    tsfile=$(_okfile "${1}")
+    deps=(${QEMU_IMG} \
+              ${QEMU_IO} \
+              "${source_iotests}/${1}" \
+              "${source_iotests}/${1}.out" \
+              "${source_iotests}/common" \
+              "${source_iotests}/common.rc" \
+              "${source_iotests}/common.env" \
+              "${source_iotests}/common.config")
+
+    if _hasdep "${1}" "scm"; then
+        deps+=("${SOCKET_SCM_HELPER}")
+    fi
+
+    if _hasdep "${1}" "nbd" || [ "$IMGPROTO" = "nbd" ]; then
+        deps+=("${QEMU_NBD}")
+    fi
+
+    if _hasdep "${1}" "qemu"; then
+        deps+=("${QEMU}")
+    fi
+
+    if [ ! -e "${tsfile}" ]; then return 1; fi
+    for dep in "${deps[@]}"; do
+        if [ "${tsfile}" -ot "${dep}" ]; then return 1; fi
+    done
+
+    # Looks skippable, .ok file is newer than all external dependencies.
+    return 0
+}
+
 _wrapup()
 {
     # for hangcheck ...
@@ -268,6 +313,10 @@ do
         echo " - expunged"
         rm -f $seq.out.bad
         echo "/^$seq\$/d" >>$tmp.expunged
+
+    elif [ "${ts_skip}" == "true" ] && _skippable "${seq}"; then
+        echo "         [skip] (Results are recent)"
+
     elif [ ! -f "$source_iotests/$seq" ]
     then
         echo " - no such test?"
@@ -275,7 +324,8 @@ do
     else
         # really going to try and run this one
         #
-        rm -f $seq.out.bad
+        tsfile="$(_okfile ${seq})"
+        rm -f $seq.out.bad "${tsfile}"
         lasttime=`sed -n -e "/^$seq /s/.* //p" <check.time`
         if [ "X$lasttime" != X ]; then
                 echo -n " ${lasttime}s ..."
@@ -346,6 +396,10 @@ do
                     then
                         :
                     else
+                        if [ ! -d "$(dirname ${tsfile})" ]; then
+                            mkdir -p "$(dirname ${tsfile})"
+                        fi
+                        touch "${tsfile}"
                         echo "$seq `expr $stop - $start`" >>$tmp.time
                     fi
                 else
diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 84b6f16..395c28c 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -163,6 +163,8 @@ check options
     -T                  output timestamps
     -r                  randomize test order
     -c mode             cache mode
+    -ts                 timestamp skip: skip tests when already ran and
+                                   no dependencies have changed on disk.
 
 testlist options
     -g group[,group...]        include tests from these groups
@@ -321,6 +323,10 @@ testlist options
             xpand=false
             ;;
 
+        -ts)
+            ts_skip=true
+            xpand=false
+            ;;
         -v)
             verbose=true
             xpand=false
-- 
2.1.0

[Qemu-devel] [PATCH v3 4/4] iotests: clarify help text

[John Snow]

Split the help text to highlight the groups of options
a little better, carving out a clear "format" and
"protocols" section.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 395c28c..edfc434 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -136,7 +136,7 @@ common options
     -v                  verbose
     -d                  debug
 
-check options
+image format options
     -raw                test raw (default)
     -bochs              test bochs
     -cloop              test cloop
@@ -148,6 +148,8 @@ check options
     -vpc                test vpc
     -vhdx               test vhdx
     -vmdk               test vmdk
+
+image protocol options
     -file               test file (default)
     -rbd                test rbd
     -sheepdog           test sheepdog
@@ -155,6 +157,8 @@ check options
     -ssh                test ssh
     -nfs                test nfs
     -archipelago        test archipelago
+
+other options
     -xdiff              graphical mode diff
     -nocache            use O_DIRECT on backing file
     -misalign           misalign memory allocations
-- 
2.1.0

Re: [Qemu-devel] [PATCH v3 1/4] iotests: fix exclusion option

[Fam Zheng]

On Tue, 06/02 15:18, John Snow wrote:
> If you are running out-of-tree, the -x option to exclude
> a certain iotest is broken.
> 
> Replace porcelain usage of ls with a sturdier awk command.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---

Reviewed-by: Fam Zheng <famz@redhat.com>

>  tests/qemu-iotests/common | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
> index 1030aaf..84b6f16 100644
> --- a/tests/qemu-iotests/common
> +++ b/tests/qemu-iotests/common
> @@ -83,7 +83,8 @@ s/ .*//p
>      elif $xgroup
>      then
>          # arg after -x
> -        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
> +        # Populate $tmp.list with all tests
> +        awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null
>          group_list=`sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
>  s/ .*//p
>  }'`
> -- 
> 2.1.0
> 

Re: [Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

Ping

On 06/02/2015 03:18 PM, John Snow wrote:
> This patchset is a little off the beaten path, but:
> 
> Add the ability to skip tests that have already been run whose external
> dependencies have remained unchanged since the last test run.
> 
> This is useful for, say, re-running a test set to see if a failure was
> a one-off or is reproducible by just re-running the last command, which
> will skip all the tests that completed, but hitting all the ones that
> failed or were not run.
> 
> It's also useful for testing patchsets for publication against bisect
> problems by only running tests after each patch that have changed anything
> worth testing -- for instance, a patch that only changes documentation
> is not going to change any of the binaries, so we can skip a large swath
> of tests here.
> 
> The new feature is, of course, optional and can be invoked with -ts.
> 
> ===
> v3:
> ===
> 
> - Fixed commit message for patch 1 (s/grep/awk/)
> - Added a pinch of dependency group documentation to patch 2
> 
> ===
> v2:
> ===
> 
> - Replaced grep script for awk in patch 1
> - Added qemu dependency in patch 2
> 
> ==
> For convenience, this branch is available at:
> https://github.com/jnsnow/qemu.git branch ch3ck
> https://github.com/jnsnow/qemu/tree/ch3ck
> 
> This version is tagged ch3ck-v3:
> https://github.com/jnsnow/qemu/releases/tag/ch3ck-v3
> ==
> 
> John Snow (4):
>   iotests: fix exclusion option
>   iotests: Add dependency info to groups list
>   iotests: add timestamp skip feature
>   iotests: clarify help text
> 
>  tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++-
>  tests/qemu-iotests/common | 15 ++++++++--
>  tests/qemu-iotests/group  | 70 +++++++++++++++++++++++++----------------------
>  3 files changed, 105 insertions(+), 36 deletions(-)
> 

-- 
—js

Re: [Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

Hi Kevin, ping?

Will probably need to respin to update the deps (for new tests) by now,
but are you willing to take this series?

--js

On 06/02/2015 03:18 PM, John Snow wrote:
> This patchset is a little off the beaten path, but:
> 
> Add the ability to skip tests that have already been run whose external
> dependencies have remained unchanged since the last test run.
> 
> This is useful for, say, re-running a test set to see if a failure was
> a one-off or is reproducible by just re-running the last command, which
> will skip all the tests that completed, but hitting all the ones that
> failed or were not run.
> 
> It's also useful for testing patchsets for publication against bisect
> problems by only running tests after each patch that have changed anything
> worth testing -- for instance, a patch that only changes documentation
> is not going to change any of the binaries, so we can skip a large swath
> of tests here.
> 
> The new feature is, of course, optional and can be invoked with -ts.
> 
> ===
> v3:
> ===
> 
> - Fixed commit message for patch 1 (s/grep/awk/)
> - Added a pinch of dependency group documentation to patch 2
> 
> ===
> v2:
> ===
> 
> - Replaced grep script for awk in patch 1
> - Added qemu dependency in patch 2
> 
> ==
> For convenience, this branch is available at:
> https://github.com/jnsnow/qemu.git branch ch3ck
> https://github.com/jnsnow/qemu/tree/ch3ck
> 
> This version is tagged ch3ck-v3:
> https://github.com/jnsnow/qemu/releases/tag/ch3ck-v3
> ==
> 
> John Snow (4):
>   iotests: fix exclusion option
>   iotests: Add dependency info to groups list
>   iotests: add timestamp skip feature
>   iotests: clarify help text
> 
>  tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++-
>  tests/qemu-iotests/common | 15 ++++++++--
>  tests/qemu-iotests/group  | 70 +++++++++++++++++++++++++----------------------
>  3 files changed, 105 insertions(+), 36 deletions(-)
> 

Re: [Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

On Tue, Jun 23, 2015 at 12:17:40PM -0400, John Snow wrote:
> Hi Kevin, ping?
> 
> Will probably need to respin to update the deps (for new tests) by now,
> but are you willing to take this series?

Kevin is on vacation, this series is unlikely to make QEMU 2.4.

I suggest rebasing around hard freeze time so it can be merged into the
Kevin's -next branch.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v3 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 06/25/2015 08:38 AM, Stefan Hajnoczi wrote:
> On Tue, Jun 23, 2015 at 12:17:40PM -0400, John Snow wrote:
>> Hi Kevin, ping?
>> 
>> Will probably need to respin to update the deps (for new tests)
>> by now, but are you willing to take this series?
> 
> Kevin is on vacation, this series is unlikely to make QEMU 2.4.
> 
> I suggest rebasing around hard freeze time so it can be merged into
> the Kevin's -next branch.
> 
> Stefan


Patches 1 and 4 are non time sensitive improvements (don't need to be
rebased) if you want to pull those.

I'll send 2/3 again during hard freeze for 2.5.

- --js
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVjC1tAAoJEH3vgQaq/DkOJ10P+gNPhXVIeOY8xU8WJPe1tMLA
ufE5q1v1FDPeFBNDLT9s2FfPyhmy+5oQUjLWYQpIraaM7CfMU++A39zyoJXrNkIp
IRtu2NoJ6qqBYxj2oeBx1/qs09cNVCj/MBKZ8OS04pda/V6tcbA/1PF7VJgKSV2t
hxNRd6RtBSiDGib5K7PM0J+Xl5EgcChl80v+DyQQplX6smsS9Vsbw30jACURYSPJ
TnOjV7gCIgpVxnmPCHZ/uHQbvhPKpX8nsIZToLIGthiOKePZD6EXpyQ28NP50xcC
gsjLqwdj+2//mH88pgFqg/ptW8KMmOZHod8lyR/tyJv2dkCbnX67ZHxStZAjk0PJ
NjOaOZzh+U0sbhXW+uKYtiNTyKCG24hATtaCpUKdfoj8KtqDTJyLn87y/097pDL2
+rRCAEB4kzAmRwKXNxy62qd16HWDCi3sESIWbuUbdl3MWxpvAtYULg5gAzl8ybaJ
IzEjlU/EnmuumBSIIUERtoJAl9LVsqAla8tBd3xy+AHwuXyYu6HkgjzdlRnsWTxi
ZA6kZyAjxeXEQHxMqj41wjTA3YuTxOSfDOrksA6yO5qEP+nuzcdPLI3eviXnZCdb
hiVyMBjpAr1AG/XfoCKrpZVc4AAYXMfFmxyNH6BZd+pOyZdIKsIp52SudFkqSCbT
wi2QVhc7kxxwrDt0q3Y5
=qlfp
-----END PGP SIGNATURE-----