[Qemu-devel] [PATCH 0/4] iotests: skip tests with unchanged dependencies

[Qemu-devel] [PATCH 0/4] iotests: skip tests with unchanged dependencies

[John Snow]

This patchset is a little off the beaten path, but:

Add the ability to skip tests that have already been run whose external
dependencies have remained unchanged since the last test run.

This is useful for, say, re-running a test set to see if a failure was
a one-off or is reproducible by just re-running the last command, which
will skip all the tests that completed, but hitting all the ones that
failed or were not run.

It's also useful for testing patchsets for publication against bisect
problems by only running tests after each patch that have changed anything
worth testing -- for instance, a patch that only changes documentation
is not going to change any of the binaries, so we can skip a large swath
of tests here.

The new feature is, of course, optional and can be invoked with -ts.

==
For convenience, this branch is available at:
https://github.com/jnsnow/qemu.git branch ch3ck
https://github.com/jnsnow/qemu/tree/ch3ck

This version is tagged ch3ck-v1:
https://github.com/jnsnow/qemu/releases/tag/ch3ck-v1
==

John Snow (4):
  iotests: fix exclusion option
  iotests: Add dependency info to groups list
  iotests: add timestamp skip feature
  iotests: clarify help text

 tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common | 15 ++++++++++--
 tests/qemu-iotests/group  | 60 +++++++++++++++++++++++------------------------
 3 files changed, 98 insertions(+), 33 deletions(-)

-- 
2.1.0

[Qemu-devel] [PATCH 1/4] iotests: fix exclusion option

[John Snow]

If you are running out-of-tree, the -x option to exclude
a certain iotest is broken.

Replace porcelain usage of ls with a sturdier grep command.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 1e556bb..c4d0742 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -82,7 +82,8 @@ s/ .*//p
     elif $xgroup
     then
         # arg after -x
-        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
+        # Populate $tmp.list with all tests
+        grep -oh -E "^([0-9]{3,})" "${source_iotests}/group" > $tmp.list 2>/dev/null
         group_list=`sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
 s/ .*//p
 }'`
-- 
2.1.0

[Qemu-devel] [PATCH 2/4] iotests: Add dependency info to groups list

[John Snow]

Add a few external dependency groups to the iotests groups list,
such as "qemu", "nbd", and "scm".

This will assist us in knowing which tests need to be re-run
when those dependencies are updated, or by helping us to avoid
tests that we know are broken for a certain file.

qemu-img and qemu-io are so ubiquitous that it's not worth
amending this information to the groups file.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/group | 60 ++++++++++++++++++++++++------------------------
 1 file changed, 30 insertions(+), 30 deletions(-)

diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 34b16cb..32f15d8 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -34,9 +34,9 @@
 025 rw auto quick
 026 rw blkdbg auto
 027 rw auto quick
-028 rw backing auto quick
+028 rw backing auto quick qemu
 029 rw auto quick
-030 rw auto backing
+030 rw auto backing qemu
 031 rw auto quick
 032 rw auto quick
 033 rw auto quick
@@ -46,38 +46,38 @@
 037 rw auto backing quick
 038 rw auto backing quick
 039 rw auto quick
-040 rw auto
-041 rw auto backing
+040 rw auto qemu
+041 rw auto backing qemu
 042 rw auto quick
 043 rw auto backing
 044 rw auto
-045 rw auto quick
+045 rw auto quick scm qemu
 046 rw auto aio quick
 047 rw auto quick
 048 img auto quick
 049 rw auto
 050 rw auto backing quick
-051 rw auto
+051 rw auto qemu
 052 rw auto backing quick
 053 rw auto quick
 054 rw auto quick
-055 rw auto
-056 rw auto backing
-057 rw auto
-058 rw auto quick
+055 rw auto qemu
+056 rw auto backing qemu
+057 rw auto qemu
+058 rw auto quick nbd
 059 rw auto quick
 060 rw auto quick
 061 rw auto
 062 rw auto quick
 063 rw auto quick
 064 rw auto quick
-065 rw auto quick
+065 rw auto quick qemu
 066 rw auto quick
-067 rw auto quick
-068 rw auto quick
+067 rw auto quick qemu
+068 rw auto quick qemu
 069 rw auto quick
 070 rw auto quick
-071 rw auto quick
+071 rw auto quick qemu
 072 rw auto quick
 073 rw auto quick
 074 rw auto quick
@@ -87,45 +87,45 @@
 078 rw auto quick
 079 rw auto
 080 rw auto
-081 rw auto quick
+081 rw auto quick qemu
 082 rw auto quick
-083 rw auto
+083 rw auto nbd
 084 img auto quick
-085 rw auto
+085 rw auto qemu
 086 rw auto quick
-087 rw auto quick
+087 rw auto quick qemu
 088 rw auto quick
 089 rw auto quick
 090 rw auto quick
-091 rw auto
+091 rw auto qemu
 092 rw auto quick
-093 auto
-094 rw auto quick
-095 rw auto quick
+093 auto qemu
+094 rw auto quick qemu nbd
+095 rw auto quick qemu
 097 rw auto backing
 098 rw auto backing quick
-099 rw auto quick
+099 rw auto quick qemu
 100 rw auto quick
 101 rw auto quick
-102 rw auto quick
+102 rw auto quick qemu
 103 rw auto quick
 104 rw auto
 105 rw auto quick
 107 rw auto quick
 108 rw auto quick
-109 rw auto
+109 rw auto qemu
 110 rw auto backing quick
 111 rw auto quick
 112 rw auto
-113 rw auto quick
+113 rw auto quick nbd
 114 rw auto quick
 115 rw auto
 116 rw auto quick
 121 rw auto
 122 rw auto
-123 rw auto quick
-124 rw auto backing
+123 rw auto quick nbd
+124 rw auto backing qemu
 128 rw auto quick
-129 rw auto quick
-130 rw auto quick
+129 rw auto quick qemu
+130 rw auto quick qemu
 131 rw auto quick
-- 
2.1.0

[Qemu-devel] [PATCH 3/4] iotests: add timestamp skip feature

[John Snow]

Like a makefile, try to skip tests if we know they have already been
executed using the current set of external dependencies.

If a user passes the -ts option to ./check, if a test or its output
or its dependencies (qemu, qemu-nbd, qemu-io, qemu-img,
socket-scm-helper) have been modified, the test will be skipped.

This is primarily to speed up running bisectability tests for larger
series where each interim patch may or may not change all of the
dependencies, so some tests can be skipped.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/common |  6 +++++
 2 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index baeae80..0db40e2 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -141,6 +141,51 @@ _timestamp()
     echo -n " [$now]"
 }
 
+_okfile()
+{
+    echo "ts/${IMGPROTO}/${IMGFMT}/${1}.ok"
+}
+
+# _hasdep <seq> <dep>
+_hasdep()
+{
+    grep -q "^${1}.*${2}" "${source_iotests}/group"
+}
+
+# _skippable <seq>
+_skippable()
+{
+    tsfile=$(_okfile "${1}")
+    deps=(${QEMU_IMG} \
+              ${QEMU_IO} \
+              "${source_iotests}/${1}" \
+              "${source_iotests}/${1}.out" \
+              "${source_iotests}/common" \
+              "${source_iotests}/common.rc" \
+              "${source_iotests}/common.env" \
+              "${source_iotests}/common.config")
+
+    if _hasdep "${1}" "scm"; then
+        deps+=("${SOCKET_SCM_HELPER}")
+    fi
+
+    if _hasdep "${1}" "nbd" || [ "$IMGPROTO" = "nbd" ]; then
+        deps+=("${QEMU_NBD}")
+    fi
+
+    if _hasdep "${1}" "qemu"; then
+        deps+=("${QEMU}")
+    fi
+
+    if [ ! -e "${tsfile}" ]; then return 1; fi
+    for dep in "${deps[@]}"; do
+        if [ "${tsfile}" -ot "${dep}" ]; then return 1; fi
+    done
+
+    # Looks skippable, .ok file is newer than all external dependencies.
+    return 0
+}
+
 _wrapup()
 {
     # for hangcheck ...
@@ -268,6 +313,10 @@ do
         echo " - expunged"
         rm -f $seq.out.bad
         echo "/^$seq\$/d" >>$tmp.expunged
+
+    elif [ "${ts_skip}" == "true" ] && _skippable "${seq}"; then
+        echo "         [skip] (Results are recent)"
+
     elif [ ! -f "$source_iotests/$seq" ]
     then
         echo " - no such test?"
@@ -275,7 +324,8 @@ do
     else
         # really going to try and run this one
         #
-        rm -f $seq.out.bad
+        tsfile="$(_okfile ${seq})"
+        rm -f $seq.out.bad "${tsfile}"
         lasttime=`sed -n -e "/^$seq /s/.* //p" <check.time`
         if [ "X$lasttime" != X ]; then
                 echo -n " ${lasttime}s ..."
@@ -340,6 +390,10 @@ do
                     then
                         :
                     else
+                        if [ ! -d "$(dirname ${tsfile})" ]; then
+                            mkdir -p "$(dirname ${tsfile})"
+                        fi
+                        touch "${tsfile}"
                         echo "$seq `expr $stop - $start`" >>$tmp.time
                     fi
                 else
diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index c4d0742..b3d869e 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -161,6 +161,8 @@ check options
     -T                  output timestamps
     -r                  randomize test order
     -c mode             cache mode
+    -ts                 timestamp skip: skip tests when already ran and
+                                   no dependencies have changed on disk.
 
 testlist options
     -g group[,group...]        include tests from these groups
@@ -319,6 +321,10 @@ testlist options
             xpand=false
             ;;
 
+        -ts)
+            ts_skip=true
+            xpand=false
+            ;;
         -v)
             verbose=true
             xpand=false
-- 
2.1.0

[Qemu-devel] [PATCH 4/4] iotests: clarify help text

[John Snow]

Split the help text to highlight the groups of options
a little better, carving out a clear "format" and
"protocols" section.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/common | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index b3d869e..9456aa0 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -134,7 +134,7 @@ s/ .*//p
 common options
     -v                  verbose
 
-check options
+image format options
     -raw                test raw (default)
     -bochs              test bochs
     -cloop              test cloop
@@ -146,6 +146,8 @@ check options
     -vpc                test vpc
     -vhdx               test vhdx
     -vmdk               test vmdk
+
+image protocol options
     -file               test file (default)
     -rbd                test rbd
     -sheepdog           test sheepdog
@@ -153,6 +155,8 @@ check options
     -ssh                test ssh
     -nfs                test nfs
     -archipelago        test archipelago
+
+other options
     -xdiff              graphical mode diff
     -nocache            use O_DIRECT on backing file
     -misalign           misalign memory allocations
-- 
2.1.0

Re: [Qemu-devel] [PATCH 1/4] iotests: fix exclusion option

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On 05/22/2015 02:17 PM, John Snow wrote:
> If you are running out-of-tree, the -x option to exclude
> a certain iotest is broken.
> 
> Replace porcelain usage of ls with a sturdier grep command.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/common | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
> index 1e556bb..c4d0742 100644
> --- a/tests/qemu-iotests/common
> +++ b/tests/qemu-iotests/common
> @@ -82,7 +82,8 @@ s/ .*//p
>      elif $xgroup
>      then
>          # arg after -x
> -        [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] >$tmp.list 2>/dev/null
> +        # Populate $tmp.list with all tests
> +        grep -oh -E "^([0-9]{3,})" "${source_iotests}/group" > $tmp.list 2>/dev/null

'grep -oh' is a GNU-ism; is it going to cause grief on BSD platforms?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH 1/4] iotests: fix exclusion option

[John Snow]

On 05/22/2015 04:31 PM, Eric Blake wrote:
> On 05/22/2015 02:17 PM, John Snow wrote:
>> If you are running out-of-tree, the -x option to exclude a
>> certain iotest is broken.
>> 
>> Replace porcelain usage of ls with a sturdier grep command.
>> 
>> Signed-off-by: John Snow <jsnow@redhat.com> --- 
>> tests/qemu-iotests/common | 3 ++- 1 file changed, 2
>> insertions(+), 1 deletion(-)
>> 
>> diff --git a/tests/qemu-iotests/common
>> b/tests/qemu-iotests/common index 1e556bb..c4d0742 100644 ---
>> a/tests/qemu-iotests/common +++ b/tests/qemu-iotests/common @@
>> -82,7 +82,8 @@ s/ .*//p elif $xgroup then # arg after -x -
>> [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9]
>> >$tmp.list 2>/dev/null +        # Populate $tmp.list with all
>> tests +        grep -oh -E "^([0-9]{3,})"
>> "${source_iotests}/group" > $tmp.list 2>/dev/null
> 
> 'grep -oh' is a GNU-ism; is it going to cause grief on BSD
> platforms?
> 

It might...  I'll look at other options. awk is probably fine here for
printing column 1.

--js

Re: [Qemu-devel] [PATCH 1/4] iotests: fix exclusion option

[John Snow]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 05/22/2015 04:31 PM, Eric Blake wrote:
> On 05/22/2015 02:17 PM, John Snow wrote:
>> If you are running out-of-tree, the -x option to exclude a
>> certain iotest is broken.
>> 
>> Replace porcelain usage of ls with a sturdier grep command.
>> 
>> Signed-off-by: John Snow <jsnow@redhat.com> --- 
>> tests/qemu-iotests/common | 3 ++- 1 file changed, 2
>> insertions(+), 1 deletion(-)
>> 
>> diff --git a/tests/qemu-iotests/common
>> b/tests/qemu-iotests/common index 1e556bb..c4d0742 100644 ---
>> a/tests/qemu-iotests/common +++ b/tests/qemu-iotests/common @@
>> -82,7 +82,8 @@ s/ .*//p elif $xgroup then # arg after -x -
>> [ ! -s $tmp.list ] && ls [0-9][0-9][0-9] [0-9][0-9][0-9][0-9]
>> >$tmp.list 2>/dev/null +        # Populate $tmp.list with all
>> tests +        grep -oh -E "^([0-9]{3,})"
>> "${source_iotests}/group" > $tmp.list 2>/dev/null
> 
> 'grep -oh' is a GNU-ism; is it going to cause grief on BSD
> platforms?
> 

I'll fix this in v2 (but I'll wait for more reviews), but I will
replace this line with:

awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group"	> $tmp.list
2>/dev/null

Which should be more friendly, I hope.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVX7JBAAoJEH3vgQaq/DkOhckP/0bvrjpbSnmjCnyk3Ku/PhAB
pD7jI87UiTUP3OYvR9cFtp3kukWYntfgIGLnNPq5QVL0ciaL52NYqt7yt5FaZEZh
hTpNYDb4HOeO/WuWYPxRjC/uxnhp+v6KlQOrQZ040wGrB6FbH+JJTrB53JtoDRTZ
1k4dn3EeK3wE0F6BRtr977+n+LCe4HMSwhL6qR6+k5HszX960Zpjc7YsvG4JIusm
arBAcQYglXhAV976QXGf/2TiZmIeChU2k1LAErueD7cpMIm1BcWoL+TYf1cFrKWU
n1hFjPaAZpw5WTnIYeLdPD95R2pIxcIB7huApw6YRCz+EZe5G6C1015D10mczB4K
Yz+p2t52IeCYp4qlwWPZuzOstHmyK6LsIRs4upiObw9eKRyt2NRi5GBXmqfTKSDY
nqoHvrFhyrTmUTfsqPmtGvA/dg7m8JAoh2kVxKXvmXE0Hq4mj/DqNwTGRFK933ux
DAtp5JY691aopj8Th1dxT7QerYqNB0WXQJrQRgWRmnPQgXx1o4CivNiTZ852f3Ux
EZHfsx/KN8Uc0fmbdNhOAxzwwlFaCSL100pR0q7vv7ddczqdjaVVqf25IrdrNZ0n
9oZNYe8Q4PrGSjz3Mmo+DgWH0Znn9p+lzHL8TEFu+OtRzSbHEAbJeSxN0RoLkOl8
M4O9Yh8fn1JnIWfnKUUZ
=8NHL
-----END PGP SIGNATURE-----

Re: [Qemu-devel] [PATCH 4/4] iotests: clarify help text

[Fam Zheng]

On Fri, 05/22 16:17, John Snow wrote:
> Split the help text to highlight the groups of options
> a little better, carving out a clear "format" and
> "protocols" section.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

> ---
>  tests/qemu-iotests/common | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
> index b3d869e..9456aa0 100644
> --- a/tests/qemu-iotests/common
> +++ b/tests/qemu-iotests/common
> @@ -134,7 +134,7 @@ s/ .*//p
>  common options
>      -v                  verbose
>  
> -check options
> +image format options
>      -raw                test raw (default)
>      -bochs              test bochs
>      -cloop              test cloop
> @@ -146,6 +146,8 @@ check options
>      -vpc                test vpc
>      -vhdx               test vhdx
>      -vmdk               test vmdk
> +
> +image protocol options
>      -file               test file (default)
>      -rbd                test rbd
>      -sheepdog           test sheepdog
> @@ -153,6 +155,8 @@ check options
>      -ssh                test ssh
>      -nfs                test nfs
>      -archipelago        test archipelago
> +
> +other options
>      -xdiff              graphical mode diff
>      -nocache            use O_DIRECT on backing file
>      -misalign           misalign memory allocations
> -- 
> 2.1.0
> 

Re: [Qemu-devel] [PATCH 3/4] iotests: add timestamp skip feature

[Fam Zheng]

On Fri, 05/22 16:17, John Snow wrote:
> Like a makefile, try to skip tests if we know they have already been
> executed using the current set of external dependencies.
> 
> If a user passes the -ts option to ./check, if a test or its output
> or its dependencies (qemu, qemu-nbd, qemu-io, qemu-img,
> socket-scm-helper) have been modified, the test will be skipped.
> 
> This is primarily to speed up running bisectability tests for larger
> series where each interim patch may or may not change all of the
> dependencies, so some tests can be skipped.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

> ---
>  tests/qemu-iotests/check  | 56 ++++++++++++++++++++++++++++++++++++++++++++++-
>  tests/qemu-iotests/common |  6 +++++
>  2 files changed, 61 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
> index baeae80..0db40e2 100755
> --- a/tests/qemu-iotests/check
> +++ b/tests/qemu-iotests/check
> @@ -141,6 +141,51 @@ _timestamp()
>      echo -n " [$now]"
>  }
>  
> +_okfile()
> +{
> +    echo "ts/${IMGPROTO}/${IMGFMT}/${1}.ok"
> +}
> +
> +# _hasdep <seq> <dep>
> +_hasdep()
> +{
> +    grep -q "^${1}.*${2}" "${source_iotests}/group"
> +}
> +
> +# _skippable <seq>
> +_skippable()
> +{
> +    tsfile=$(_okfile "${1}")
> +    deps=(${QEMU_IMG} \
> +              ${QEMU_IO} \
> +              "${source_iotests}/${1}" \
> +              "${source_iotests}/${1}.out" \
> +              "${source_iotests}/common" \
> +              "${source_iotests}/common.rc" \
> +              "${source_iotests}/common.env" \
> +              "${source_iotests}/common.config")
> +
> +    if _hasdep "${1}" "scm"; then
> +        deps+=("${SOCKET_SCM_HELPER}")
> +    fi
> +
> +    if _hasdep "${1}" "nbd" || [ "$IMGPROTO" = "nbd" ]; then
> +        deps+=("${QEMU_NBD}")
> +    fi
> +
> +    if _hasdep "${1}" "qemu"; then
> +        deps+=("${QEMU}")
> +    fi
> +
> +    if [ ! -e "${tsfile}" ]; then return 1; fi
> +    for dep in "${deps[@]}"; do
> +        if [ "${tsfile}" -ot "${dep}" ]; then return 1; fi
> +    done
> +
> +    # Looks skippable, .ok file is newer than all external dependencies.
> +    return 0
> +}
> +
>  _wrapup()
>  {
>      # for hangcheck ...
> @@ -268,6 +313,10 @@ do
>          echo " - expunged"
>          rm -f $seq.out.bad
>          echo "/^$seq\$/d" >>$tmp.expunged
> +
> +    elif [ "${ts_skip}" == "true" ] && _skippable "${seq}"; then
> +        echo "         [skip] (Results are recent)"
> +
>      elif [ ! -f "$source_iotests/$seq" ]
>      then
>          echo " - no such test?"
> @@ -275,7 +324,8 @@ do
>      else
>          # really going to try and run this one
>          #
> -        rm -f $seq.out.bad
> +        tsfile="$(_okfile ${seq})"
> +        rm -f $seq.out.bad "${tsfile}"
>          lasttime=`sed -n -e "/^$seq /s/.* //p" <check.time`
>          if [ "X$lasttime" != X ]; then
>                  echo -n " ${lasttime}s ..."
> @@ -340,6 +390,10 @@ do
>                      then
>                          :
>                      else
> +                        if [ ! -d "$(dirname ${tsfile})" ]; then
> +                            mkdir -p "$(dirname ${tsfile})"
> +                        fi
> +                        touch "${tsfile}"
>                          echo "$seq `expr $stop - $start`" >>$tmp.time
>                      fi
>                  else
> diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
> index c4d0742..b3d869e 100644
> --- a/tests/qemu-iotests/common
> +++ b/tests/qemu-iotests/common
> @@ -161,6 +161,8 @@ check options
>      -T                  output timestamps
>      -r                  randomize test order
>      -c mode             cache mode
> +    -ts                 timestamp skip: skip tests when already ran and
> +                                   no dependencies have changed on disk.
>  
>  testlist options
>      -g group[,group...]        include tests from these groups
> @@ -319,6 +321,10 @@ testlist options
>              xpand=false
>              ;;
>  
> +        -ts)
> +            ts_skip=true
> +            xpand=false
> +            ;;
>          -v)
>              verbose=true
>              xpand=false
> -- 
> 2.1.0
> 

Re: [Qemu-devel] [PATCH 2/4] iotests: Add dependency info to groups list

[Fam Zheng]

On Fri, 05/22 16:17, John Snow wrote:
> Add a few external dependency groups to the iotests groups list,
> such as "qemu", "nbd", and "scm".
> 
> This will assist us in knowing which tests need to be re-run
> when those dependencies are updated, or by helping us to avoid
> tests that we know are broken for a certain file.
> 
> qemu-img and qemu-io are so ubiquitous that it's not worth
> amending this information to the groups file.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>

007 uses $QEMU too ;-) Others looks good.

Reviewed-by: Fam Zheng <famz@redhat.com>

> ---
>  tests/qemu-iotests/group | 60 ++++++++++++++++++++++++------------------------
>  1 file changed, 30 insertions(+), 30 deletions(-)
> 
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 34b16cb..32f15d8 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -34,9 +34,9 @@
>  025 rw auto quick
>  026 rw blkdbg auto
>  027 rw auto quick
> -028 rw backing auto quick
> +028 rw backing auto quick qemu
>  029 rw auto quick
> -030 rw auto backing
> +030 rw auto backing qemu
>  031 rw auto quick
>  032 rw auto quick
>  033 rw auto quick
> @@ -46,38 +46,38 @@
>  037 rw auto backing quick
>  038 rw auto backing quick
>  039 rw auto quick
> -040 rw auto
> -041 rw auto backing
> +040 rw auto qemu
> +041 rw auto backing qemu
>  042 rw auto quick
>  043 rw auto backing
>  044 rw auto
> -045 rw auto quick
> +045 rw auto quick scm qemu
>  046 rw auto aio quick
>  047 rw auto quick
>  048 img auto quick
>  049 rw auto
>  050 rw auto backing quick
> -051 rw auto
> +051 rw auto qemu
>  052 rw auto backing quick
>  053 rw auto quick
>  054 rw auto quick
> -055 rw auto
> -056 rw auto backing
> -057 rw auto
> -058 rw auto quick
> +055 rw auto qemu
> +056 rw auto backing qemu
> +057 rw auto qemu
> +058 rw auto quick nbd
>  059 rw auto quick
>  060 rw auto quick
>  061 rw auto
>  062 rw auto quick
>  063 rw auto quick
>  064 rw auto quick
> -065 rw auto quick
> +065 rw auto quick qemu
>  066 rw auto quick
> -067 rw auto quick
> -068 rw auto quick
> +067 rw auto quick qemu
> +068 rw auto quick qemu
>  069 rw auto quick
>  070 rw auto quick
> -071 rw auto quick
> +071 rw auto quick qemu
>  072 rw auto quick
>  073 rw auto quick
>  074 rw auto quick
> @@ -87,45 +87,45 @@
>  078 rw auto quick
>  079 rw auto
>  080 rw auto
> -081 rw auto quick
> +081 rw auto quick qemu
>  082 rw auto quick
> -083 rw auto
> +083 rw auto nbd
>  084 img auto quick
> -085 rw auto
> +085 rw auto qemu
>  086 rw auto quick
> -087 rw auto quick
> +087 rw auto quick qemu
>  088 rw auto quick
>  089 rw auto quick
>  090 rw auto quick
> -091 rw auto
> +091 rw auto qemu
>  092 rw auto quick
> -093 auto
> -094 rw auto quick
> -095 rw auto quick
> +093 auto qemu
> +094 rw auto quick qemu nbd
> +095 rw auto quick qemu
>  097 rw auto backing
>  098 rw auto backing quick
> -099 rw auto quick
> +099 rw auto quick qemu
>  100 rw auto quick
>  101 rw auto quick
> -102 rw auto quick
> +102 rw auto quick qemu
>  103 rw auto quick
>  104 rw auto
>  105 rw auto quick
>  107 rw auto quick
>  108 rw auto quick
> -109 rw auto
> +109 rw auto qemu
>  110 rw auto backing quick
>  111 rw auto quick
>  112 rw auto
> -113 rw auto quick
> +113 rw auto quick nbd
>  114 rw auto quick
>  115 rw auto
>  116 rw auto quick
>  121 rw auto
>  122 rw auto
> -123 rw auto quick
> -124 rw auto backing
> +123 rw auto quick nbd
> +124 rw auto backing qemu
>  128 rw auto quick
> -129 rw auto quick
> -130 rw auto quick
> +129 rw auto quick qemu
> +130 rw auto quick qemu
>  131 rw auto quick
> -- 
> 2.1.0
> 

Re: [Qemu-devel] [PATCH 2/4] iotests: Add dependency info to groups list

[John Snow]

On 05/27/2015 11:34 PM, Fam Zheng wrote:
> On Fri, 05/22 16:17, John Snow wrote:
>> Add a few external dependency groups to the iotests groups list,
>> such as "qemu", "nbd", and "scm".
>>
>> This will assist us in knowing which tests need to be re-run
>> when those dependencies are updated, or by helping us to avoid
>> tests that we know are broken for a certain file.
>>
>> qemu-img and qemu-io are so ubiquitous that it's not worth
>> amending this information to the groups file.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
> 
> 007 uses $QEMU too ;-) Others looks good.
> 

Hm, nuts... I actually tried to double check my work by deleting the
binary and seeing what tests failed/passed, but I missed this one.

Will fix up for v2, thanks.

> Reviewed-by: Fam Zheng <famz@redhat.com>
> 
>> ---
>>  tests/qemu-iotests/group | 60 ++++++++++++++++++++++++------------------------
>>  1 file changed, 30 insertions(+), 30 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>> index 34b16cb..32f15d8 100644
>> --- a/tests/qemu-iotests/group
>> +++ b/tests/qemu-iotests/group
>> @@ -34,9 +34,9 @@
>>  025 rw auto quick
>>  026 rw blkdbg auto
>>  027 rw auto quick
>> -028 rw backing auto quick
>> +028 rw backing auto quick qemu
>>  029 rw auto quick
>> -030 rw auto backing
>> +030 rw auto backing qemu
>>  031 rw auto quick
>>  032 rw auto quick
>>  033 rw auto quick
>> @@ -46,38 +46,38 @@
>>  037 rw auto backing quick
>>  038 rw auto backing quick
>>  039 rw auto quick
>> -040 rw auto
>> -041 rw auto backing
>> +040 rw auto qemu
>> +041 rw auto backing qemu
>>  042 rw auto quick
>>  043 rw auto backing
>>  044 rw auto
>> -045 rw auto quick
>> +045 rw auto quick scm qemu
>>  046 rw auto aio quick
>>  047 rw auto quick
>>  048 img auto quick
>>  049 rw auto
>>  050 rw auto backing quick
>> -051 rw auto
>> +051 rw auto qemu
>>  052 rw auto backing quick
>>  053 rw auto quick
>>  054 rw auto quick
>> -055 rw auto
>> -056 rw auto backing
>> -057 rw auto
>> -058 rw auto quick
>> +055 rw auto qemu
>> +056 rw auto backing qemu
>> +057 rw auto qemu
>> +058 rw auto quick nbd
>>  059 rw auto quick
>>  060 rw auto quick
>>  061 rw auto
>>  062 rw auto quick
>>  063 rw auto quick
>>  064 rw auto quick
>> -065 rw auto quick
>> +065 rw auto quick qemu
>>  066 rw auto quick
>> -067 rw auto quick
>> -068 rw auto quick
>> +067 rw auto quick qemu
>> +068 rw auto quick qemu
>>  069 rw auto quick
>>  070 rw auto quick
>> -071 rw auto quick
>> +071 rw auto quick qemu
>>  072 rw auto quick
>>  073 rw auto quick
>>  074 rw auto quick
>> @@ -87,45 +87,45 @@
>>  078 rw auto quick
>>  079 rw auto
>>  080 rw auto
>> -081 rw auto quick
>> +081 rw auto quick qemu
>>  082 rw auto quick
>> -083 rw auto
>> +083 rw auto nbd
>>  084 img auto quick
>> -085 rw auto
>> +085 rw auto qemu
>>  086 rw auto quick
>> -087 rw auto quick
>> +087 rw auto quick qemu
>>  088 rw auto quick
>>  089 rw auto quick
>>  090 rw auto quick
>> -091 rw auto
>> +091 rw auto qemu
>>  092 rw auto quick
>> -093 auto
>> -094 rw auto quick
>> -095 rw auto quick
>> +093 auto qemu
>> +094 rw auto quick qemu nbd
>> +095 rw auto quick qemu
>>  097 rw auto backing
>>  098 rw auto backing quick
>> -099 rw auto quick
>> +099 rw auto quick qemu
>>  100 rw auto quick
>>  101 rw auto quick
>> -102 rw auto quick
>> +102 rw auto quick qemu
>>  103 rw auto quick
>>  104 rw auto
>>  105 rw auto quick
>>  107 rw auto quick
>>  108 rw auto quick
>> -109 rw auto
>> +109 rw auto qemu
>>  110 rw auto backing quick
>>  111 rw auto quick
>>  112 rw auto
>> -113 rw auto quick
>> +113 rw auto quick nbd
>>  114 rw auto quick
>>  115 rw auto
>>  116 rw auto quick
>>  121 rw auto
>>  122 rw auto
>> -123 rw auto quick
>> -124 rw auto backing
>> +123 rw auto quick nbd
>> +124 rw auto backing qemu
>>  128 rw auto quick
>> -129 rw auto quick
>> -130 rw auto quick
>> +129 rw auto quick qemu
>> +130 rw auto quick qemu
>>  131 rw auto quick
>> -- 
>> 2.1.0
>>

-- 
—js