* [PATCH v3 00/10] Rework iotests/check
@ 2020-04-21 7:35 Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running Vladimir Sementsov-Ogievskiy
` (9 more replies)
0 siblings, 10 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Hi all!
This is a continuation for "[PATCH v2 0/2] Rework iotests finding",
so main user-visible feature here is removing tests/qemu-iotests/group
file, define groups in tests and allow human-readable names for iotests.
While being here, rewrite check into python :) I tried to keep most of
its behavior, still some changes are made and described in three main
commits 06-08.
Vladimir Sementsov-Ogievskiy (10):
iotests/277: use dot slash for nbd-fault-injector.py running
iotests: fix some whitespaces in test output files
iotests/283: make executable
iotests/check: move QEMU_VXHS_PROG to common.rc
iotests: define group in each iotest
iotests: add testfinder.py
iotests: add testenv.py
iotests: add testrunner.py
iotests: rewrite check into python
iotests: rename 169 and 199
docs/devel/testing.rst | 52 +-
tests/qemu-iotests/001 | 1 +
tests/qemu-iotests/002 | 1 +
tests/qemu-iotests/003 | 1 +
tests/qemu-iotests/004 | 1 +
tests/qemu-iotests/005 | 1 +
tests/qemu-iotests/007 | 1 +
tests/qemu-iotests/008 | 1 +
tests/qemu-iotests/009 | 1 +
tests/qemu-iotests/010 | 1 +
tests/qemu-iotests/011 | 1 +
tests/qemu-iotests/012 | 1 +
tests/qemu-iotests/013 | 1 +
tests/qemu-iotests/014 | 1 +
tests/qemu-iotests/015 | 1 +
tests/qemu-iotests/017 | 1 +
tests/qemu-iotests/018 | 1 +
tests/qemu-iotests/019 | 1 +
tests/qemu-iotests/020 | 1 +
tests/qemu-iotests/021 | 1 +
tests/qemu-iotests/022 | 1 +
tests/qemu-iotests/023 | 1 +
tests/qemu-iotests/024 | 1 +
tests/qemu-iotests/025 | 1 +
tests/qemu-iotests/026 | 1 +
tests/qemu-iotests/027 | 1 +
tests/qemu-iotests/028 | 1 +
tests/qemu-iotests/029 | 1 +
tests/qemu-iotests/030 | 1 +
tests/qemu-iotests/031 | 1 +
tests/qemu-iotests/032 | 1 +
tests/qemu-iotests/033 | 1 +
tests/qemu-iotests/034 | 1 +
tests/qemu-iotests/035 | 1 +
tests/qemu-iotests/036 | 1 +
tests/qemu-iotests/037 | 1 +
tests/qemu-iotests/038 | 1 +
tests/qemu-iotests/039 | 1 +
tests/qemu-iotests/040 | 1 +
tests/qemu-iotests/041 | 1 +
tests/qemu-iotests/042 | 1 +
tests/qemu-iotests/043 | 1 +
tests/qemu-iotests/044 | 1 +
tests/qemu-iotests/045 | 1 +
tests/qemu-iotests/046 | 1 +
tests/qemu-iotests/047 | 1 +
tests/qemu-iotests/048 | 1 +
tests/qemu-iotests/049 | 1 +
tests/qemu-iotests/050 | 1 +
tests/qemu-iotests/051 | 1 +
tests/qemu-iotests/052 | 1 +
tests/qemu-iotests/053 | 1 +
tests/qemu-iotests/054 | 1 +
tests/qemu-iotests/055 | 1 +
tests/qemu-iotests/056 | 1 +
tests/qemu-iotests/057 | 1 +
tests/qemu-iotests/058 | 1 +
tests/qemu-iotests/059 | 1 +
tests/qemu-iotests/060 | 1 +
tests/qemu-iotests/061 | 1 +
tests/qemu-iotests/062 | 1 +
tests/qemu-iotests/063 | 1 +
tests/qemu-iotests/064 | 1 +
tests/qemu-iotests/065 | 1 +
tests/qemu-iotests/066 | 1 +
tests/qemu-iotests/067 | 1 +
tests/qemu-iotests/068 | 1 +
tests/qemu-iotests/069 | 1 +
tests/qemu-iotests/070 | 1 +
tests/qemu-iotests/071 | 1 +
tests/qemu-iotests/072 | 1 +
tests/qemu-iotests/073 | 1 +
tests/qemu-iotests/074 | 1 +
tests/qemu-iotests/075 | 1 +
tests/qemu-iotests/076 | 1 +
tests/qemu-iotests/077 | 1 +
tests/qemu-iotests/078 | 1 +
tests/qemu-iotests/079 | 1 +
tests/qemu-iotests/080 | 1 +
tests/qemu-iotests/081 | 1 +
tests/qemu-iotests/082 | 1 +
tests/qemu-iotests/083 | 1 +
tests/qemu-iotests/084 | 1 +
tests/qemu-iotests/085 | 1 +
tests/qemu-iotests/086 | 1 +
tests/qemu-iotests/087 | 1 +
tests/qemu-iotests/088 | 1 +
tests/qemu-iotests/089 | 1 +
tests/qemu-iotests/090 | 1 +
tests/qemu-iotests/091 | 1 +
tests/qemu-iotests/092 | 1 +
tests/qemu-iotests/093 | 1 +
tests/qemu-iotests/094 | 1 +
tests/qemu-iotests/095 | 1 +
tests/qemu-iotests/096 | 1 +
tests/qemu-iotests/097 | 1 +
tests/qemu-iotests/098 | 1 +
tests/qemu-iotests/099 | 1 +
tests/qemu-iotests/101 | 1 +
tests/qemu-iotests/102 | 1 +
tests/qemu-iotests/103 | 1 +
tests/qemu-iotests/104 | 1 +
tests/qemu-iotests/105 | 1 +
tests/qemu-iotests/106 | 1 +
tests/qemu-iotests/107 | 1 +
tests/qemu-iotests/108 | 1 +
tests/qemu-iotests/109 | 1 +
tests/qemu-iotests/110 | 1 +
tests/qemu-iotests/111 | 1 +
tests/qemu-iotests/112 | 1 +
tests/qemu-iotests/113 | 1 +
tests/qemu-iotests/114 | 1 +
tests/qemu-iotests/115 | 1 +
tests/qemu-iotests/116 | 1 +
tests/qemu-iotests/117 | 1 +
tests/qemu-iotests/118 | 1 +
tests/qemu-iotests/119 | 1 +
tests/qemu-iotests/120 | 1 +
tests/qemu-iotests/121 | 1 +
tests/qemu-iotests/122 | 1 +
tests/qemu-iotests/123 | 1 +
tests/qemu-iotests/124 | 1 +
tests/qemu-iotests/125 | 1 +
tests/qemu-iotests/126 | 1 +
tests/qemu-iotests/127 | 1 +
tests/qemu-iotests/128 | 1 +
tests/qemu-iotests/129 | 1 +
tests/qemu-iotests/130 | 1 +
tests/qemu-iotests/131 | 1 +
tests/qemu-iotests/132 | 1 +
tests/qemu-iotests/133 | 1 +
tests/qemu-iotests/134 | 1 +
tests/qemu-iotests/135 | 1 +
tests/qemu-iotests/136 | 1 +
tests/qemu-iotests/137 | 1 +
tests/qemu-iotests/138 | 1 +
tests/qemu-iotests/139 | 1 +
tests/qemu-iotests/140 | 1 +
tests/qemu-iotests/141 | 1 +
tests/qemu-iotests/143 | 1 +
tests/qemu-iotests/144 | 1 +
tests/qemu-iotests/145 | 1 +
tests/qemu-iotests/146 | 1 +
tests/qemu-iotests/147 | 1 +
tests/qemu-iotests/148 | 1 +
tests/qemu-iotests/149 | 1 +
tests/qemu-iotests/150 | 1 +
tests/qemu-iotests/151 | 1 +
tests/qemu-iotests/152 | 1 +
tests/qemu-iotests/153 | 1 +
tests/qemu-iotests/154 | 1 +
tests/qemu-iotests/155 | 1 +
tests/qemu-iotests/156 | 1 +
tests/qemu-iotests/157 | 1 +
tests/qemu-iotests/158 | 1 +
tests/qemu-iotests/159 | 1 +
tests/qemu-iotests/160 | 1 +
tests/qemu-iotests/161 | 1 +
tests/qemu-iotests/162 | 1 +
tests/qemu-iotests/163 | 1 +
tests/qemu-iotests/165 | 1 +
tests/qemu-iotests/170 | 1 +
tests/qemu-iotests/171 | 1 +
tests/qemu-iotests/172 | 1 +
tests/qemu-iotests/173 | 1 +
tests/qemu-iotests/174 | 1 +
tests/qemu-iotests/175 | 1 +
tests/qemu-iotests/175.out | 2 +-
tests/qemu-iotests/176 | 1 +
tests/qemu-iotests/177 | 1 +
tests/qemu-iotests/178 | 1 +
tests/qemu-iotests/179 | 1 +
tests/qemu-iotests/181 | 1 +
tests/qemu-iotests/182 | 1 +
tests/qemu-iotests/183 | 1 +
tests/qemu-iotests/184 | 1 +
tests/qemu-iotests/185 | 1 +
tests/qemu-iotests/186 | 1 +
tests/qemu-iotests/187 | 1 +
tests/qemu-iotests/188 | 1 +
tests/qemu-iotests/189 | 1 +
tests/qemu-iotests/190 | 1 +
tests/qemu-iotests/191 | 1 +
tests/qemu-iotests/192 | 1 +
tests/qemu-iotests/194 | 1 +
tests/qemu-iotests/195 | 1 +
tests/qemu-iotests/196 | 1 +
tests/qemu-iotests/197 | 1 +
tests/qemu-iotests/198 | 1 +
tests/qemu-iotests/200 | 1 +
tests/qemu-iotests/201 | 1 +
tests/qemu-iotests/202 | 1 +
tests/qemu-iotests/203 | 1 +
tests/qemu-iotests/204 | 1 +
tests/qemu-iotests/205 | 1 +
tests/qemu-iotests/206 | 1 +
tests/qemu-iotests/207 | 1 +
tests/qemu-iotests/208 | 1 +
tests/qemu-iotests/209 | 1 +
tests/qemu-iotests/210 | 1 +
tests/qemu-iotests/211 | 1 +
tests/qemu-iotests/212 | 1 +
tests/qemu-iotests/213 | 1 +
tests/qemu-iotests/214 | 1 +
tests/qemu-iotests/215 | 1 +
tests/qemu-iotests/216 | 1 +
tests/qemu-iotests/217 | 1 +
tests/qemu-iotests/218 | 1 +
tests/qemu-iotests/219 | 1 +
tests/qemu-iotests/220 | 1 +
tests/qemu-iotests/221 | 1 +
tests/qemu-iotests/222 | 1 +
tests/qemu-iotests/223 | 1 +
tests/qemu-iotests/224 | 1 +
tests/qemu-iotests/225 | 1 +
tests/qemu-iotests/226 | 1 +
tests/qemu-iotests/227 | 1 +
tests/qemu-iotests/228 | 1 +
tests/qemu-iotests/229 | 1 +
tests/qemu-iotests/231 | 1 +
tests/qemu-iotests/232 | 1 +
tests/qemu-iotests/233 | 1 +
tests/qemu-iotests/234 | 1 +
tests/qemu-iotests/235 | 1 +
tests/qemu-iotests/236 | 1 +
tests/qemu-iotests/237 | 1 +
tests/qemu-iotests/238 | 1 +
tests/qemu-iotests/239 | 1 +
tests/qemu-iotests/240 | 1 +
tests/qemu-iotests/241 | 1 +
tests/qemu-iotests/242 | 1 +
tests/qemu-iotests/243 | 1 +
tests/qemu-iotests/244 | 1 +
tests/qemu-iotests/245 | 1 +
tests/qemu-iotests/246 | 1 +
tests/qemu-iotests/247 | 1 +
tests/qemu-iotests/248 | 1 +
tests/qemu-iotests/249 | 1 +
tests/qemu-iotests/250 | 1 +
tests/qemu-iotests/251 | 1 +
tests/qemu-iotests/252 | 1 +
tests/qemu-iotests/253 | 1 +
tests/qemu-iotests/254 | 1 +
tests/qemu-iotests/255 | 1 +
tests/qemu-iotests/256 | 1 +
tests/qemu-iotests/257 | 1 +
tests/qemu-iotests/258 | 1 +
tests/qemu-iotests/259 | 1 +
tests/qemu-iotests/260 | 1 +
tests/qemu-iotests/261 | 1 +
tests/qemu-iotests/262 | 1 +
tests/qemu-iotests/263 | 1 +
tests/qemu-iotests/264 | 1 +
tests/qemu-iotests/265 | 1 +
tests/qemu-iotests/266 | 1 +
tests/qemu-iotests/267 | 1 +
tests/qemu-iotests/267.out | 48 +-
tests/qemu-iotests/268 | 1 +
tests/qemu-iotests/270 | 1 +
tests/qemu-iotests/272 | 1 +
tests/qemu-iotests/273 | 1 +
tests/qemu-iotests/277 | 3 +-
tests/qemu-iotests/279 | 1 +
tests/qemu-iotests/280 | 1 +
tests/qemu-iotests/281 | 1 +
tests/qemu-iotests/282 | 1 +
tests/qemu-iotests/283 | 1 +
tests/qemu-iotests/284 | 1 +
tests/qemu-iotests/286 | 1 +
tests/qemu-iotests/288 | 1 +
tests/qemu-iotests/289 | 1 +
tests/qemu-iotests/290 | 1 +
tests/qemu-iotests/check | 980 +-----------------
tests/qemu-iotests/common.rc | 14 +
tests/qemu-iotests/group | 299 ------
.../{199 => migrate-bitmaps-postcopy-test} | 1 +
....out => migrate-bitmaps-postcopy-test.out} | 0
.../{169 => migrate-bitmaps-test} | 1 +
.../{169.out => migrate-bitmaps-test.out} | 0
tests/qemu-iotests/testenv.py | 332 ++++++
tests/qemu-iotests/testfinder.py | 167 +++
tests/qemu-iotests/testrunner.py | 334 ++++++
282 files changed, 1223 insertions(+), 1278 deletions(-)
mode change 100644 => 100755 tests/qemu-iotests/283
delete mode 100644 tests/qemu-iotests/group
rename tests/qemu-iotests/{199 => migrate-bitmaps-postcopy-test} (99%)
rename tests/qemu-iotests/{199.out => migrate-bitmaps-postcopy-test.out} (100%)
rename tests/qemu-iotests/{169 => migrate-bitmaps-test} (99%)
rename tests/qemu-iotests/{169.out => migrate-bitmaps-test.out} (100%)
create mode 100755 tests/qemu-iotests/testenv.py
create mode 100755 tests/qemu-iotests/testfinder.py
create mode 100644 tests/qemu-iotests/testrunner.py
--
2.21.0
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 12:54 ` Eric Blake
2020-04-21 7:35 ` [PATCH v3 02/10] iotests: fix some whitespaces in test output files Vladimir Sementsov-Ogievskiy
` (8 subsequent siblings)
9 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Somehow, this doesn't work if you just call ./277. But check 277 works.
We are going to new iotests running framework, which will consider test
files as self-executable, so let's just change it now.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/277 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277
index 04aa15a3d5..a4ef1a38c6 100755
--- a/tests/qemu-iotests/277
+++ b/tests/qemu-iotests/277
@@ -40,7 +40,7 @@ def make_conf_file(event):
def start_server_NBD(event):
make_conf_file(event)
- srv = subprocess.Popen(['nbd-fault-injector.py', '--classic-negotiation',
+ srv = subprocess.Popen(['./nbd-fault-injector.py', '--classic-negotiation',
nbd_sock, conf_file], stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True)
line = srv.stdout.readline()
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 02/10] iotests: fix some whitespaces in test output files
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 03/10] iotests/283: make executable Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
9 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
We are going to be stricter about comparing test result with .out
files. So, fix some whitespaces now.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/175.out | 2 +-
tests/qemu-iotests/267.out | 48 +++++++++++++++++++-------------------
2 files changed, 25 insertions(+), 25 deletions(-)
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index 39c2ee0f62..40a5bd1ce6 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -23,4 +23,4 @@ size=4096, min allocation
== resize empty image with block_resize ==
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
size=1048576, min allocation
- *** done
+*** done
diff --git a/tests/qemu-iotests/267.out b/tests/qemu-iotests/267.out
index 8dddb4baa4..26ef5f64b9 100644
--- a/tests/qemu-iotests/267.out
+++ b/tests/qemu-iotests/267.out
@@ -33,8 +33,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -44,8 +44,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -69,8 +69,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -94,8 +94,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -105,8 +105,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -119,8 +119,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -134,8 +134,8 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
@@ -145,15 +145,15 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
Internal snapshots on overlay:
Snapshot list:
-ID TAG VM SIZE DATE VM CLOCK
-1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
Internal snapshots on backing file:
=== -blockdev with NBD server on the backing file ===
@@ -166,17 +166,17 @@ QEMU X.Y.Z monitor - type 'help' for more information
(qemu) savevm snap0
(qemu) info snapshots
List of snapshots present on all disks:
-ID TAG VM SIZE DATE VM CLOCK
--- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+-- snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
(qemu) loadvm snap0
(qemu) quit
Internal snapshots on overlay:
Snapshot list:
-ID TAG VM SIZE DATE VM CLOCK
-1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
Internal snapshots on backing file:
Snapshot list:
-ID TAG VM SIZE DATE VM CLOCK
-1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
+ID TAG VM SIZE DATE VM CLOCK
+1 snap0 SIZE yyyy-mm-dd hh:mm:ss 00:00:00.000
*** done
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 03/10] iotests/283: make executable
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 02/10] iotests: fix some whitespaces in test output files Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 12:55 ` Eric Blake
2020-05-14 6:17 ` Philippe Mathieu-Daudé
2020-04-21 7:35 ` [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
9 siblings, 2 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
All other test files are executable, except for this one. Fix that.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/283 | 0
1 file changed, 0 insertions(+), 0 deletions(-)
mode change 100644 => 100755 tests/qemu-iotests/283
diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283
old mode 100644
new mode 100755
--
2.21.0
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 03/10] iotests/283: make executable Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 16:03 ` Kevin Wolf
2020-04-21 7:35 ` [PATCH v3 05/10] iotests: define group in each iotest Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
9 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
QEMU_VXHS_PROG is used only in common.rc. So, move it to common.rc,
simplifying a bit further conversion of check into python
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/check | 15 ---------------
tests/qemu-iotests/common.rc | 14 ++++++++++++++
2 files changed, 14 insertions(+), 15 deletions(-)
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index f7a2d3d6c3..03016e1e91 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -103,17 +103,6 @@ SOCKET_SCM_HELPER -- $SOCKET_SCM_HELPER
EOF
}
-# $1 = prog to look for
-set_prog_path()
-{
- p=$(command -v $1 2> /dev/null)
- if [ -n "$p" -a -x "$p" ]; then
- type -p "$p"
- else
- return 1
- fi
-}
-
if [ -z "$TEST_DIR" ]; then
TEST_DIR=$PWD/scratch
fi
@@ -643,10 +632,6 @@ if [ -z $QEMU_NBD_PROG ]; then
fi
export QEMU_NBD_PROG="$(type -p "$QEMU_NBD_PROG")"
-if [ -z "$QEMU_VXHS_PROG" ]; then
- export QEMU_VXHS_PROG="$(set_prog_path qnio_server)"
-fi
-
if [ -x "$build_iotests/socket_scm_helper" ]
then
export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index bf3b9fdea0..aaab17c718 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -109,6 +109,16 @@ peek_file_raw()
dd if="$1" bs=1 skip="$2" count="$3" status=none
}
+# $1 = prog to look for
+set_prog_path()
+{
+ p=$(command -v $1 2> /dev/null)
+ if [ -n "$p" -a -x "$p" ]; then
+ type -p "$p"
+ else
+ return 1
+ fi
+}
if ! . ./common.config
then
@@ -116,6 +126,10 @@ if ! . ./common.config
exit 1
fi
+if [ -z "$QEMU_VXHS_PROG" ]; then
+ QEMU_VXHS_PROG="$(set_prog_path qnio_server)"
+fi
+
# Set the variables to the empty string to turn Valgrind off
# for specific processes, e.g.
# $ VALGRIND_QEMU_IO= ./check -qcow2 -valgrind 015
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 05/10] iotests: define group in each iotest
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 06/10] iotests: add testfinder.py Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
9 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
We are going to drop group file. Define group in tests as a preparatory
step.
The patch is generated by
cd tests/qemu-iotests
grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/001 | 1 +
tests/qemu-iotests/002 | 1 +
tests/qemu-iotests/003 | 1 +
tests/qemu-iotests/004 | 1 +
tests/qemu-iotests/005 | 1 +
tests/qemu-iotests/007 | 1 +
tests/qemu-iotests/008 | 1 +
tests/qemu-iotests/009 | 1 +
tests/qemu-iotests/010 | 1 +
tests/qemu-iotests/011 | 1 +
tests/qemu-iotests/012 | 1 +
tests/qemu-iotests/013 | 1 +
tests/qemu-iotests/014 | 1 +
tests/qemu-iotests/015 | 1 +
tests/qemu-iotests/017 | 1 +
tests/qemu-iotests/018 | 1 +
tests/qemu-iotests/019 | 1 +
tests/qemu-iotests/020 | 1 +
tests/qemu-iotests/021 | 1 +
tests/qemu-iotests/022 | 1 +
tests/qemu-iotests/023 | 1 +
tests/qemu-iotests/024 | 1 +
tests/qemu-iotests/025 | 1 +
tests/qemu-iotests/026 | 1 +
tests/qemu-iotests/027 | 1 +
tests/qemu-iotests/028 | 1 +
tests/qemu-iotests/029 | 1 +
tests/qemu-iotests/030 | 1 +
tests/qemu-iotests/031 | 1 +
tests/qemu-iotests/032 | 1 +
tests/qemu-iotests/033 | 1 +
tests/qemu-iotests/034 | 1 +
tests/qemu-iotests/035 | 1 +
tests/qemu-iotests/036 | 1 +
tests/qemu-iotests/037 | 1 +
tests/qemu-iotests/038 | 1 +
tests/qemu-iotests/039 | 1 +
tests/qemu-iotests/040 | 1 +
tests/qemu-iotests/041 | 1 +
tests/qemu-iotests/042 | 1 +
tests/qemu-iotests/043 | 1 +
tests/qemu-iotests/044 | 1 +
tests/qemu-iotests/045 | 1 +
tests/qemu-iotests/046 | 1 +
tests/qemu-iotests/047 | 1 +
tests/qemu-iotests/048 | 1 +
tests/qemu-iotests/049 | 1 +
tests/qemu-iotests/050 | 1 +
tests/qemu-iotests/051 | 1 +
tests/qemu-iotests/052 | 1 +
tests/qemu-iotests/053 | 1 +
tests/qemu-iotests/054 | 1 +
tests/qemu-iotests/055 | 1 +
tests/qemu-iotests/056 | 1 +
tests/qemu-iotests/057 | 1 +
tests/qemu-iotests/058 | 1 +
tests/qemu-iotests/059 | 1 +
tests/qemu-iotests/060 | 1 +
tests/qemu-iotests/061 | 1 +
tests/qemu-iotests/062 | 1 +
tests/qemu-iotests/063 | 1 +
tests/qemu-iotests/064 | 1 +
tests/qemu-iotests/065 | 1 +
tests/qemu-iotests/066 | 1 +
tests/qemu-iotests/067 | 1 +
tests/qemu-iotests/068 | 1 +
tests/qemu-iotests/069 | 1 +
tests/qemu-iotests/070 | 1 +
tests/qemu-iotests/071 | 1 +
tests/qemu-iotests/072 | 1 +
tests/qemu-iotests/073 | 1 +
tests/qemu-iotests/074 | 1 +
tests/qemu-iotests/075 | 1 +
tests/qemu-iotests/076 | 1 +
tests/qemu-iotests/077 | 1 +
tests/qemu-iotests/078 | 1 +
tests/qemu-iotests/079 | 1 +
tests/qemu-iotests/080 | 1 +
tests/qemu-iotests/081 | 1 +
tests/qemu-iotests/082 | 1 +
tests/qemu-iotests/083 | 1 +
tests/qemu-iotests/084 | 1 +
tests/qemu-iotests/085 | 1 +
tests/qemu-iotests/086 | 1 +
tests/qemu-iotests/087 | 1 +
tests/qemu-iotests/088 | 1 +
tests/qemu-iotests/089 | 1 +
tests/qemu-iotests/090 | 1 +
tests/qemu-iotests/091 | 1 +
tests/qemu-iotests/092 | 1 +
tests/qemu-iotests/093 | 1 +
tests/qemu-iotests/094 | 1 +
tests/qemu-iotests/095 | 1 +
tests/qemu-iotests/096 | 1 +
tests/qemu-iotests/097 | 1 +
tests/qemu-iotests/098 | 1 +
tests/qemu-iotests/099 | 1 +
tests/qemu-iotests/101 | 1 +
tests/qemu-iotests/102 | 1 +
tests/qemu-iotests/103 | 1 +
tests/qemu-iotests/104 | 1 +
tests/qemu-iotests/105 | 1 +
tests/qemu-iotests/106 | 1 +
tests/qemu-iotests/107 | 1 +
tests/qemu-iotests/108 | 1 +
tests/qemu-iotests/109 | 1 +
tests/qemu-iotests/110 | 1 +
tests/qemu-iotests/111 | 1 +
tests/qemu-iotests/112 | 1 +
tests/qemu-iotests/113 | 1 +
tests/qemu-iotests/114 | 1 +
tests/qemu-iotests/115 | 1 +
tests/qemu-iotests/116 | 1 +
tests/qemu-iotests/117 | 1 +
tests/qemu-iotests/118 | 1 +
tests/qemu-iotests/119 | 1 +
tests/qemu-iotests/120 | 1 +
tests/qemu-iotests/121 | 1 +
tests/qemu-iotests/122 | 1 +
tests/qemu-iotests/123 | 1 +
tests/qemu-iotests/124 | 1 +
tests/qemu-iotests/125 | 1 +
tests/qemu-iotests/126 | 1 +
tests/qemu-iotests/127 | 1 +
tests/qemu-iotests/128 | 1 +
tests/qemu-iotests/129 | 1 +
tests/qemu-iotests/130 | 1 +
tests/qemu-iotests/131 | 1 +
tests/qemu-iotests/132 | 1 +
tests/qemu-iotests/133 | 1 +
tests/qemu-iotests/134 | 1 +
tests/qemu-iotests/135 | 1 +
tests/qemu-iotests/136 | 1 +
tests/qemu-iotests/137 | 1 +
tests/qemu-iotests/138 | 1 +
tests/qemu-iotests/139 | 1 +
tests/qemu-iotests/140 | 1 +
tests/qemu-iotests/141 | 1 +
tests/qemu-iotests/143 | 1 +
tests/qemu-iotests/144 | 1 +
tests/qemu-iotests/145 | 1 +
tests/qemu-iotests/146 | 1 +
tests/qemu-iotests/147 | 1 +
tests/qemu-iotests/148 | 1 +
tests/qemu-iotests/149 | 1 +
tests/qemu-iotests/150 | 1 +
tests/qemu-iotests/151 | 1 +
tests/qemu-iotests/152 | 1 +
tests/qemu-iotests/153 | 1 +
tests/qemu-iotests/154 | 1 +
tests/qemu-iotests/155 | 1 +
tests/qemu-iotests/156 | 1 +
tests/qemu-iotests/157 | 1 +
tests/qemu-iotests/158 | 1 +
tests/qemu-iotests/159 | 1 +
tests/qemu-iotests/160 | 1 +
tests/qemu-iotests/161 | 1 +
tests/qemu-iotests/162 | 1 +
tests/qemu-iotests/163 | 1 +
tests/qemu-iotests/165 | 1 +
tests/qemu-iotests/169 | 1 +
tests/qemu-iotests/170 | 1 +
tests/qemu-iotests/171 | 1 +
tests/qemu-iotests/172 | 1 +
tests/qemu-iotests/173 | 1 +
tests/qemu-iotests/174 | 1 +
tests/qemu-iotests/175 | 1 +
tests/qemu-iotests/176 | 1 +
tests/qemu-iotests/177 | 1 +
tests/qemu-iotests/178 | 1 +
tests/qemu-iotests/179 | 1 +
tests/qemu-iotests/181 | 1 +
tests/qemu-iotests/182 | 1 +
tests/qemu-iotests/183 | 1 +
tests/qemu-iotests/184 | 1 +
tests/qemu-iotests/185 | 1 +
tests/qemu-iotests/186 | 1 +
tests/qemu-iotests/187 | 1 +
tests/qemu-iotests/188 | 1 +
tests/qemu-iotests/189 | 1 +
tests/qemu-iotests/190 | 1 +
tests/qemu-iotests/191 | 1 +
tests/qemu-iotests/192 | 1 +
tests/qemu-iotests/194 | 1 +
tests/qemu-iotests/195 | 1 +
tests/qemu-iotests/196 | 1 +
tests/qemu-iotests/197 | 1 +
tests/qemu-iotests/198 | 1 +
tests/qemu-iotests/199 | 1 +
tests/qemu-iotests/200 | 1 +
tests/qemu-iotests/201 | 1 +
tests/qemu-iotests/202 | 1 +
tests/qemu-iotests/203 | 1 +
tests/qemu-iotests/204 | 1 +
tests/qemu-iotests/205 | 1 +
tests/qemu-iotests/206 | 1 +
tests/qemu-iotests/207 | 1 +
tests/qemu-iotests/208 | 1 +
tests/qemu-iotests/209 | 1 +
tests/qemu-iotests/210 | 1 +
tests/qemu-iotests/211 | 1 +
tests/qemu-iotests/212 | 1 +
tests/qemu-iotests/213 | 1 +
tests/qemu-iotests/214 | 1 +
tests/qemu-iotests/215 | 1 +
tests/qemu-iotests/216 | 1 +
tests/qemu-iotests/217 | 1 +
tests/qemu-iotests/218 | 1 +
tests/qemu-iotests/219 | 1 +
tests/qemu-iotests/220 | 1 +
tests/qemu-iotests/221 | 1 +
tests/qemu-iotests/222 | 1 +
tests/qemu-iotests/223 | 1 +
tests/qemu-iotests/224 | 1 +
tests/qemu-iotests/225 | 1 +
tests/qemu-iotests/226 | 1 +
tests/qemu-iotests/227 | 1 +
tests/qemu-iotests/228 | 1 +
tests/qemu-iotests/229 | 1 +
tests/qemu-iotests/231 | 1 +
tests/qemu-iotests/232 | 1 +
tests/qemu-iotests/233 | 1 +
tests/qemu-iotests/234 | 1 +
tests/qemu-iotests/235 | 1 +
tests/qemu-iotests/236 | 1 +
tests/qemu-iotests/237 | 1 +
tests/qemu-iotests/238 | 1 +
tests/qemu-iotests/239 | 1 +
tests/qemu-iotests/240 | 1 +
tests/qemu-iotests/241 | 1 +
tests/qemu-iotests/242 | 1 +
tests/qemu-iotests/243 | 1 +
tests/qemu-iotests/244 | 1 +
tests/qemu-iotests/245 | 1 +
tests/qemu-iotests/246 | 1 +
tests/qemu-iotests/247 | 1 +
tests/qemu-iotests/248 | 1 +
tests/qemu-iotests/249 | 1 +
tests/qemu-iotests/250 | 1 +
tests/qemu-iotests/251 | 1 +
tests/qemu-iotests/252 | 1 +
tests/qemu-iotests/253 | 1 +
tests/qemu-iotests/254 | 1 +
tests/qemu-iotests/255 | 1 +
tests/qemu-iotests/256 | 1 +
tests/qemu-iotests/257 | 1 +
tests/qemu-iotests/258 | 1 +
tests/qemu-iotests/259 | 1 +
tests/qemu-iotests/260 | 1 +
tests/qemu-iotests/261 | 1 +
tests/qemu-iotests/262 | 1 +
tests/qemu-iotests/263 | 1 +
tests/qemu-iotests/264 | 1 +
tests/qemu-iotests/265 | 1 +
tests/qemu-iotests/266 | 1 +
tests/qemu-iotests/267 | 1 +
tests/qemu-iotests/268 | 1 +
tests/qemu-iotests/270 | 1 +
tests/qemu-iotests/272 | 1 +
tests/qemu-iotests/273 | 1 +
tests/qemu-iotests/277 | 1 +
tests/qemu-iotests/279 | 1 +
tests/qemu-iotests/280 | 1 +
tests/qemu-iotests/281 | 1 +
tests/qemu-iotests/282 | 1 +
tests/qemu-iotests/283 | 1 +
tests/qemu-iotests/284 | 1 +
tests/qemu-iotests/286 | 1 +
tests/qemu-iotests/288 | 1 +
tests/qemu-iotests/289 | 1 +
tests/qemu-iotests/290 | 1 +
271 files changed, 271 insertions(+)
diff --git a/tests/qemu-iotests/001 b/tests/qemu-iotests/001
index d87a535c33..67ff7e3291 100755
--- a/tests/qemu-iotests/001
+++ b/tests/qemu-iotests/001
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test simple read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index 1a0d411df5..5ce1647531 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test simple read/write using plain bdrv_pread/bdrv_pwrite
#
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index 33eeade0de..03f902a83c 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test simple read/write using bdrv_aio_readv/bdrv_aio_writev
#
diff --git a/tests/qemu-iotests/004 b/tests/qemu-iotests/004
index d308dc4b49..e955579a67 100755
--- a/tests/qemu-iotests/004
+++ b/tests/qemu-iotests/004
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Make sure we can't read and write outside of the image size.
#
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index b6d03ac37d..40e64a9a8f 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: img auto quick
#
# Make sure qemu-img can create 5TB images
#
diff --git a/tests/qemu-iotests/007 b/tests/qemu-iotests/007
index 160683adf8..936d3f14fb 100755
--- a/tests/qemu-iotests/007
+++ b/tests/qemu-iotests/007
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: snapshot auto
#
# Check for one possible case of qcow2 refcount corruption.
#
diff --git a/tests/qemu-iotests/008 b/tests/qemu-iotests/008
index 2b81b119bf..fa4990b513 100755
--- a/tests/qemu-iotests/008
+++ b/tests/qemu-iotests/008
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test simple asynchronous read/write operations.
#
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index 4dc7d210f9..efa852bad3 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Nolan I qcow2 corruption - incorrectly reports free clusters
#
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index df809b3088..4ae9027b47 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Nolan II qcow2 corruption - wrong used cluster
#
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index 57b99ae4a9..5c99ac987f 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test for AIO allocation on the same cluster
#
diff --git a/tests/qemu-iotests/012 b/tests/qemu-iotests/012
index 12957285b3..3a24d2ca8d 100755
--- a/tests/qemu-iotests/012
+++ b/tests/qemu-iotests/012
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto quick
#
# Make sure we can open read-only images
#
diff --git a/tests/qemu-iotests/013 b/tests/qemu-iotests/013
index 5cb9032f16..d39d0cd88b 100755
--- a/tests/qemu-iotests/013
+++ b/tests/qemu-iotests/013
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# qcow2 pattern test, empty and compressed image - 4k cluster patterns
#
diff --git a/tests/qemu-iotests/014 b/tests/qemu-iotests/014
index e1221c0fff..2d23469332 100755
--- a/tests/qemu-iotests/014
+++ b/tests/qemu-iotests/014
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# qcow2 pattern test, complex patterns including compression and snapshots
# Using patterns for 4k cluster size.
diff --git a/tests/qemu-iotests/015 b/tests/qemu-iotests/015
index 4d8effd0ae..40c23235a6 100755
--- a/tests/qemu-iotests/015
+++ b/tests/qemu-iotests/015
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw snapshot
#
# Combined test to grow the refcount table and test snapshots.
#
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index 0a4b854e65..3cab97ad94 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing auto quick
#
# Simple backing file reads
#
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index c69ce09209..a946243450 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing auto quick
#
# Merge backing file into test image when converting the image
#
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index 813a84acac..a1a2065f6a 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing auto quick
#
# When using a backing file for the output image in qemu-img convert,
# the backing file clusters must not copied. The data must still be
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index 20f8f185d0..a4ae2e1017 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing auto quick
#
# Commit changes to backing file
#
diff --git a/tests/qemu-iotests/021 b/tests/qemu-iotests/021
index f888269fd4..0fc89df2fe 100755
--- a/tests/qemu-iotests/021
+++ b/tests/qemu-iotests/021
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: io auto quick
#
# Test handling of invalid patterns arguments to qemu-io
#
diff --git a/tests/qemu-iotests/022 b/tests/qemu-iotests/022
index 99eb08f57f..a116cfe255 100755
--- a/tests/qemu-iotests/022
+++ b/tests/qemu-iotests/022
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw snapshot auto
#
# Test bdrv_load/save_vmstate using the usual patterns
#
diff --git a/tests/qemu-iotests/023 b/tests/qemu-iotests/023
index 02ed047820..d19d13ff5d 100755
--- a/tests/qemu-iotests/023
+++ b/tests/qemu-iotests/023
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# qcow2 pattern test with various cluster sizes
#
diff --git a/tests/qemu-iotests/024 b/tests/qemu-iotests/024
index e2e766241e..de70aeac1c 100755
--- a/tests/qemu-iotests/024
+++ b/tests/qemu-iotests/024
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing auto quick
#
# Rebasing COW images
#
diff --git a/tests/qemu-iotests/025 b/tests/qemu-iotests/025
index e05d833452..0047969d17 100755
--- a/tests/qemu-iotests/025
+++ b/tests/qemu-iotests/025
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Resizing images
#
diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index b9713eb591..204af667b3 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw blkdbg
#
# qcow2 error path testing
#
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index 494be0921f..b279c88f33 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test that sub-cluster allocating writes zero the rest of the cluster
#
diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
index 797dae5350..665889297f 100755
--- a/tests/qemu-iotests/028
+++ b/tests/qemu-iotests/028
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing quick
#
# Test that backing files can be smaller than the image
#
diff --git a/tests/qemu-iotests/029 b/tests/qemu-iotests/029
index 2161a4b87a..3a20d2d1ff 100755
--- a/tests/qemu-iotests/029
+++ b/tests/qemu-iotests/029
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# qcow2 internal snapshots/VM state tests
#
diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index aa911d266a..a9066d4947 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw auto backing
#
# Tests for image streaming.
#
diff --git a/tests/qemu-iotests/031 b/tests/qemu-iotests/031
index 646ecd593f..af167e85be 100755
--- a/tests/qemu-iotests/031
+++ b/tests/qemu-iotests/031
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test that all qcow2 header extensions survive a header rewrite
#
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index 8337a4d825..ebbe7cb0ba 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test that AIO requests are drained before an image is closed. This used
# to segfault because the request coroutine kept running even after the
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index 8b40991d55..da9133c44b 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test aligned and misaligned write zeroes operations.
#
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index da4cea1571..e5d5996aa0 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test bdrv_pwrite_zeroes with backing files (see also 154)
#
diff --git a/tests/qemu-iotests/035 b/tests/qemu-iotests/035
index d950a0dd1e..0c0c4fdd42 100755
--- a/tests/qemu-iotests/035
+++ b/tests/qemu-iotests/035
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Let a few AIO requests run in parallel and have them access different L2
# tables so that the cache has a chance to get used up.
diff --git a/tests/qemu-iotests/036 b/tests/qemu-iotests/036
index cf522de7a1..a27d4125d2 100755
--- a/tests/qemu-iotests/036
+++ b/tests/qemu-iotests/036
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qcow2 feature bits
#
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index e6517acbd4..9bcc7695a6 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test COW from backing files
#
diff --git a/tests/qemu-iotests/038 b/tests/qemu-iotests/038
index 707e2d72e9..63dff13827 100755
--- a/tests/qemu-iotests/038
+++ b/tests/qemu-iotests/038
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test COW from backing files with AIO
#
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index ddce48ab47..3cddb8a720 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qcow2 lazy refcounts
#
diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 32c82b4ec6..1a8bd3f613 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw auto
#
# Tests for image block commit.
#
diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index 5d67bf14bf..b6a14dc1b2 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw auto backing
#
# Tests for image mirroring.
#
diff --git a/tests/qemu-iotests/042 b/tests/qemu-iotests/042
index 0e85b356b7..6cf823fa74 100755
--- a/tests/qemu-iotests/042
+++ b/tests/qemu-iotests/042
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qemu-img operation on zero size images
#
diff --git a/tests/qemu-iotests/043 b/tests/qemu-iotests/043
index b102e49208..83b97cccaf 100755
--- a/tests/qemu-iotests/043
+++ b/tests/qemu-iotests/043
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing
#
# Test that qemu-img info --backing-chain detects infinite loops
#
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 7e99ea7c68..64b18eb7c8 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests growing a large refcount table.
#
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 5acc89099c..45eb239baa 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for fdsets and getfd.
#
diff --git a/tests/qemu-iotests/046 b/tests/qemu-iotests/046
index ecbe5fc0f4..47b216f4bf 100755
--- a/tests/qemu-iotests/046
+++ b/tests/qemu-iotests/046
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto aio quick
#
# Test concurrent cluster allocations
#
diff --git a/tests/qemu-iotests/047 b/tests/qemu-iotests/047
index 4528465fb0..8dd21e0a81 100755
--- a/tests/qemu-iotests/047
+++ b/tests/qemu-iotests/047
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Regression test for commit b7ab0fea (which was a corruption fix,
# despite the commit message claiming otherwise)
diff --git a/tests/qemu-iotests/048 b/tests/qemu-iotests/048
index 2af6b74b41..bf8e4bf528 100755
--- a/tests/qemu-iotests/048
+++ b/tests/qemu-iotests/048
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: img auto quick
##
## qemu-img compare test
##
diff --git a/tests/qemu-iotests/049 b/tests/qemu-iotests/049
index 051a1c79e0..ed6730c4e8 100755
--- a/tests/qemu-iotests/049
+++ b/tests/qemu-iotests/049
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Check qemu-img option parsing
#
diff --git a/tests/qemu-iotests/050 b/tests/qemu-iotests/050
index cdc5356541..62cc29f756 100755
--- a/tests/qemu-iotests/050
+++ b/tests/qemu-iotests/050
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test qemu-img rebase with zero clusters
#
diff --git a/tests/qemu-iotests/051 b/tests/qemu-iotests/051
index 034d3a3250..833aeda9f7 100755
--- a/tests/qemu-iotests/051
+++ b/tests/qemu-iotests/051
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test command line configuration of block devices and driver-specific options
#
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index 45a140910d..44c7895b80 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test bdrv_read/bdrv_write using BDRV_O_SNAPSHOT
#
diff --git a/tests/qemu-iotests/053 b/tests/qemu-iotests/053
index 71d299c4f9..9a2958d42d 100755
--- a/tests/qemu-iotests/053
+++ b/tests/qemu-iotests/053
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qemu-img convert when image length is not a multiple of cluster size
#
diff --git a/tests/qemu-iotests/054 b/tests/qemu-iotests/054
index a8905b60d0..a796f565e0 100755
--- a/tests/qemu-iotests/054
+++ b/tests/qemu-iotests/054
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test huge qcow2 images
#
diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055
index 82b9f5f47d..c70664dac6 100755
--- a/tests/qemu-iotests/055
+++ b/tests/qemu-iotests/055
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests for drive-backup and blockdev-backup
#
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index f73fc74457..ffb2077a03 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw backing
#
# Tests for drive-backup
#
diff --git a/tests/qemu-iotests/057 b/tests/qemu-iotests/057
index a8b4bb60e0..b0d431999e 100755
--- a/tests/qemu-iotests/057
+++ b/tests/qemu-iotests/057
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests for internal snapshot.
#
diff --git a/tests/qemu-iotests/058 b/tests/qemu-iotests/058
index d84740ed9f..ce35ff4ee0 100755
--- a/tests/qemu-iotests/058
+++ b/tests/qemu-iotests/058
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test export internal snapshot by qemu-nbd, convert it by qemu-img.
#
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index 5438025285..33f5450edd 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for vmdk
#
diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060
index 32c0ecce9e..3c72602806 100755
--- a/tests/qemu-iotests/060
+++ b/tests/qemu-iotests/060
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for image corruption (overlapping data structures) in qcow2
#
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index ce285d3084..1455b2957a 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test case for image option amendment in qcow2.
#
diff --git a/tests/qemu-iotests/062 b/tests/qemu-iotests/062
index f26b88df9d..321252298d 100755
--- a/tests/qemu-iotests/062
+++ b/tests/qemu-iotests/062
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for snapshotting images with unallocated zero clusters in
# qcow2
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index c750b3806e..3a44758053 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# test of qemu-img convert -n - convert without creation
#
diff --git a/tests/qemu-iotests/064 b/tests/qemu-iotests/064
index 90673186ec..71fc575b21 100755
--- a/tests/qemu-iotests/064
+++ b/tests/qemu-iotests/064
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test VHDX read/write from a sample image created with Hyper-V
#
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 6426474271..2d76287d44 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test for additional information emitted by qemu-img info on qcow2
# images
diff --git a/tests/qemu-iotests/066 b/tests/qemu-iotests/066
index a4ac613f8e..a780ed7ab5 100755
--- a/tests/qemu-iotests/066
+++ b/tests/qemu-iotests/066
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for preallocated zero clusters in qcow2
#
diff --git a/tests/qemu-iotests/067 b/tests/qemu-iotests/067
index a63be9cabf..b23f8c65de 100755
--- a/tests/qemu-iotests/067
+++ b/tests/qemu-iotests/067
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test automatic deletion of BDSes created by -drive/drive_add
#
diff --git a/tests/qemu-iotests/068 b/tests/qemu-iotests/068
index ccd1a9f1db..03e03508a6 100755
--- a/tests/qemu-iotests/068
+++ b/tests/qemu-iotests/068
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for loading a saved VM state from a qcow2 image
#
diff --git a/tests/qemu-iotests/069 b/tests/qemu-iotests/069
index b997b127f0..ccb24bc948 100755
--- a/tests/qemu-iotests/069
+++ b/tests/qemu-iotests/069
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for deleting a backing file
#
diff --git a/tests/qemu-iotests/070 b/tests/qemu-iotests/070
index cb0f927c16..b181e00f9b 100755
--- a/tests/qemu-iotests/070
+++ b/tests/qemu-iotests/070
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test VHDX log replay from an image with a journal that needs to be
# replayed
diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index 88faebcc1d..1d92f7f4a6 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for the QMP blkdebug and blkverify interfaces
#
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index f0b73e7e65..c492ab8a78 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for nested image formats
#
diff --git a/tests/qemu-iotests/073 b/tests/qemu-iotests/073
index 23a1bdf890..ccdd465bbf 100755
--- a/tests/qemu-iotests/073
+++ b/tests/qemu-iotests/073
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test count_contiguous_clusters in qcow2
#
diff --git a/tests/qemu-iotests/074 b/tests/qemu-iotests/074
index db03edf0b0..c32c94b50d 100755
--- a/tests/qemu-iotests/074
+++ b/tests/qemu-iotests/074
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
##
## qemu-img compare test (qcow2 only ones)
##
diff --git a/tests/qemu-iotests/075 b/tests/qemu-iotests/075
index 389d5675fa..ca2ed2a05c 100755
--- a/tests/qemu-iotests/075
+++ b/tests/qemu-iotests/075
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# cloop format input validation tests
#
diff --git a/tests/qemu-iotests/076 b/tests/qemu-iotests/076
index 0d405ef3f2..1a8927d765 100755
--- a/tests/qemu-iotests/076
+++ b/tests/qemu-iotests/076
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: io
#
# parallels format input validation tests
#
diff --git a/tests/qemu-iotests/077 b/tests/qemu-iotests/077
index c284952082..fbb90d8036 100755
--- a/tests/qemu-iotests/077
+++ b/tests/qemu-iotests/077
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test concurrent pread/pwrite
#
diff --git a/tests/qemu-iotests/078 b/tests/qemu-iotests/078
index 54fc654d8e..0b48b7f137 100755
--- a/tests/qemu-iotests/078
+++ b/tests/qemu-iotests/078
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# bochs format input validation tests
#
diff --git a/tests/qemu-iotests/079 b/tests/qemu-iotests/079
index 3642b51feb..e00a3cda2f 100755
--- a/tests/qemu-iotests/079
+++ b/tests/qemu-iotests/079
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test qcow2 preallocation with different cluster_sizes
#
diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080
index a3d13c414e..e0bdef11f4 100755
--- a/tests/qemu-iotests/080
+++ b/tests/qemu-iotests/080
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# qcow2 format input validation tests
#
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index 537d40dfd5..a8185036cb 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test Quorum block driver
#
diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
index 3286c2c6db..2259c9d9f7 100755
--- a/tests/qemu-iotests/082
+++ b/tests/qemu-iotests/082
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qemu-img command line parsing
#
diff --git a/tests/qemu-iotests/083 b/tests/qemu-iotests/083
index 10fdfc8ebb..bc32b537b2 100755
--- a/tests/qemu-iotests/083
+++ b/tests/qemu-iotests/083
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test NBD client unexpected disconnect
#
diff --git a/tests/qemu-iotests/084 b/tests/qemu-iotests/084
index c29d7395e9..e51e91a7c8 100755
--- a/tests/qemu-iotests/084
+++ b/tests/qemu-iotests/084
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: img quick
#
# Test case for VDI header corruption; image too large, and too many blocks.
# Also simple test for creating dynamic and static VDI images.
diff --git a/tests/qemu-iotests/085 b/tests/qemu-iotests/085
index dd3c993a2d..20bdea3952 100755
--- a/tests/qemu-iotests/085
+++ b/tests/qemu-iotests/085
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Live snapshot tests
#
diff --git a/tests/qemu-iotests/086 b/tests/qemu-iotests/086
index fea1a7bd8a..c055e7bfe1 100755
--- a/tests/qemu-iotests/086
+++ b/tests/qemu-iotests/086
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qemu-img progress output
#
diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index bdfdad3454..6eff852f22 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test unsupported blockdev-add cases
#
diff --git a/tests/qemu-iotests/088 b/tests/qemu-iotests/088
index ef1163346c..e3102fe888 100755
--- a/tests/qemu-iotests/088
+++ b/tests/qemu-iotests/088
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# vpc (VHD) format input validation tests
#
diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089
index 059ad75e28..1041cde0a1 100755
--- a/tests/qemu-iotests/089
+++ b/tests/qemu-iotests/089
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for support of JSON filenames
#
diff --git a/tests/qemu-iotests/090 b/tests/qemu-iotests/090
index 1246e4f910..cda38eeef2 100755
--- a/tests/qemu-iotests/090
+++ b/tests/qemu-iotests/090
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test for discarding compressed clusters on qcow2 images
#
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index d2a2aca347..988ea12b39 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw migration
#
# Live migration test
#
diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092
index 40ec62b6f1..bfa116d191 100755
--- a/tests/qemu-iotests/092
+++ b/tests/qemu-iotests/092
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# qcow1 format input validation tests
#
diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index 32ded11430..7745cb04b6 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: throttle
#
# Tests for IO throttling
#
diff --git a/tests/qemu-iotests/094 b/tests/qemu-iotests/094
index 2d3e1004d3..a295fb20ef 100755
--- a/tests/qemu-iotests/094
+++ b/tests/qemu-iotests/094
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for drive-mirror to NBD
#
diff --git a/tests/qemu-iotests/095 b/tests/qemu-iotests/095
index 155ae86aa7..24c8ca2672 100755
--- a/tests/qemu-iotests/095
+++ b/tests/qemu-iotests/095
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test for commit of larger active layer
#
diff --git a/tests/qemu-iotests/096 b/tests/qemu-iotests/096
index 5915f92786..b5d7636bdc 100755
--- a/tests/qemu-iotests/096
+++ b/tests/qemu-iotests/096
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test that snapshots move the throttling configuration to the active
# layer
diff --git a/tests/qemu-iotests/097 b/tests/qemu-iotests/097
index 690f3d3ce1..2a95ee0c0f 100755
--- a/tests/qemu-iotests/097
+++ b/tests/qemu-iotests/097
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing
#
# Commit changes into backing chains and empty the top image if the
# backing image is not explicitly specified
diff --git a/tests/qemu-iotests/098 b/tests/qemu-iotests/098
index 1e29d96b3d..2b029a41c8 100755
--- a/tests/qemu-iotests/098
+++ b/tests/qemu-iotests/098
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test qcow2's bdrv_make_empty for images without internal snapshots
#
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index 65e8e92572..2f1199ce04 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test valid filenames for blkdebug and blkverify representatively for
# other protocols (such as NBD) when queried
diff --git a/tests/qemu-iotests/101 b/tests/qemu-iotests/101
index a4c1b6366a..4c4a8cea11 100755
--- a/tests/qemu-iotests/101
+++ b/tests/qemu-iotests/101
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test short file I/O
#
diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102
index b898df436f..0dd3cee426 100755
--- a/tests/qemu-iotests/102
+++ b/tests/qemu-iotests/102
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for qemu-io -c map and qemu-img map
#
diff --git a/tests/qemu-iotests/103 b/tests/qemu-iotests/103
index 8c1ebe0443..5fee8a0119 100755
--- a/tests/qemu-iotests/103
+++ b/tests/qemu-iotests/103
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for qcow2 metadata cache size specification
#
diff --git a/tests/qemu-iotests/104 b/tests/qemu-iotests/104
index c70f28a9a1..3ebb74cf6e 100755
--- a/tests/qemu-iotests/104
+++ b/tests/qemu-iotests/104
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test image creation with aligned and unaligned sizes
#
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index 4d55a2d3ef..d804685110 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Create, read, write big image
#
diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/106
index b5d1ec4078..9e833a105c 100755
--- a/tests/qemu-iotests/106
+++ b/tests/qemu-iotests/106
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test preallocated resize of raw images
#
diff --git a/tests/qemu-iotests/107 b/tests/qemu-iotests/107
index 268ba27688..86dc27343e 100755
--- a/tests/qemu-iotests/107
+++ b/tests/qemu-iotests/107
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Tests updates of the qcow2 L1 table
#
diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108
index 5f7076fba4..6b1ddbb5b6 100755
--- a/tests/qemu-iotests/108
+++ b/tests/qemu-iotests/108
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for repairing qcow2 images which cannot be repaired using
# the on-disk refcount structures
diff --git a/tests/qemu-iotests/109 b/tests/qemu-iotests/109
index a51dd84b3d..cbdaab88dd 100755
--- a/tests/qemu-iotests/109
+++ b/tests/qemu-iotests/109
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test writing image headers of other formats into raw images
#
diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index 139c02c2cf..4e2a3ce04d 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test case for relative backing file names in complex BDS trees
#
diff --git a/tests/qemu-iotests/111 b/tests/qemu-iotests/111
index 3b43d1bd83..1a3d1ee741 100755
--- a/tests/qemu-iotests/111
+++ b/tests/qemu-iotests/111
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for non-existing backing file when creating a qcow2 image
# and not specifying the size
diff --git a/tests/qemu-iotests/112 b/tests/qemu-iotests/112
index 20ff5c224a..7a5b859c81 100755
--- a/tests/qemu-iotests/112
+++ b/tests/qemu-iotests/112
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test cases for different refcount_bits values
#
diff --git a/tests/qemu-iotests/113 b/tests/qemu-iotests/113
index f2703a2c50..5dc92b49ca 100755
--- a/tests/qemu-iotests/113
+++ b/tests/qemu-iotests/113
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for accessing creation options on image formats and
# protocols not supporting image creation
diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114
index 26104fff6c..a3a0e2c807 100755
--- a/tests/qemu-iotests/114
+++ b/tests/qemu-iotests/114
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test invalid backing file format in qcow2 images
#
diff --git a/tests/qemu-iotests/115 b/tests/qemu-iotests/115
index d254b18342..5c3f60bc1b 100755
--- a/tests/qemu-iotests/115
+++ b/tests/qemu-iotests/115
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test case for non-self-referential qcow2 refcount blocks
#
diff --git a/tests/qemu-iotests/116 b/tests/qemu-iotests/116
index 941b07a1a9..4f40fcb3d2 100755
--- a/tests/qemu-iotests/116
+++ b/tests/qemu-iotests/116
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test error code paths for invalid QED images
#
diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
index f37b34f8b1..8210a0f70d 100755
--- a/tests/qemu-iotests/117
+++ b/tests/qemu-iotests/117
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test case for shared BDS between backend trees
#
diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index adc8a848b5..d705a01b78 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test case for the QMP 'change' command and all other associated
# commands
diff --git a/tests/qemu-iotests/119 b/tests/qemu-iotests/119
index ea6770a484..5770b50045 100755
--- a/tests/qemu-iotests/119
+++ b/tests/qemu-iotests/119
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# NBD test case for overriding BDRV_O_PROTOCOL by explicitly specifying
# a driver
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index 2931a7550f..03d5a3ba33 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Non-NBD test cases for overriding BDRV_O_PROTOCOL by explicitly
# specifying a driver
diff --git a/tests/qemu-iotests/121 b/tests/qemu-iotests/121
index 90ea0db737..0abe27f9f7 100755
--- a/tests/qemu-iotests/121
+++ b/tests/qemu-iotests/121
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test cases for qcow2 refcount table growth
#
diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122
index f7a3ae684a..5faf9ab034 100755
--- a/tests/qemu-iotests/122
+++ b/tests/qemu-iotests/122
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test some qemu-img convert cases
#
diff --git a/tests/qemu-iotests/123 b/tests/qemu-iotests/123
index 01b771c76e..e19111f70d 100755
--- a/tests/qemu-iotests/123
+++ b/tests/qemu-iotests/123
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for qemu-img convert to NBD
#
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3705cbb6b3..3b21bc497f 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw backing
#
# Tests for incremental drive-backup
#
diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125
index d510984045..55e45657a6 100755
--- a/tests/qemu-iotests/125
+++ b/tests/qemu-iotests/125
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test preallocated growth of qcow2 images
#
diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
index b7fce1e59d..7e4d1b3345 100755
--- a/tests/qemu-iotests/126
+++ b/tests/qemu-iotests/126
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing
#
# Tests handling of colons in filenames (which may be confused with protocol
# prefixes)
diff --git a/tests/qemu-iotests/127 b/tests/qemu-iotests/127
index a4fc866038..b30a49579f 100755
--- a/tests/qemu-iotests/127
+++ b/tests/qemu-iotests/127
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Test case for mirroring with dataplane
#
diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
index 3606c41760..d0e00d24b1 100755
--- a/tests/qemu-iotests/128
+++ b/tests/qemu-iotests/128
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test that opening O_DIRECT succeeds when image file I/O produces EIO
#
diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index b0da4a5541..0873b4800d 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests that "bdrv_drain_all" doesn't drain block jobs
#
diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
index 77ad2aa13a..9d50960d66 100755
--- a/tests/qemu-iotests/130
+++ b/tests/qemu-iotests/130
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test that temporary backing file overrides (on the command line or in
# blockdev-add) don't replace the original path stored in the image during
diff --git a/tests/qemu-iotests/131 b/tests/qemu-iotests/131
index 27870231cf..d7456cae5b 100755
--- a/tests/qemu-iotests/131
+++ b/tests/qemu-iotests/131
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# parallels format validation tests (created by QEMU)
#
diff --git a/tests/qemu-iotests/132 b/tests/qemu-iotests/132
index 39ea43067e..367ea08036 100755
--- a/tests/qemu-iotests/132
+++ b/tests/qemu-iotests/132
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test mirror with unmap
#
diff --git a/tests/qemu-iotests/133 b/tests/qemu-iotests/133
index 6f7cacc091..602c4b7a96 100755
--- a/tests/qemu-iotests/133
+++ b/tests/qemu-iotests/133
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto quick
#
# Test for reopen
#
diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index 5f0fb86211..30cc53088e 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test encrypted read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/135 b/tests/qemu-iotests/135
index 3b3d1dc2a5..299075b4c9 100755
--- a/tests/qemu-iotests/135
+++ b/tests/qemu-iotests/135
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test VPC open of image with large Max Table Entries value.
#
diff --git a/tests/qemu-iotests/136 b/tests/qemu-iotests/136
index d59400c9fc..8fce88bd67 100755
--- a/tests/qemu-iotests/136
+++ b/tests/qemu-iotests/136
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests for block device statistics
#
diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137
index 7ae86892f7..1cb2fe1653 100755
--- a/tests/qemu-iotests/137
+++ b/tests/qemu-iotests/137
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test qcow2 reopen
#
diff --git a/tests/qemu-iotests/138 b/tests/qemu-iotests/138
index 1d5b0bed6d..04d0bb5487 100755
--- a/tests/qemu-iotests/138
+++ b/tests/qemu-iotests/138
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# General test case for qcow2's image check
#
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index 7120d3142b..401e8a915d 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test cases for the QMP 'blockdev-del' command
#
diff --git a/tests/qemu-iotests/140 b/tests/qemu-iotests/140
index 8d2ce5d9e3..13bc4ca2d7 100755
--- a/tests/qemu-iotests/140
+++ b/tests/qemu-iotests/140
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for ejecting a BlockBackend with an NBD server attached to it
#
diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141
index 5192d256e3..38736e16f2 100755
--- a/tests/qemu-iotests/141
+++ b/tests/qemu-iotests/141
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for ejecting BDSs with block jobs still running on them
#
diff --git a/tests/qemu-iotests/143 b/tests/qemu-iotests/143
index f649b36195..8e2009df26 100755
--- a/tests/qemu-iotests/143
+++ b/tests/qemu-iotests/143
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto quick
#
# Test case for connecting to a non-existing NBD export name
#
diff --git a/tests/qemu-iotests/144 b/tests/qemu-iotests/144
index 4569ac0b4b..60e9ddd75f 100755
--- a/tests/qemu-iotests/144
+++ b/tests/qemu-iotests/144
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
# Check live snapshot, followed by active commit, and another snapshot.
#
# This test is to catch the error case of BZ #1300209:
diff --git a/tests/qemu-iotests/145 b/tests/qemu-iotests/145
index 9427549651..a2ce92516d 100755
--- a/tests/qemu-iotests/145
+++ b/tests/qemu-iotests/145
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test the combination of -incoming and snapshot=on
#
diff --git a/tests/qemu-iotests/146 b/tests/qemu-iotests/146
index 2e43abddfc..8bc9bbefa3 100755
--- a/tests/qemu-iotests/146
+++ b/tests/qemu-iotests/146
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test VHD image format creator detection and override
#
diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index d7a9f31089..47dfa62e6b 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: img
#
# Test case for NBD's blockdev-add interface
#
diff --git a/tests/qemu-iotests/148 b/tests/qemu-iotests/148
index 90931948e3..35f3861dfc 100755
--- a/tests/qemu-iotests/148
+++ b/tests/qemu-iotests/148
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test the rate limit of QMP events
#
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index b4a21bf7b7..e84f0f8e1f 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw sudo
#
# Copyright (C) 2016 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/150 b/tests/qemu-iotests/150
index 3b1f32197a..ac6930ae20 100755
--- a/tests/qemu-iotests/150
+++ b/tests/qemu-iotests/150
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test that qemu-img convert -S 0 fully allocates the target image
#
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index f2df72c29c..182f6b5321 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests for active mirroring
#
diff --git a/tests/qemu-iotests/152 b/tests/qemu-iotests/152
index cc2ea09654..4e179c340f 100755
--- a/tests/qemu-iotests/152
+++ b/tests/qemu-iotests/152
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for drive-mirror with source size unaligned to granularity
#
diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153
index 2b13111768..ef6a3cfd18 100755
--- a/tests/qemu-iotests/153
+++ b/tests/qemu-iotests/153
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test image locking
#
diff --git a/tests/qemu-iotests/154 b/tests/qemu-iotests/154
index d68f66b9e0..83900cc072 100755
--- a/tests/qemu-iotests/154
+++ b/tests/qemu-iotests/154
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# qcow2 specific bdrv_pwrite_zeroes tests with backing files (complements 034)
#
diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155
index 571bce9de4..4ac3f86a8b 100755
--- a/tests/qemu-iotests/155
+++ b/tests/qemu-iotests/155
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test whether the backing BDSs are correct after completion of a
# mirror block job; in "existing" modes (drive-mirror with
diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156
index 5559df63a5..4019ada313 100755
--- a/tests/qemu-iotests/156
+++ b/tests/qemu-iotests/156
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Tests oVirt-like storage migration:
# - Create snapshot
diff --git a/tests/qemu-iotests/157 b/tests/qemu-iotests/157
index 7cbac38099..0dc9ba68d2 100755
--- a/tests/qemu-iotests/157
+++ b/tests/qemu-iotests/157
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test command line configuration of block devices with qdev
#
diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158
index ba4db6116a..6dcd14aeb2 100755
--- a/tests/qemu-iotests/158
+++ b/tests/qemu-iotests/158
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test encrypted read/write using backing files
#
diff --git a/tests/qemu-iotests/159 b/tests/qemu-iotests/159
index f9690053a2..4eb476d3a8 100755
--- a/tests/qemu-iotests/159
+++ b/tests/qemu-iotests/159
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# qemu-img dd test with different block sizes
#
diff --git a/tests/qemu-iotests/160 b/tests/qemu-iotests/160
index 0572b5ae9a..7984a9c6f7 100755
--- a/tests/qemu-iotests/160
+++ b/tests/qemu-iotests/160
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# qemu-img dd test for the skip option
#
diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
index f572a19af2..e161fb4679 100755
--- a/tests/qemu-iotests/161
+++ b/tests/qemu-iotests/161
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test reopening a backing image after block-stream and block-commit
#
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index c0053ed975..cf17f494d8 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test case for specifying runtime options of the wrong type to some
# block drivers
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 5a3cc840a5..dedce8ef43 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Tests for shrinking images
#
diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index fb56a769b4..abc4ffadd5 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for persistent dirty bitmaps.
#
diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index 2c5a132aa3..2d41e1cc03 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick migration
#
# Tests for dirty bitmaps migration.
#
diff --git a/tests/qemu-iotests/170 b/tests/qemu-iotests/170
index 6c8f0e8085..41387e4d66 100755
--- a/tests/qemu-iotests/170
+++ b/tests/qemu-iotests/170
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# qemu-img dd test
#
diff --git a/tests/qemu-iotests/171 b/tests/qemu-iotests/171
index 341064a1c6..babfe98753 100755
--- a/tests/qemu-iotests/171
+++ b/tests/qemu-iotests/171
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test 'offset' and 'size' options of the raw driver. Make sure we can't
# (or can) read and write outside of the image size.
diff --git a/tests/qemu-iotests/172 b/tests/qemu-iotests/172
index 7195fb895a..4548e3aab9 100755
--- a/tests/qemu-iotests/172
+++ b/tests/qemu-iotests/172
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto
#
# Test floppy configuration
#
diff --git a/tests/qemu-iotests/173 b/tests/qemu-iotests/173
index ec6d1705e5..9594f3c5ea 100755
--- a/tests/qemu-iotests/173
+++ b/tests/qemu-iotests/173
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test QAPI commands looking up protocol based images with relative
# filename backing strings
diff --git a/tests/qemu-iotests/174 b/tests/qemu-iotests/174
index e2f14a38c6..1a141fbc0f 100755
--- a/tests/qemu-iotests/174
+++ b/tests/qemu-iotests/174
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto
#
# Test that qemu-io fail with non-zero exit code
#
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index 020ed8e61f..9bb7ea6936 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test creating raw image preallocation mode
#
diff --git a/tests/qemu-iotests/176 b/tests/qemu-iotests/176
index 117c8b6954..e5be07d6a6 100755
--- a/tests/qemu-iotests/176
+++ b/tests/qemu-iotests/176
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing
#
# Commit changes into backing chains and empty the top image if the
# backing image is not explicitly specified.
diff --git a/tests/qemu-iotests/177 b/tests/qemu-iotests/177
index eadc2c7ef6..88b2c977c5 100755
--- a/tests/qemu-iotests/177
+++ b/tests/qemu-iotests/177
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test corner cases with unusual block geometries
#
diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178
index 7cf0e27154..b911b79915 100755
--- a/tests/qemu-iotests/178
+++ b/tests/qemu-iotests/178
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: img
#
# qemu-img measure sub-command tests
#
diff --git a/tests/qemu-iotests/179 b/tests/qemu-iotests/179
index 9372dc30ef..ebd48e6ad2 100755
--- a/tests/qemu-iotests/179
+++ b/tests/qemu-iotests/179
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test case for write zeroes with unmap
#
diff --git a/tests/qemu-iotests/181 b/tests/qemu-iotests/181
index 438c2dcd80..dadf3bf88e 100755
--- a/tests/qemu-iotests/181
+++ b/tests/qemu-iotests/181
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto migration
#
# Test postcopy live migration with shared storage
#
diff --git a/tests/qemu-iotests/182 b/tests/qemu-iotests/182
index 56a2dd58e6..55a0384c08 100755
--- a/tests/qemu-iotests/182
+++ b/tests/qemu-iotests/182
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test image locking for POSIX locks
#
diff --git a/tests/qemu-iotests/183 b/tests/qemu-iotests/183
index acdbefa310..16390fe93d 100755
--- a/tests/qemu-iotests/183
+++ b/tests/qemu-iotests/183
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw migration
#
# Test old-style block migration (migrate -b)
#
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index 33dd8d2a4f..1d923f509d 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test I/O throttle block filter driver interface
#
diff --git a/tests/qemu-iotests/185 b/tests/qemu-iotests/185
index fd5e6ebe11..7bc8fe5767 100755
--- a/tests/qemu-iotests/185
+++ b/tests/qemu-iotests/185
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test exiting qemu while jobs are still running
#
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index 3ea0442d44..66f2418182 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test 'info block' with all kinds of configurations
#
diff --git a/tests/qemu-iotests/187 b/tests/qemu-iotests/187
index c6e1dc57a0..46fd5678c7 100755
--- a/tests/qemu-iotests/187
+++ b/tests/qemu-iotests/187
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test switching between read-only and read-write
#
diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188
index afca44df54..87e902352d 100755
--- a/tests/qemu-iotests/188
+++ b/tests/qemu-iotests/188
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test encrypted read/write using plain bdrv_read/bdrv_write
#
diff --git a/tests/qemu-iotests/189 b/tests/qemu-iotests/189
index c9ce9d3bed..2f6f8a9309 100755
--- a/tests/qemu-iotests/189
+++ b/tests/qemu-iotests/189
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test encrypted read/write using backing files
#
diff --git a/tests/qemu-iotests/190 b/tests/qemu-iotests/190
index 6d41650438..6011be0861 100755
--- a/tests/qemu-iotests/190
+++ b/tests/qemu-iotests/190
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# qemu-img measure sub-command tests on huge qcow2 files
#
diff --git a/tests/qemu-iotests/191 b/tests/qemu-iotests/191
index b05db68141..bce1d8445e 100755
--- a/tests/qemu-iotests/191
+++ b/tests/qemu-iotests/191
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test commit block job where top has two parents
#
diff --git a/tests/qemu-iotests/192 b/tests/qemu-iotests/192
index d2ba55dd90..d809187fca 100755
--- a/tests/qemu-iotests/192
+++ b/tests/qemu-iotests/192
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test NBD export with -incoming (non-shared storage migration use case from
# libvirt)
diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
index 9dc1bd3510..b668120ebb 100755
--- a/tests/qemu-iotests/194
+++ b/tests/qemu-iotests/194
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw migration quick
#
# Copyright (C) 2017 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/195 b/tests/qemu-iotests/195
index 48984b7ac1..dab6e0fe9c 100755
--- a/tests/qemu-iotests/195
+++ b/tests/qemu-iotests/195
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test change-backing-file command
#
diff --git a/tests/qemu-iotests/196 b/tests/qemu-iotests/196
index e8fcf37273..2451515094 100755
--- a/tests/qemu-iotests/196
+++ b/tests/qemu-iotests/196
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick migration
#
# Test clearing unknown autoclear_features flag by qcow2 after
# migration. This test mimics migration to older qemu.
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index 95f05b0e34..74e751d06f 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for copy-on-read into qcow2
#
diff --git a/tests/qemu-iotests/198 b/tests/qemu-iotests/198
index fb0d5a29d3..b4c498015f 100755
--- a/tests/qemu-iotests/198
+++ b/tests/qemu-iotests/198
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test commit of encrypted qcow2 files
#
diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
index 40774eed74..9cb8c55956 100755
--- a/tests/qemu-iotests/199
+++ b/tests/qemu-iotests/199
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw migration
#
# Tests for dirty bitmaps postcopy migration.
#
diff --git a/tests/qemu-iotests/200 b/tests/qemu-iotests/200
index 59f7854b9f..c72cd72e56 100755
--- a/tests/qemu-iotests/200
+++ b/tests/qemu-iotests/200
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Block job co-routine race condition test.
#
diff --git a/tests/qemu-iotests/201 b/tests/qemu-iotests/201
index 483eb189c5..412a6f1834 100755
--- a/tests/qemu-iotests/201
+++ b/tests/qemu-iotests/201
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw migration
#
# Test savevm and loadvm after live migration with postcopy flag
#
diff --git a/tests/qemu-iotests/202 b/tests/qemu-iotests/202
index 920a8683ef..5f8afc85bd 100755
--- a/tests/qemu-iotests/202
+++ b/tests/qemu-iotests/202
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Copyright (C) 2017 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/203 b/tests/qemu-iotests/203
index 49eff5d405..3e2e35ad6b 100755
--- a/tests/qemu-iotests/203
+++ b/tests/qemu-iotests/203
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw auto migration
#
# Copyright (C) 2017 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/204 b/tests/qemu-iotests/204
index abb73dc381..e528b796f6 100755
--- a/tests/qemu-iotests/204
+++ b/tests/qemu-iotests/204
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test corner cases with unusual block geometries
#
diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
index 43432cb599..c0e107328f 100755
--- a/tests/qemu-iotests/205
+++ b/tests/qemu-iotests/205
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for qmp command nbd-server-remove.
#
diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206
index e2b50ae24d..4cf41eca7f 100755
--- a/tests/qemu-iotests/206
+++ b/tests/qemu-iotests/206
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test qcow2 and file image creation
#
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index 3d9c1208ca..813cfe7a27 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test ssh image creation
#
diff --git a/tests/qemu-iotests/208 b/tests/qemu-iotests/208
index 1c3fc8c7fd..5ef7bc3caa 100755
--- a/tests/qemu-iotests/208
+++ b/tests/qemu-iotests/208
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Copyright (C) 2018 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/209 b/tests/qemu-iotests/209
index 65c1a1e70a..2dd0bbb905 100755
--- a/tests/qemu-iotests/209
+++ b/tests/qemu-iotests/209
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for NBD BLOCK_STATUS extension
#
diff --git a/tests/qemu-iotests/210 b/tests/qemu-iotests/210
index e49896e23d..f03c13c186 100755
--- a/tests/qemu-iotests/210
+++ b/tests/qemu-iotests/210
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test luks and file image creation
#
diff --git a/tests/qemu-iotests/211 b/tests/qemu-iotests/211
index 163994d559..e66737970f 100755
--- a/tests/qemu-iotests/211
+++ b/tests/qemu-iotests/211
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test VDI and file image creation
#
diff --git a/tests/qemu-iotests/212 b/tests/qemu-iotests/212
index 800f92dd84..b7a53949f8 100755
--- a/tests/qemu-iotests/212
+++ b/tests/qemu-iotests/212
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test parallels and file image creation
#
diff --git a/tests/qemu-iotests/213 b/tests/qemu-iotests/213
index 1eee45276a..c7b5c0e7e2 100755
--- a/tests/qemu-iotests/213
+++ b/tests/qemu-iotests/213
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test vhdx and file image creation
#
diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214
index af677d90b8..4c0e8f2b48 100755
--- a/tests/qemu-iotests/214
+++ b/tests/qemu-iotests/214
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# Test qcow2 image compression
#
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index f99bae78c7..d464596f14 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for copy-on-read into qcow2, using the COR filter driver
#
diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216
index 372f042d3e..7374bede19 100755
--- a/tests/qemu-iotests/216
+++ b/tests/qemu-iotests/216
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Copy-on-read tests using a COR filter node
#
diff --git a/tests/qemu-iotests/217 b/tests/qemu-iotests/217
index d89116ccad..df370549f8 100755
--- a/tests/qemu-iotests/217
+++ b/tests/qemu-iotests/217
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# I/O errors when working with internal qcow2 snapshots, and repairing
# the result
diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218
index 1325ba9eaa..c8cca23161 100755
--- a/tests/qemu-iotests/218
+++ b/tests/qemu-iotests/218
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# This test covers what happens when a mirror block job is cancelled
# in various phases of its existence.
diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index b8774770c4..9523627451 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Copyright (C) 2018 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/220 b/tests/qemu-iotests/220
index a9259b7127..126749c707 100755
--- a/tests/qemu-iotests/220
+++ b/tests/qemu-iotests/220
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto
#
# max limits on compression in huge qcow2 files
#
diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221
index 0e9096fec7..2db84a3962 100755
--- a/tests/qemu-iotests/221
+++ b/tests/qemu-iotests/221
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qemu-img vs. unaligned images
# (See also 253, which is the O_DIRECT version)
diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222
index bf1718e179..f1ee316abb 100755
--- a/tests/qemu-iotests/222
+++ b/tests/qemu-iotests/222
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# This test covers the basic fleecing workflow, which provides a
# point-in-time snapshot of a node that can be queried over NBD.
diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223
index 56fbc5fb09..18ff35ac3a 100755
--- a/tests/qemu-iotests/223
+++ b/tests/qemu-iotests/223
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test reading dirty bitmap over NBD
#
diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
index e91fb26fd8..40e6beaa07 100755
--- a/tests/qemu-iotests/224
+++ b/tests/qemu-iotests/224
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test json:{} filenames with qemu-internal BDSs
# (the one of commit, to be precise)
diff --git a/tests/qemu-iotests/225 b/tests/qemu-iotests/225
index c9a334c7e9..c25ac26949 100755
--- a/tests/qemu-iotests/225
+++ b/tests/qemu-iotests/225
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test vmdk backing file correlation
#
diff --git a/tests/qemu-iotests/226 b/tests/qemu-iotests/226
index c1e1fb2b1c..6a9adb4a0b 100755
--- a/tests/qemu-iotests/226
+++ b/tests/qemu-iotests/226
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto quick
#
# This test covers expected filetypes for the file, host_cdrom and
# host_device drivers.
diff --git a/tests/qemu-iotests/227 b/tests/qemu-iotests/227
index 637d7c3726..7e45a47ac6 100755
--- a/tests/qemu-iotests/227
+++ b/tests/qemu-iotests/227
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test query-blockstats with different ways to create a BB
#
diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
index 64bc82ee23..5e723d342c 100755
--- a/tests/qemu-iotests/228
+++ b/tests/qemu-iotests/228
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test for when a backing file is considered overridden (thus, a
# json:{} filename is generated for the overlay) and when it is not
diff --git a/tests/qemu-iotests/229 b/tests/qemu-iotests/229
index 866168b236..9b8ecee2a7 100755
--- a/tests/qemu-iotests/229
+++ b/tests/qemu-iotests/229
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: auto quick
#
# Test for force canceling a running blockjob that is paused in
# an error state.
diff --git a/tests/qemu-iotests/231 b/tests/qemu-iotests/231
index c0b053ac30..0f66d0ca36 100755
--- a/tests/qemu-iotests/231
+++ b/tests/qemu-iotests/231
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test legacy and modern option parsing for rbd/ceph. This will not
# actually connect to a ceph server, but rather looks for the appropriate
diff --git a/tests/qemu-iotests/232 b/tests/qemu-iotests/232
index 685356ac3b..b30faaa218 100755
--- a/tests/qemu-iotests/232
+++ b/tests/qemu-iotests/232
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test for auto-read-only
#
diff --git a/tests/qemu-iotests/233 b/tests/qemu-iotests/233
index a5c17c3963..7ce5764903 100755
--- a/tests/qemu-iotests/233
+++ b/tests/qemu-iotests/233
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test NBD TLS certificate / authorization integration
#
diff --git a/tests/qemu-iotests/234 b/tests/qemu-iotests/234
index 324c1549fd..e443fc59cc 100755
--- a/tests/qemu-iotests/234
+++ b/tests/qemu-iotests/234
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: quick migration
#
# Copyright (C) 2018 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235
index 760826128e..7517f78626 100755
--- a/tests/qemu-iotests/235
+++ b/tests/qemu-iotests/235
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: quick
#
# Simple mirror test
#
diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236
index 8ce927a16c..75b63ad4cf 100755
--- a/tests/qemu-iotests/236
+++ b/tests/qemu-iotests/236
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: quick
#
# Test bitmap merges.
#
diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237
index 50ba364a3e..4068298f8c 100755
--- a/tests/qemu-iotests/237
+++ b/tests/qemu-iotests/237
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test vmdk and file image creation
#
diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238
index d4e060228c..662ba94131 100755
--- a/tests/qemu-iotests/238
+++ b/tests/qemu-iotests/238
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: quick
#
# Regression test for throttle group member unregister segfault with iothread
#
diff --git a/tests/qemu-iotests/239 b/tests/qemu-iotests/239
index b0991ffe59..4f0037148d 100755
--- a/tests/qemu-iotests/239
+++ b/tests/qemu-iotests/239
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test case for dmg
#
diff --git a/tests/qemu-iotests/240 b/tests/qemu-iotests/240
index 8b4337b58d..d9428e50bc 100755
--- a/tests/qemu-iotests/240
+++ b/tests/qemu-iotests/240
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# Test hot plugging and unplugging with iothreads
#
diff --git a/tests/qemu-iotests/241 b/tests/qemu-iotests/241
index 8dae8d39e4..c962c8b607 100755
--- a/tests/qemu-iotests/241
+++ b/tests/qemu-iotests/241
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qemu-nbd vs. unaligned images
#
diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index 97617876bc..757ac569bc 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test for qcow2 bitmap printed information
#
diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243
index a61852f6d9..160ea02538 100755
--- a/tests/qemu-iotests/243
+++ b/tests/qemu-iotests/243
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qcow2 preallocation
#
diff --git a/tests/qemu-iotests/244 b/tests/qemu-iotests/244
index 2ec1815e6f..037a961422 100755
--- a/tests/qemu-iotests/244
+++ b/tests/qemu-iotests/244
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qcow2 with external data files
#
diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index 1001275a44..9fc1e3e0c8 100755
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test cases for the QMP 'x-blockdev-reopen' command
#
diff --git a/tests/qemu-iotests/246 b/tests/qemu-iotests/246
index 59a216a839..71a17a024d 100755
--- a/tests/qemu-iotests/246
+++ b/tests/qemu-iotests/246
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test persistent bitmap resizing.
#
diff --git a/tests/qemu-iotests/247 b/tests/qemu-iotests/247
index 87e37b39e2..70b9a42a33 100755
--- a/tests/qemu-iotests/247
+++ b/tests/qemu-iotests/247
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test for auto-read-only with commit block job
#
diff --git a/tests/qemu-iotests/248 b/tests/qemu-iotests/248
index 68c374692e..a4af7b99df 100755
--- a/tests/qemu-iotests/248
+++ b/tests/qemu-iotests/248
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test resume mirror after auto pause on ENOSPC
#
diff --git a/tests/qemu-iotests/249 b/tests/qemu-iotests/249
index 2b99c9789e..623acdb97c 100755
--- a/tests/qemu-iotests/249
+++ b/tests/qemu-iotests/249
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test that a backing image is put back in read-only mode after
# block-commit (both when it fails and when it succeeds).
diff --git a/tests/qemu-iotests/250 b/tests/qemu-iotests/250
index 9bb6b94d74..b9ee43214a 100755
--- a/tests/qemu-iotests/250
+++ b/tests/qemu-iotests/250
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test big discard in qcow2 shrink
#
diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
index 7918ba3559..b0900898fa 100755
--- a/tests/qemu-iotests/251
+++ b/tests/qemu-iotests/251
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test qemu-img convert --salvage
#
diff --git a/tests/qemu-iotests/252 b/tests/qemu-iotests/252
index 83280c1715..a6df22ff3f 100755
--- a/tests/qemu-iotests/252
+++ b/tests/qemu-iotests/252
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto backing quick
#
# Tests for rebasing COW images that require zero cluster support
#
diff --git a/tests/qemu-iotests/253 b/tests/qemu-iotests/253
index d88d5afa45..35039d20a8 100755
--- a/tests/qemu-iotests/253
+++ b/tests/qemu-iotests/253
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qemu-img vs. unaligned images; O_DIRECT version
# (Originates from 221)
diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index ee66c986db..f4965d343b 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw backing quick
#
# Test external snapshot with bitmap copying and moving.
#
diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
index 4a4818bafb..2cb2a98f7c 100755
--- a/tests/qemu-iotests/255
+++ b/tests/qemu-iotests/255
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test commit job graph modifications while requests are active
#
diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
index e34074c83e..a34f7f7785 100755
--- a/tests/qemu-iotests/256
+++ b/tests/qemu-iotests/256
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw auto quick
#
# Test incremental/backup across iothread contexts
#
diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257
index 004a433b8b..ce1918fa37 100755
--- a/tests/qemu-iotests/257
+++ b/tests/qemu-iotests/257
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test bitmap-sync backups (incremental, differential, and partials)
#
diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258
index 091755a45c..d09174912b 100755
--- a/tests/qemu-iotests/258
+++ b/tests/qemu-iotests/258
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Very specific tests for adjacent commit/stream block jobs
#
diff --git a/tests/qemu-iotests/259 b/tests/qemu-iotests/259
index 62e29af05f..76cde429c4 100755
--- a/tests/qemu-iotests/259
+++ b/tests/qemu-iotests/259
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test generic image creation fallback (by using NBD)
#
diff --git a/tests/qemu-iotests/260 b/tests/qemu-iotests/260
index 30c0de380d..f2de0dbf87 100755
--- a/tests/qemu-iotests/260
+++ b/tests/qemu-iotests/260
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Tests for temporary external snapshot when we have bitmaps.
#
diff --git a/tests/qemu-iotests/261 b/tests/qemu-iotests/261
index ddcb04f285..e8e61f8d6e 100755
--- a/tests/qemu-iotests/261
+++ b/tests/qemu-iotests/261
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test case for qcow2's handling of extra data in snapshot table entries
# (and more generally, how certain cases of broken snapshot tables are
diff --git a/tests/qemu-iotests/262 b/tests/qemu-iotests/262
index 8835dce7be..6a9925bcac 100755
--- a/tests/qemu-iotests/262
+++ b/tests/qemu-iotests/262
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick migration
#
# Copyright (C) 2019 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
index d2c030fae9..54d009ce24 100755
--- a/tests/qemu-iotests/263
+++ b/tests/qemu-iotests/263
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test encrypted write that crosses cluster boundary of two unallocated clusters
# Based on 188
diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264
index 879123a343..9ef7fe31de 100755
--- a/tests/qemu-iotests/264
+++ b/tests/qemu-iotests/264
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw
#
# Test nbd reconnect
#
diff --git a/tests/qemu-iotests/265 b/tests/qemu-iotests/265
index 00f2ec769e..f796361903 100755
--- a/tests/qemu-iotests/265
+++ b/tests/qemu-iotests/265
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test reverse-ordered qcow2 writes on a sub-cluster level
#
diff --git a/tests/qemu-iotests/266 b/tests/qemu-iotests/266
index 91bdf8729e..71ce81d0df 100755
--- a/tests/qemu-iotests/266
+++ b/tests/qemu-iotests/266
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test VPC and file image creation
#
diff --git a/tests/qemu-iotests/267 b/tests/qemu-iotests/267
index 3146273eef..4fd18fefdb 100755
--- a/tests/qemu-iotests/267
+++ b/tests/qemu-iotests/267
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick snapshot
#
# Test which nodes are involved in internal snapshots
#
diff --git a/tests/qemu-iotests/268 b/tests/qemu-iotests/268
index 78c3f4db3a..4fce2ef75f 100755
--- a/tests/qemu-iotests/268
+++ b/tests/qemu-iotests/268
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test write request with required alignment larger than the cluster size
#
diff --git a/tests/qemu-iotests/270 b/tests/qemu-iotests/270
index b9a12b908c..f49894149f 100755
--- a/tests/qemu-iotests/270
+++ b/tests/qemu-iotests/270
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing quick
#
# Test large write to a qcow2 image
#
diff --git a/tests/qemu-iotests/272 b/tests/qemu-iotests/272
index c2f782d47b..015347a0c9 100755
--- a/tests/qemu-iotests/272
+++ b/tests/qemu-iotests/272
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test compressed write to a qcow2 image at an offset above 4 GB
#
diff --git a/tests/qemu-iotests/273 b/tests/qemu-iotests/273
index 00ff79bcf8..e537c0be62 100755
--- a/tests/qemu-iotests/273
+++ b/tests/qemu-iotests/273
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: backing quick
#
# Test multiple blockdev-snapshot calls with 'backing': null
#
diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277
index a4ef1a38c6..9b0b2317c4 100755
--- a/tests/qemu-iotests/277
+++ b/tests/qemu-iotests/277
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test NBD client reconnection
#
diff --git a/tests/qemu-iotests/279 b/tests/qemu-iotests/279
index 75a4747e6b..2190adf995 100755
--- a/tests/qemu-iotests/279
+++ b/tests/qemu-iotests/279
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw backing quick
#
# Test qemu-img --backing-chain --image-opts
#
diff --git a/tests/qemu-iotests/280 b/tests/qemu-iotests/280
index 69288fdd0e..5a3fca906e 100755
--- a/tests/qemu-iotests/280
+++ b/tests/qemu-iotests/280
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw migration quick
#
# Copyright (C) 2019 Red Hat, Inc.
#
diff --git a/tests/qemu-iotests/281 b/tests/qemu-iotests/281
index 0bf973bca6..956698083f 100755
--- a/tests/qemu-iotests/281
+++ b/tests/qemu-iotests/281
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: rw quick
#
# Test cases for blockdev + IOThread interactions
#
diff --git a/tests/qemu-iotests/282 b/tests/qemu-iotests/282
index 081eb12080..dfc61a65a8 100755
--- a/tests/qemu-iotests/282
+++ b/tests/qemu-iotests/282
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw img quick
#
# Test qemu-img file cleanup for LUKS when using a non-UTF8 secret
#
diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283
index 55b7cff953..e0eefa685c 100755
--- a/tests/qemu-iotests/283
+++ b/tests/qemu-iotests/283
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
+# group: auto quick
#
# Test for backup-top filter permission activation failure
#
diff --git a/tests/qemu-iotests/284 b/tests/qemu-iotests/284
index 071e89b33e..d0e46d064e 100755
--- a/tests/qemu-iotests/284
+++ b/tests/qemu-iotests/284
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw
#
# Test ref count checks on encrypted images
#
diff --git a/tests/qemu-iotests/286 b/tests/qemu-iotests/286
index f14445ba4a..1fd24d5f74 100755
--- a/tests/qemu-iotests/286
+++ b/tests/qemu-iotests/286
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# Test qemu-img snapshot -l
#
diff --git a/tests/qemu-iotests/288 b/tests/qemu-iotests/288
index 6c62065aef..47aca6592a 100755
--- a/tests/qemu-iotests/288
+++ b/tests/qemu-iotests/288
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: quick
#
# qemu-img measure tests for LUKS images
#
diff --git a/tests/qemu-iotests/289 b/tests/qemu-iotests/289
index 1c11d4030e..2710ce882b 100755
--- a/tests/qemu-iotests/289
+++ b/tests/qemu-iotests/289
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw quick
#
# qcow2 v3-exclusive error path testing
# (026 tests paths common to v2 and v3)
diff --git a/tests/qemu-iotests/290 b/tests/qemu-iotests/290
index 776b65e915..d3d004fb56 100755
--- a/tests/qemu-iotests/290
+++ b/tests/qemu-iotests/290
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
+# group: rw auto quick
#
# Test how 'qemu-io -c discard' behaves on v2 and v3 qcow2 images
#
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 06/10] iotests: add testfinder.py
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 05/10] iotests: define group in each iotest Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 16:56 ` Kevin Wolf
2020-04-21 7:35 ` [PATCH v3 07/10] iotests: add testenv.py Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
9 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Add python script with new logic of searching for tests:
Current ./check behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong
to any group, like 142)
Behavior of new test:
- group file is dropped
- tests are searched by file-name instead of group file, so it's not
needed more to "register the test", just create it with name
*-test. Old names like [0-9][0-9][0-9] are supported too, but not
recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional
groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of
commenting tests in old 'group' file you now can add them to
disabled group with help of 'group.local' file
- selecting test ranges like 5-15 are not supported more
Benefits:
- no rebase conflicts in group file on patch porting from branch to
branch
- no conflicts in upstream, when different series want to occupy same
test number
- meaningful names for test files
For example, with digital number, when some person wants to add some
test about block-stream, he most probably will just create a new
test. But if there would be test-block-stream test already, he will
at first look at it and may be just add a test-case into it.
And anyway meaningful names are better.
This commit just adds class, which is unused now, and will be used in
further patches, to finally substitute ./check selecting tests logic.
Still, the documentation changed like new behavior is already here.
Let's live with this small inconsistency for the following few commits,
until final change.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
docs/devel/testing.rst | 52 +++++++++-
tests/qemu-iotests/testfinder.py | 167 +++++++++++++++++++++++++++++++
2 files changed, 218 insertions(+), 1 deletion(-)
create mode 100755 tests/qemu-iotests/testfinder.py
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 770a987ea4..6c9d5b126b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -153,7 +153,7 @@ check-block
-----------
``make check-block`` runs a subset of the block layer iotests (the tests that
-are in the "auto" group in ``tests/qemu-iotests/group``).
+are in the "auto" group).
See the "QEMU iotests" section below for more information.
GCC gcov support
@@ -267,6 +267,56 @@ another application on the host may have locked the file, possibly leading to a
test failure. If using such devices are explicitly desired, consider adding
``locking=off`` option to disable image locking.
+Test case groups
+----------------
+
+Test may belong to some groups, you may define it in the comment inside the
+test. By convention, test groups are listed in the second line of the test
+file, after "#!/..." line, like this:
+
+.. code::
+
+ #!/usr/bin/env python3
+ # group: auto quick
+ #
+ ...
+
+Additional way of defining groups is creating tests/qemu-iotests/group.local
+file. This should be used only for downstream (this file should never appear
+in upstream). This file may be used for defining some downstream test groups
+or for temporary disable tests, like this:
+
+.. code::
+
+ # groups for some company downstream process
+ #
+ # ci - tests to run on build
+ # down - our downstream tests, not for upstream
+ #
+ # Format of each line is:
+ # TEST_NAME TEST_GROUP [TEST_GROUP ]...
+
+ 013 ci
+ 210 disabled
+ 215 disabled
+ our-ugly-workaround-test down ci
+
+The following groups are defined:
+
+- quick : Tests in this group should finish within some few seconds.
+
+- img : Tests in this group can be used to excercise the qemu-img tool.
+
+- auto : Tests in this group are used during "make check" and should be
+ runnable in any case. That means they should run with every QEMU binary
+ (also non-x86), with every QEMU configuration (i.e. must not fail if
+ an optional feature is not compiled in - but reporting a "skip" is ok),
+ work at least with the qcow2 file format, work with all kind of host
+ filesystems and users (e.g. "nobody" or "root") and must not take too
+ much memory and disk space (since CI pipelines tend to fail otherwise).
+
+- disabled : Tests in this group are disabled and ignored by check.
+
.. _docker-ref:
Docker based tests
diff --git a/tests/qemu-iotests/testfinder.py b/tests/qemu-iotests/testfinder.py
new file mode 100755
index 0000000000..1da28564c0
--- /dev/null
+++ b/tests/qemu-iotests/testfinder.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+#
+# Parse command line options to define set of tests to run.
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import sys
+import glob
+import argparse
+import re
+from collections import defaultdict
+from contextlib import contextmanager
+
+
+@contextmanager
+def chdir(path):
+ if path is None:
+ yield
+ return
+
+ saved_dir = os.getcwd()
+ os.chdir(path)
+ try:
+ yield
+ finally:
+ os.chdir(saved_dir)
+
+
+class TestFinder:
+ @staticmethod
+ def create_argparser():
+ p = argparse.ArgumentParser(description="Select set of tests",
+ add_help=False, usage=argparse.SUPPRESS)
+
+ p.add_argument('-g', metavar='group1,...', dest='groups',
+ help='include tests from these groups')
+ p.add_argument('-x', metavar='group1,...', dest='exclude_groups',
+ help='exclude tests from these groups')
+ p.add_argument('tests', metavar='TEST_FILES', nargs='*',
+ help='tests to run')
+
+ return p
+
+ argparser = create_argparser.__func__()
+
+ def __init__(self, test_dir=None):
+ self.groups = defaultdict(set)
+
+ with chdir(test_dir):
+ self.all_tests = glob.glob('[0-9][0-9][0-9]')
+ self.all_tests += [f for f in glob.iglob('*-test')]
+
+ for t in self.all_tests:
+ with open(t) as f:
+ for line in f:
+ if line.startswith('# group: '):
+ for g in line.split()[2:]:
+ self.groups[g].add(t)
+
+ def add_group_file(self, fname):
+ with open(fname) as f:
+ for line in f:
+ line = line.strip()
+
+ if (not line) or line[0] == '#':
+ continue
+
+ words = line.split()
+ test_file = words[0]
+ groups = words[1:]
+
+ if test_file not in self.all_tests:
+ print('Warning: {}: "{}" test is not found. '
+ 'Skip.'.format(fname, test_file))
+ continue
+
+ for g in groups:
+ self.groups[g].add(test_file)
+
+ def find_tests(self, groups=None, exclude_groups=None, tests=None):
+ """
+ 1. Take all tests from @groups,
+ or just all tests if @groups is None and @tests is None
+ or nothing if @groups is None and @tests is not None
+ 2. Drop tests, which are in at least one of @exclude_groups or in
+ 'disabled' group (if 'disabled' is not listed in @groups)
+ 3. Add tests from @tests
+ """
+ if groups is None:
+ groups = []
+ if exclude_groups is None:
+ exclude_groups = []
+ if tests is None:
+ tests = []
+
+ if groups:
+ res = set().union(*(self.groups[g] for g in groups))
+ elif tests:
+ res = set()
+ else:
+ res = set(self.all_tests)
+
+ if 'disabled' not in groups and 'disabled' not in exclude_groups:
+ exclude_groups.append('disabled')
+
+ res = res.difference(*(self.groups[g] for g in exclude_groups))
+
+ # We want to add @tests. But for compatibility with old test names,
+ # we should convert any number < 100 to number padded by
+ # leading zeroes, like 1 -> 001 and 23 -> 023.
+ for t in tests:
+ if re.fullmatch(r'\d{1,2}', t):
+ res.add('0' * (3 - len(t)) + t)
+ else:
+ res.add(t)
+
+ return sorted(res)
+
+ def find_tests_argv(self, argv):
+ args, remaining = self.argparser.parse_known_args(argv)
+
+ if args.groups is not None:
+ args.groups = args.groups.split(',')
+
+ if args.exclude_groups is not None:
+ args.exclude_groups = args.exclude_groups.split(',')
+
+ return self.find_tests(**vars(args)), remaining
+
+
+def find_tests(argv, test_dir=None):
+ tf = TestFinder(test_dir)
+
+ if test_dir is None:
+ group_local = 'group.local'
+ else:
+ group_local = os.path.join(test_dir, 'group.local')
+ if os.path.isfile(group_local):
+ tf.add_group_file(group_local)
+
+ return tf.find_tests_argv(argv)
+
+
+if __name__ == '__main__':
+ if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
+ TestFinder.argparser.print_help()
+ exit()
+
+ tests, remaining_argv = find_tests(sys.argv[1:])
+ print('\n'.join(tests))
+ if remaining_argv:
+ print('\nUnhandled options: ', remaining_argv)
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 07/10] iotests: add testenv.py
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 06/10] iotests: add testfinder.py Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 08/10] iotests: add testrunner.py Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
9 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Add TestEnv class, which will handle test environment in a new python
iotests running framework.
Difference with current ./check interface:
- -v (verbose) option dropped, as it is unused
- -xdiff option is dropped, until somebody complains that it is needed
- same for -n option
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
RFC question here:
What about moving to classic double-dash long options (and short
one-dash-one-letter options)?
So, to convert
-qcow2 --> -f qcow2
-misalign --> --misalign
etc.
This may be done later, or I can do it in next version, if all agree
that it's good idea.
tests/qemu-iotests/testenv.py | 332 ++++++++++++++++++++++++++++++++++
1 file changed, 332 insertions(+)
create mode 100755 tests/qemu-iotests/testenv.py
diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
new file mode 100755
index 0000000000..d1c2d41974
--- /dev/null
+++ b/tests/qemu-iotests/testenv.py
@@ -0,0 +1,332 @@
+#!/usr/bin/env python3
+#
+# Parse command line options to manage test environment variables.
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import sys
+import tempfile
+from pathlib import Path
+import shutil
+import collections
+import subprocess
+import argparse
+
+
+def get_default_machine(qemu_prog):
+ outp = subprocess.run([qemu_prog, '-machine', 'help'], check=True,
+ text=True, stdout=subprocess.PIPE).stdout
+
+ machines = outp.split('\n')
+ default_machine = next(m for m in machines if m.endswith(' (default)'))
+ default_machine = default_machine.split(' ', 1)[0]
+
+ alias_suf = ' (alias of {})'.format(default_machine)
+ alias = next((m for m in machines if m.endswith(alias_suf)), None)
+ if alias is not None:
+ default_machine = alias.split(' ', 1)[0]
+
+ return default_machine
+
+
+""" EnvVarDescriptor
+name - variable name in upper case.
+inherit - variable defaults to same variable from os.environ.
+default - default value for the variable. None means absence by default.
+help - description. Unused now, just in-place documentation. May take place in
+ help message at some point.
+"""
+EnvVarDescriptor = collections.namedtuple('EnvVarDescriptor',
+ ('name', 'inherit', 'default',
+ 'help'))
+
+
+class TestEnv:
+ """
+ Manage system environment for running tests
+
+ The following variables are supported/provided. They are represented by
+ lower-cased TestEnv attributes.
+ """
+ env_descriptors = [EnvVarDescriptor(*d) for d in (
+ # name, inherit, default, description
+
+ # Directories
+ ('TEST_DIR', True, os.path.join(os.getcwd(), 'scratch'),
+ 'directory for all intermediate test files'),
+ ('SOCK_DIR', True, None, 'directory for unix sockets'),
+ ('SAMPLE_IMG_DIR', True, None, 'TODO: ???'),
+ ('OUTPUT_DIR', False, os.getcwd(), 'TODO: ???'),
+
+ # Binaries
+ ('PYTHON', False, '/usr/bin/python3 -B', 'python3 for bash tests'),
+ ('QEMU_PROG', True, None, 'qemu binary'),
+ ('QEMU_IMG_PROG', True, None, 'qemu-img binary'),
+ ('QEMU_IO_PROG', True, None, 'qemu-io binary'),
+ ('QEMU_NBD_PROG', True, None, 'qemu-nbd binary'),
+ ('SOCKET_SCM_HELPER', False, None, 'socket_scm_helper binary'),
+
+ # Options for binaries
+ # RFC: Interesting, that only IMG and NBD options may be passed by
+ # user, QEMU and IO options are only calculated. Looks inconsistent.
+ ('QEMU_OPTIONS', False, None, 'qemu command line arguments'),
+ ('QEMU_IMG_OPTIONS', True, None, 'qemu command line arguments'),
+ ('QEMU_IO_OPTIONS', False, None, 'qemu command line arguments'),
+ ('QEMU_NBD_OPTIONS', True, None, 'qemu command line arguments'),
+
+ ('IMGOPTS', False, None, 'options to pass to qemu-img create/convert'),
+
+ ('IMGFMT', False, 'raw', 'image format, set by cmdline'),
+ ('IMGPROTO', False, 'file', 'image protocol, set by cmdline'),
+ ('AIOMODE', False, 'threads', 'image protocol, set by cmdline'),
+ ('CACHEMODE', False, 'writeback', 'cache mode, set by cmdline'),
+
+ ('VALGRIND_QEMU', False, None, 'use valgrind, set by cmdline'),
+
+ # Helping variables, not passed by user, only calculated
+ ('CACHEMODE_IS_DEFAULT', False, None,
+ 'cache mode was not set by user'),
+ ('IMGFMT_GENERIC', False, None, 'TODO: ???'),
+ ('IMGOPTSSYNTAX', False, None, 'TODO: ???'),
+ ('IMGKEYSECRET', False, None, 'TODO: ???'),
+ ('QEMU_DEFAULT_MACHINE', False, None, 'TODO: ???'),
+ )]
+
+ @staticmethod
+ def create_argparser():
+ p = argparse.ArgumentParser(description="Test environment preparation",
+ add_help=False, usage=argparse.SUPPRESS)
+
+ p.add_argument('-d', dest='debug', action='store_true', help='debug')
+ p.add_argument('-misalign', action='store_true',
+ help='misalign memory allocations')
+
+ format_list = ['raw', 'bochs', 'parallels', 'qcow', 'qcow2', 'qed',
+ 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg']
+ g = p.add_argument_group(
+ 'image format options',
+ 'The following options sets IMGFMT environment variable. '
+ 'At most one chose is allowed, default is "raw"')
+ g = g.add_mutually_exclusive_group()
+ for fmt in format_list:
+ g.add_argument('-' + fmt, dest='imgfmt', action='store_const',
+ const=fmt)
+
+ protocol_list = ['file', 'rbd', 'sheepdoc', 'nbd', 'ssh', 'nfs',
+ 'vxhs']
+ g = p.add_argument_group(
+ 'image protocol options',
+ 'The following options sets IMGPROTO environment variably. '
+ 'At most one chose is allowed, default is "file"')
+ g = g.add_mutually_exclusive_group()
+ for prt in protocol_list:
+ g.add_argument('-' + prt, dest='imgproto', action='store_const',
+ const=prt)
+
+ g = p.add_mutually_exclusive_group()
+ g.add_argument('-nocache', dest='cachemode', action='store_const',
+ const='none', help='set cache mode "none" (O_DIRECT), '
+ 'sets CACHEMODE environment variable')
+ g.add_argument('-c', dest='cachemode',
+ help='sets CACHEMODE environment variable')
+
+ p.add_argument('-i', dest='aiomode', default='threads',
+ help='sets AIOMODE environment variable')
+
+ g = p.add_argument_group('bash tests options',
+ 'The following options are ignored by '
+ 'python tests. TODO: support them in '
+ 'iotests.py')
+ g.add_argument('-o', dest='imgopts',
+ help='options to pass to qemu-img create/convert, sets '
+ 'IMGOPTS environment variable')
+ p.add_argument('-valgrind', dest='VALGRIND_QEMU', action='store_const',
+ const='y', help='use valgrind, sets VALGRIND_QEMU '
+ 'environment variable')
+
+ return p
+
+ argparser = create_argparser.__func__()
+
+ def __contains__(self, item):
+ return hasattr(self, item)
+
+ def __setitem__(self, key, value):
+ setattr(self, key, value)
+
+ def __getitem__(self, key):
+ return getattr(self, key)
+
+ def setdefault(self, attr, value):
+ if attr not in self:
+ self[attr] = value
+
+ def init_paths(self):
+ self.build_iotests = os.getcwd()
+
+ if os.path.islink(sys.argv[0]):
+ # called from the build tree
+ self.source_iotests = os.path.dirname(os.readlink(sys.argv[0]))
+ else:
+ self.source_iotests = self.build_iotests
+
+ self.build_root = os.path.join(self.build_iotests, '..', '..')
+
+ def init_handle_argv(self, argv):
+ self.args, self.remaining_argv = self.argparser.parse_known_args(argv)
+
+ for k, v in vars(self.args).items():
+ if v is not None:
+ self[k] = v
+
+ def init_handle_env_descriptors(self):
+ for d in self.env_descriptors:
+ if d.name.lower() in self:
+ continue # set by command line argument
+
+ if d.inherit and d.name in os.environ:
+ self[d.name.lower()] = os.environ[d.name]
+ elif d.default is not None:
+ self[d.name.lower()] = d.default
+
+ def init_find_binaries(self):
+ self.setdefault('qemu_img_prog',
+ os.path.join(self.build_root, 'qemu-img'))
+ self.setdefault('qemu_io_prog',
+ os.path.join(self.build_root, 'qemu-io'))
+ self.setdefault('qemu_nbd_prog',
+ os.path.join(self.build_root, 'qemu-nbd'))
+
+ if 'qemu_prog' not in self:
+ arch = os.uname().machine
+ if 'ppc64' in arch:
+ arch = 'ppc64'
+ self.qemu_prog = os.path.join(self.build_root, arch + '-softmmu',
+ 'qemu-system-' + arch)
+
+ for b in [self.qemu_img_prog, self.qemu_io_prog, self.qemu_nbd_prog,
+ self.qemu_prog]:
+ if not os.path.exists(b):
+ exit('Not such file: ' + b)
+ if not os.access(b, os.X_OK):
+ exit('Not executable: ' + b)
+
+ helper_path = os.path.join(self.build_iotests, 'socket_scm_helper')
+ if os.access(helper_path, os.X_OK):
+ self.socket_scm_helper = helper_path
+
+ def __init__(self, argv):
+ """ Parse args and environment """
+
+ self.init_paths()
+ self.init_handle_argv(argv)
+
+ self.cachemode_is_default = 'false' if 'cachemode' in self else 'true'
+
+ self.init_handle_env_descriptors()
+ self.init_find_binaries()
+
+ self.imgfmt_generic = self.imgfmt not in ['bochs', 'cloop', 'dmg']
+ self.imgfmt_generic = 'true' if self.imgfmt_generic else 'false'
+
+ self.qemu_io_options = f'--cache {self.cachemode} --aio {self.aiomode}'
+ if self.misalign:
+ self.qemu_io_options += ' --misalign'
+
+ self.qemu_io_options_no_fmt = self.qemu_io_options
+
+ if self.imgfmt == 'luks':
+ self.imgoptssyntax = 'true'
+ self.imgkeysecret = '123456'
+ if 'imgopts' not in self:
+ self.imgopts = 'iter-time=10'
+ elif 'iter-time=' not in self.imgopts:
+ self.imgopts += ',iter-time=10'
+ else:
+ self.imgoptssyntax = 'false'
+ self.qemu_io_options += ' -f ' + self.imgfmt
+
+ self.qemu_options = '-nodefaults -display none -accel qtest'
+ if self.qemu_prog.endswith(('qemu-system-arm', 'qemu-system-aarch64')):
+ self.qemu_options += ' -machine virt'
+ elif self.qemu_prog.endswith('qemu-system-tricore'):
+ self.qemu_options += ' -machine tricore_testboard'
+
+ self.setdefault('sample_img_dir',
+ os.path.join(self.source_iotests, 'sample_images'))
+
+ self.qemu_default_machine = get_default_machine(self.qemu_prog)
+
+ self.tmp_sock_dir = False
+ if 'sock_dir' not in self:
+ self.sock_dir = tempfile.mkdtemp()
+ self.tmp_sock_dir = True
+
+ for d in (self.test_dir, self.sock_dir):
+ Path(d).mkdir(parents=True, exist_ok=True)
+
+ def close(self):
+ if self.tmp_sock_dir:
+ shutil.rmtree(self.sock_dir)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args):
+ self.close()
+
+ def get_env(self):
+ env = {}
+ for d in self.env_descriptors:
+ if d.name.lower() in self:
+ env[d.name] = self[d.name.lower()]
+
+ return env
+
+ def print_env(self):
+ template = """\
+QEMU -- "{QEMU_PROG}" {QEMU_OPTIONS}
+QEMU_IMG -- "{QEMU_IMG_PROG}" {QEMU_IMG_OPTIONS}
+QEMU_IO -- "{QEMU_IO_PROG}" {QEMU_IO_OPTIONS}
+QEMU_NBD -- "{QEMU_NBD_PROG}" {QEMU_NBD_OPTIONS}
+IMGFMT -- {IMGFMT}{imgopts}
+IMGPROTO -- {IMGPROTO}
+PLATFORM -- {platform}
+TEST_DIR -- {TEST_DIR}
+SOCK_DIR -- {SOCK_DIR}
+SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}"""
+
+ args = collections.defaultdict(str, self.get_env())
+
+ if 'IMGOPTS' in args:
+ args['imgopts'] = ' (' + args['IMGOPTS'] + ')'
+
+ u = os.uname()
+ args['platform'] = f'{u.sysname}/{u.machine} {u.nodename} {u.release}'
+
+ print(template.format_map(args))
+
+
+if __name__ == '__main__':
+ if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
+ TestEnv.argparser.print_help()
+ exit()
+
+ with TestEnv(sys.argv) as te:
+ te.print_env()
+ print('\nUnhandled options: ', te.remaining_argv)
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 08/10] iotests: add testrunner.py
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 07/10] iotests: add testenv.py Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:35 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:36 ` [PATCH v3 09/10] iotests: rewrite check into python Vladimir Sementsov-Ogievskiy
2020-04-21 7:36 ` [PATCH v3 10/10] iotests: rename 169 and 199 Vladimir Sementsov-Ogievskiy
9 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:35 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Add TestRunner class, which will run tests in a new python iotests
running framework.
There are some differences with current ./check behavior, most
significant are:
- Consider all tests self-executable, just run them, don't run python
by hand.
- Elapsed time is cached in json file
- Elapsed time precision increased a bit
- use python difflib instead of "diff -w", to ignore spaces at line
ends strip lines by hand. Do not ignore other spaces.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/testrunner.py | 334 +++++++++++++++++++++++++++++++
1 file changed, 334 insertions(+)
create mode 100644 tests/qemu-iotests/testrunner.py
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
new file mode 100644
index 0000000000..bb9bbe304b
--- /dev/null
+++ b/tests/qemu-iotests/testrunner.py
@@ -0,0 +1,334 @@
+# Class for actual tests running.
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import random
+from pathlib import Path
+import datetime
+import time
+import difflib
+import subprocess
+import collections
+import contextlib
+import json
+import argparse
+
+from testenv import TestEnv
+
+
+def silent_unlink(path: Path):
+ try:
+ path.unlink()
+ except OSError:
+ pass
+
+
+def file_diff(file1, file2):
+ with open(file1) as f1, open(file2) as f2:
+ # We want to ignore spaces at line ends. There are a lot of mess about
+ # it in iotests.
+ # TODO: fix all tests to not produce extra spaces, fix all .out files
+ # and use strict diff here!
+ seq1 = [line.rstrip() for line in f1]
+ seq2 = [line.rstrip() for line in f2]
+ return list(difflib.unified_diff(seq1, seq2, file1, file2))
+
+
+class LastElapsedTime:
+ """ Cache for elapsed time for tests, to show it during new test run
+
+ Use get() in any time. But, if use update you should then call save(),
+ or use update() inside with-block.
+ """
+ def __init__(self, cache_file, env):
+ self.env = env
+ self.cache_file = cache_file
+
+ try:
+ with open(cache_file) as f:
+ self.cache = json.load(f)
+ except (OSError, ValueError):
+ self.cache = {}
+
+ def get(self, test, default=None):
+ if test not in self.cache:
+ return default
+
+ if self.env.imgproto not in self.cache[test]:
+ return default
+
+ return self.cache[test][self.env.imgproto].get(self.env.imgfmt,
+ default)
+
+ def update(self, test, elapsed):
+ d = self.cache.setdefault(test, {})
+ d = d.setdefault(self.env.imgproto, {})
+ d[self.env.imgfmt] = elapsed
+
+ def save(self):
+ with open(self.cache_file, 'w') as f:
+ json.dump(self.cache, f)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, type, value, traceback):
+ self.save()
+
+
+TestResult = collections.namedtuple(
+ 'TestResult',
+ ['status', 'description', 'elapsed', 'diff', 'casenotrun'],
+ defaults=('', '', '', ''))
+
+
+class TestRunner:
+ @staticmethod
+ def colored_status(status):
+ if status == 'pass':
+ col = '\033[32m'
+ elif status == 'fail':
+ col = '\033[1m\033[31m'
+ elif status == 'not run':
+ col = '\033[33m'
+ else:
+ col = ''
+
+ return col + status + '\033[0m'
+
+ @staticmethod
+ def create_argparser():
+ p = argparse.ArgumentParser(description="Run tests",
+ add_help=False, usage=argparse.SUPPRESS)
+
+ p.add_argument('-makecheck', action='store_true',
+ help='pretty print output for make check')
+
+ return p
+
+ argparser = create_argparser.__func__()
+
+ def __init__(self, argv, env: TestEnv):
+ self.env = env
+ self.test_run_env = self.env.get_env()
+ if 'MALLOC_PERTURB_' not in os.environ and \
+ 'MALLOC_PERTURB_' not in self.test_run_env:
+ x = random.randrange(1, 255)
+ self.test_run_env['MALLOC_PERTURB_'] = str(x)
+
+ args, self.remaining_argv = self.argparser.parse_known_args(argv)
+ self.makecheck = args.makecheck
+
+ self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
+
+ def __enter__(self):
+ self._stack = contextlib.ExitStack()
+ self._stack.enter_context(self.env)
+ self._stack.enter_context(self.last_elapsed)
+ return self
+
+ def __exit__(self, *args):
+ self._stack.close()
+
+ def test_print_one_line(self, test, starttime, endtime=None, status='...',
+ lasttime=None, thistime=None, description='',
+ test_field_width=None, end='\n'):
+ """ Print short test info before/after test run """
+ test = os.path.basename(test)
+
+ if test_field_width is None:
+ test_field_width = 8
+
+ if self.makecheck and status != '...':
+ if status and status != 'pass':
+ status = f' [{status}]'
+ else:
+ status = ''
+
+ print(f' TEST iotest-{self.env.imgfmt}: {test}{status}')
+ return
+
+ if lasttime:
+ lasttime = f' (last: {lasttime}s)'
+ else:
+ lasttime = ''
+ if thistime:
+ thistime = f'{thistime:.1f}s'
+ else:
+ thistime = '...'
+
+ if endtime:
+ endtime = f'[{endtime}]'
+ else:
+ endtime = ''
+
+ if status == 'pass':
+ col = '\033[32m'
+ elif status == 'fail':
+ col = '\033[1m\033[31m'
+ elif status == 'not run':
+ col = '\033[33m'
+ else:
+ col = ''
+
+ col_end = '\033[0m'
+
+ print(f'{test:{test_field_width}} {col}{status:10}{col_end} '
+ f'[{starttime}] {endtime:13}{thistime:5} {lasttime:14} '
+ f'{description}', end=end)
+
+ def find_reference(self, test):
+ if self.env.cachemode == 'none':
+ ref = f'{test}.out.nocache'
+ if os.path.isfile(ref):
+ return ref
+
+ ref = f'{test}.out.{self.env.imgfmt}'
+ if os.path.isfile(ref):
+ return ref
+
+ ref = f'{test}.{self.env.qemu_default_machine}.out'
+ if os.path.isfile(ref):
+ return ref
+
+ return f'{test}.out'
+
+ def do_run_test(self, test):
+ f_test = Path(test)
+ f_bad = Path(f_test.name + '.out.bad')
+ f_notrun = Path(f_test.name + '.notrun')
+ f_casenotrun = Path(f_test.name + '.casenotrun')
+ f_reference = Path(self.find_reference(test))
+
+ if not f_test.exists():
+ return TestResult(status='fail',
+ description=f'No such test file: {f_test}')
+
+ if not os.access(str(f_test), os.X_OK):
+ exit(f'Not executable: {f_test}')
+
+ if not f_reference.exists():
+ return TestResult(status='fail',
+ description='No qualified output '
+ f'(expected {f_reference})')
+
+ for f in (f_bad, f_notrun, f_casenotrun):
+ silent_unlink(f)
+
+ args = [str(f_test.resolve())]
+ if self.env.debug:
+ args.append('-d')
+
+ env = os.environ.copy()
+ env.update(self.test_run_env)
+
+ t0 = time.time()
+ with f_bad.open('w') as f:
+ ret = subprocess.run(args, cwd=str(f_test.parent), env=env,
+ stdout=f, stderr=subprocess.STDOUT).returncode
+ elapsed = round(time.time() - t0, 1)
+
+ if ret != 0:
+ return TestResult(status='fail', elapsed=elapsed,
+ description=f'failed, exit status {ret}',
+ diff=file_diff(str(f_reference), str(f_bad)))
+
+ if f_notrun.exists():
+ return TestResult(status='not run', elapsed=elapsed,
+ description=f_notrun.read_text().strip())
+
+ casenotrun = False
+ if f_casenotrun.exists():
+ casenotrun = f_casenotrun.read_text()
+
+ diff = file_diff(str(f_reference), str(f_bad))
+ if diff:
+ return TestResult(status='fail', elapsed=elapsed,
+ description='output mismatch (see {f_bad})',
+ diff=diff, casenotrun=casenotrun)
+ else:
+ f_bad.unlink()
+ self.last_elapsed.update(test, elapsed)
+ return TestResult(status='pass', elapsed=elapsed,
+ casenotrun=casenotrun)
+
+ def run_test(self, test, test_field_width=None):
+ last_el = self.last_elapsed.get(test)
+ start = datetime.datetime.now().strftime('%H:%M:%S')
+
+ self.test_print_one_line(test=test, starttime=start, lasttime=last_el,
+ end='\r', test_field_width=test_field_width)
+
+ res = self.do_run_test(test)
+
+ end = datetime.datetime.now().strftime('%H:%M:%S')
+ self.test_print_one_line(test=test, status=res.status,
+ starttime=start, endtime=end,
+ lasttime=last_el, thistime=res.elapsed,
+ description=res.description,
+ test_field_width=test_field_width)
+
+ if res.casenotrun:
+ print(res.casenotrun)
+
+ return res
+
+ def run_tests(self, tests):
+ n_run = 0
+ failed = []
+ notrun = []
+ casenotrun = []
+
+ if not self.makecheck:
+ self.env.print_env()
+ print()
+
+ test_field_width = max(len(os.path.basename(t)) for t in tests) + 2
+
+ for t in tests:
+ name = os.path.basename(t)
+ res = self.run_test(t, test_field_width=test_field_width)
+
+ assert res.status in ('pass', 'fail', 'not run')
+
+ if res.casenotrun:
+ casenotrun.append(t)
+
+ if res.status != 'not run':
+ n_run += 1
+
+ if res.status == 'fail':
+ failed.append(name)
+ if self.makecheck:
+ self.env.print_env()
+ if res.diff:
+ print('\n'.join(res.diff))
+ elif res.status == 'not run':
+ notrun.append(name)
+
+ if notrun:
+ print('Not run:', ' '.join(notrun))
+
+ if casenotrun:
+ print('Some cases not run in:', ' '.join(notrun))
+
+ if failed:
+ print('Failures:', ' '.join(failed))
+ print(f'Failed {len(failed)} of {n_run} iotests')
+ else:
+ print(f'Passed all {n_run} iotests')
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 09/10] iotests: rewrite check into python
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2020-04-21 7:35 ` [PATCH v3 08/10] iotests: add testrunner.py Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:36 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:40 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:36 ` [PATCH v3 10/10] iotests: rename 169 and 199 Vladimir Sementsov-Ogievskiy
9 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:36 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Just use classes introduced in previous three commits. Behavior
difference is described in these three commits.
Drop group file, as it becomes unused.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/check | 965 ++-------------------------------------
tests/qemu-iotests/group | 299 ------------
2 files changed, 28 insertions(+), 1236 deletions(-)
delete mode 100644 tests/qemu-iotests/group
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 03016e1e91..4cb6fd5113 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -1,7 +1,8 @@
-#!/usr/bin/env bash
+#!/usr/bin/env python3
#
-# Copyright (C) 2009 Red Hat, Inc.
-# Copyright (c) 2000-2002,2006 Silicon Graphics, Inc. All Rights Reserved.
+# Configure environment and run group of tests in it.
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
@@ -14,938 +15,28 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-#
-#
-# Control script for QA
-#
-
-status=0
-needwrap=true
-try=0
-n_bad=0
-bad=""
-notrun=""
-casenotrun=""
-interrupt=true
-makecheck=false
-
-_init_error()
-{
- echo "check: $1" >&2
- exit 1
-}
-
-if [ -L "$0" ]
-then
- # called from the build tree
- source_iotests=$(dirname "$(readlink "$0")")
- if [ -z "$source_iotests" ]
- then
- _init_error "failed to obtain source tree name from check symlink"
- fi
- source_iotests=$(cd "$source_iotests"; pwd) || _init_error "failed to enter source tree"
- build_iotests=$PWD
-else
- # called from the source tree
- source_iotests=$PWD
- # this may be an in-tree build (note that in the following code we may not
- # assume that it truly is and have to test whether the build results
- # actually exist)
- build_iotests=$PWD
-fi
-
-build_root="$build_iotests/../.."
-
-# we need common.env
-if ! . "$build_iotests/common.env"
-then
- _init_error "failed to source common.env (make sure the qemu-iotests are run from tests/qemu-iotests in the build tree)"
-fi
-
-# we need common.config
-if ! . "$source_iotests/common.config"
-then
- _init_error "failed to source common.config"
-fi
-
-_full_imgfmt_details()
-{
- if [ -n "$IMGOPTS" ]; then
- echo "$IMGFMT ($IMGOPTS)"
- else
- echo "$IMGFMT"
- fi
-}
-
-_full_platform_details()
-{
- os=$(uname -s)
- host=$(hostname -s)
- kernel=$(uname -r)
- platform=$(uname -m)
- echo "$os/$platform $host $kernel"
-}
-
-_full_env_details()
-{
- cat <<EOF
-QEMU -- "$QEMU_PROG" $QEMU_OPTIONS
-QEMU_IMG -- "$QEMU_IMG_PROG" $QEMU_IMG_OPTIONS
-QEMU_IO -- "$QEMU_IO_PROG" $QEMU_IO_OPTIONS
-QEMU_NBD -- "$QEMU_NBD_PROG" $QEMU_NBD_OPTIONS
-IMGFMT -- $FULL_IMGFMT_DETAILS
-IMGPROTO -- $IMGPROTO
-PLATFORM -- $FULL_HOST_DETAILS
-TEST_DIR -- $TEST_DIR
-SOCK_DIR -- $SOCK_DIR
-SOCKET_SCM_HELPER -- $SOCKET_SCM_HELPER
-
-EOF
-}
-
-if [ -z "$TEST_DIR" ]; then
- TEST_DIR=$PWD/scratch
-fi
-mkdir -p "$TEST_DIR" || _init_error 'Failed to create TEST_DIR'
-
-tmp_sock_dir=false
-if [ -z "$SOCK_DIR" ]; then
- SOCK_DIR=$(mktemp -d)
- tmp_sock_dir=true
-fi
-mkdir -p "$SOCK_DIR" || _init_error 'Failed to create SOCK_DIR'
-
-diff="diff -u"
-verbose=false
-debug=false
-group=false
-xgroup=false
-imgopts=false
-showme=false
-sortme=false
-expunge=true
-have_test_arg=false
-cachemode=false
-aiomode=false
-
-tmp="${TEST_DIR}"/$$
-rm -f $tmp.list $tmp.tmp $tmp.sed
-
-export IMGFMT=raw
-export IMGFMT_GENERIC=true
-export IMGPROTO=file
-export IMGOPTS=""
-export CACHEMODE="writeback"
-export AIOMODE="threads"
-export QEMU_IO_OPTIONS=""
-export QEMU_IO_OPTIONS_NO_FMT=""
-export CACHEMODE_IS_DEFAULT=true
-export VALGRIND_QEMU=
-export IMGKEYSECRET=
-export IMGOPTSSYNTAX=false
-
-# Save current tty settings, since an aborting qemu call may leave things
-# screwed up
-STTY_RESTORE=
-if test -t 0; then
- STTY_RESTORE=$(stty -g)
-fi
-
-for r
-do
-
- if $group
- then
- # arg after -g
- group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
-s/ .*//p
-}')
- if [ -z "$group_list" ]
- then
- echo "Group \"$r\" is empty or not defined?"
- exit 1
- fi
- [ ! -s $tmp.list ] && touch $tmp.list
- for t in $group_list
- do
- if grep -s "^$t\$" $tmp.list >/dev/null
- then
- :
- else
- echo "$t" >>$tmp.list
- fi
- done
- group=false
- continue
-
- elif $xgroup
- then
- # arg after -x
- # Populate $tmp.list with all tests
- awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list 2>/dev/null
- group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e "/^[0-9][0-9][0-9].* $r /"'{
-s/ .*//p
-}')
- if [ -z "$group_list" ]
- then
- echo "Group \"$r\" is empty or not defined?"
- exit 1
- fi
- numsed=0
- rm -f $tmp.sed
- for t in $group_list
- do
- if [ $numsed -gt 100 ]
- then
- sed -f $tmp.sed <$tmp.list >$tmp.tmp
- mv $tmp.tmp $tmp.list
- numsed=0
- rm -f $tmp.sed
- fi
- echo "/^$t\$/d" >>$tmp.sed
- numsed=$(expr $numsed + 1)
- done
- sed -f $tmp.sed <$tmp.list >$tmp.tmp
- mv $tmp.tmp $tmp.list
- xgroup=false
- continue
-
- elif $imgopts
- then
- IMGOPTS="$r"
- imgopts=false
- continue
- elif $cachemode
- then
- CACHEMODE="$r"
- CACHEMODE_IS_DEFAULT=false
- cachemode=false
- continue
- elif $aiomode
- then
- AIOMODE="$r"
- aiomode=false
- continue
- fi
-
- xpand=true
- case "$r"
- in
-
- -\? | -h | --help) # usage
- echo "Usage: $0 [options] [testlist]"'
-
-common options
- -v verbose
- -d debug
-
-image format options
- -raw test raw (default)
- -bochs test bochs
- -cloop test cloop
- -parallels test parallels
- -qcow test qcow
- -qcow2 test qcow2
- -qed test qed
- -vdi test vdi
- -vpc test vpc
- -vhdx test vhdx
- -vmdk test vmdk
- -luks test luks
- -dmg test dmg
-
-image protocol options
- -file test file (default)
- -rbd test rbd
- -sheepdog test sheepdog
- -nbd test nbd
- -ssh test ssh
- -nfs test nfs
- -vxhs test vxhs
-
-other options
- -xdiff graphical mode diff
- -nocache use O_DIRECT on backing file
- -misalign misalign memory allocations
- -n show me, do not run tests
- -o options -o options to pass to qemu-img create/convert
- -c mode cache mode
- -i mode AIO mode
- -makecheck pretty print output for make check
-
-testlist options
- -g group[,group...] include tests from these groups
- -x group[,group...] exclude tests from these groups
- NNN include test NNN
- NNN-NNN include test range (eg. 012-021)
-'
- exit 0
- ;;
-
- -raw)
- IMGFMT=raw
- xpand=false
- ;;
-
- -bochs)
- IMGFMT=bochs
- IMGFMT_GENERIC=false
- xpand=false
- ;;
-
- -cloop)
- IMGFMT=cloop
- IMGFMT_GENERIC=false
- xpand=false
- ;;
-
- -parallels)
- IMGFMT=parallels
- xpand=false
- ;;
-
- -qcow)
- IMGFMT=qcow
- xpand=false
- ;;
-
- -qcow2)
- IMGFMT=qcow2
- xpand=false
- ;;
-
- -luks)
- IMGOPTSSYNTAX=true
- IMGFMT=luks
- IMGKEYSECRET=123456
- xpand=false
- ;;
-
- -dmg)
- IMGFMT=dmg
- IMGFMT_GENERIC=false
- xpand=false
- ;;
-
- -qed)
- IMGFMT=qed
- xpand=false
- ;;
-
- -vdi)
- IMGFMT=vdi
- xpand=false
- ;;
-
- -vmdk)
- IMGFMT=vmdk
- xpand=false
- ;;
-
- -vpc)
- IMGFMT=vpc
- xpand=false
- ;;
-
- -vhdx)
- IMGFMT=vhdx
- xpand=false
- ;;
-
- -file)
- IMGPROTO=file
- xpand=false
- ;;
-
- -rbd)
- IMGPROTO=rbd
- xpand=false
- ;;
-
- -sheepdog)
- IMGPROTO=sheepdog
- xpand=false
- ;;
-
- -nbd)
- IMGPROTO=nbd
- xpand=false
- ;;
-
- -vxhs)
- IMGPROTO=vxhs
- xpand=false
- ;;
-
- -ssh)
- IMGPROTO=ssh
- xpand=false
- ;;
-
- -nfs)
- IMGPROTO=nfs
- xpand=false
- ;;
-
- -nocache)
- CACHEMODE="none"
- CACHEMODE_IS_DEFAULT=false
- xpand=false
- ;;
-
- -misalign)
- QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS --misalign"
- xpand=false
- ;;
-
- -valgrind)
- VALGRIND_QEMU='y'
- xpand=false
- ;;
-
- -g) # -g group ... pick from group file
- group=true
- xpand=false
- ;;
-
- -xdiff) # graphical diff mode
- xpand=false
-
- if [ ! -z "$DISPLAY" ]
- then
- command -v xdiff >/dev/null 2>&1 && diff=xdiff
- command -v gdiff >/dev/null 2>&1 && diff=gdiff
- command -v tkdiff >/dev/null 2>&1 && diff=tkdiff
- command -v xxdiff >/dev/null 2>&1 && diff=xxdiff
- fi
- ;;
- -makecheck) # makecheck friendly output
- makecheck=true
- xpand=false
- ;;
- -n) # show me, don't do it
- showme=true
- xpand=false
- ;;
- -o)
- imgopts=true
- xpand=false
- ;;
- -c)
- cachemode=true
- xpand=false
- ;;
- -i)
- aiomode=true
- xpand=false
- ;;
- -T) # deprecated timestamp option
- xpand=false
- ;;
- -v)
- verbose=true
- xpand=false
- ;;
- -d)
- debug=true
- xpand=false
- ;;
- -x) # -x group ... exclude from group file
- xgroup=true
- xpand=false
- ;;
- '[0-9][0-9][0-9] [0-9][0-9][0-9][0-9]')
- echo "No tests?"
- status=1
- exit $status
- ;;
-
- [0-9]*-[0-9]*)
- eval $(echo $r | sed -e 's/^/start=/' -e 's/-/ end=/')
- ;;
-
- [0-9]*-)
- eval $(echo $r | sed -e 's/^/start=/' -e 's/-//')
- end=$(echo [0-9][0-9][0-9] [0-9][0-9][0-9][0-9] | sed -e 's/\[0-9]//g' -e 's/ *$//' -e 's/.* //')
- if [ -z "$end" ]
- then
- echo "No tests in range \"$r\"?"
- status=1
- exit $status
- fi
- ;;
-
- *)
- start=$r
- end=$r
- ;;
-
- esac
-
- # get rid of leading 0s as can be interpreted as octal
- start=$(echo $start | sed 's/^0*//')
- end=$(echo $end | sed 's/^0*//')
-
- if $xpand
- then
- have_test_arg=true
- awk </dev/null '
-BEGIN { for (t='$start'; t<='$end'; t++) printf "%03d\n",t }' \
- | while read id
- do
- if grep -s "^$id\( \|\$\)" "$source_iotests/group" >/dev/null
- then
- # in group file ... OK
- echo $id >>$tmp.list
- else
- if [ -f expunged ] && $expunge && egrep "^$id([ ]|\$)" expunged >/dev/null
- then
- # expunged ... will be reported, but not run, later
- echo $id >>$tmp.list
- else
- # oops
- if [ "$start" == "$end" -a "$id" == "$end" ]
- then
- echo "$id - unknown test"
- exit 1
- else
- echo "$id - unknown test, ignored"
- fi
- fi
- fi
- done || exit 1
- fi
-
-done
-
-# Set qemu-io cache mode with $CACHEMODE we have
-QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS --cache $CACHEMODE"
-# Set qemu-io aio mode with $AIOMODE we have
-QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS --aio $AIOMODE"
-
-QEMU_IO_OPTIONS_NO_FMT="$QEMU_IO_OPTIONS"
-if [ "$IMGOPTSSYNTAX" != "true" ]; then
- QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS -f $IMGFMT"
-fi
-
-# Set default options for qemu-img create -o if they were not specified
-if [ "$IMGFMT" == "qcow2" ] && ! (echo "$IMGOPTS" | grep "compat=" > /dev/null); then
- IMGOPTS=$(_optstr_add "$IMGOPTS" "compat=1.1")
-fi
-if [ "$IMGFMT" == "luks" ] && ! (echo "$IMGOPTS" | grep "iter-time=" > /dev/null); then
- IMGOPTS=$(_optstr_add "$IMGOPTS" "iter-time=10")
-fi
-
-if [ -z "$SAMPLE_IMG_DIR" ]; then
- SAMPLE_IMG_DIR="$source_iotests/sample_images"
-fi
-
-export TEST_DIR
-export SOCK_DIR
-export SAMPLE_IMG_DIR
-
-if [ -s $tmp.list ]
-then
- # found some valid test numbers ... this is good
- :
-else
- if $have_test_arg
- then
- # had test numbers, but none in group file ... do nothing
- touch $tmp.list
- else
- # no test numbers, do everything from group file
- sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p' <"$source_iotests/group" >$tmp.list
- fi
-fi
-
-# should be sort -n, but this did not work for Linux when this
-# was ported from IRIX
-#
-list=$(sort $tmp.list)
-rm -f $tmp.list $tmp.tmp $tmp.sed
-
-if [ -z "$QEMU_PROG" ]
-then
- if [ -x "$build_iotests/qemu" ]; then
- export QEMU_PROG="$build_iotests/qemu"
- elif [ -x "$build_root/${qemu_arch}-softmmu/qemu-system-${qemu_arch}" ]; then
- export QEMU_PROG="$build_root/${qemu_arch}-softmmu/qemu-system-${qemu_arch}"
- else
- pushd "$build_root" > /dev/null
- for binary in *-softmmu/qemu-system-*
- do
- if [ -x "$binary" ]
- then
- export QEMU_PROG="$build_root/$binary"
- break
- fi
- done
- popd > /dev/null
- [ "$QEMU_PROG" = "" ] && _init_error "qemu not found"
- fi
-fi
-export QEMU_PROG="$(type -p "$QEMU_PROG")"
-
-case "$QEMU_PROG" in
- *qemu-system-arm|*qemu-system-aarch64)
- export QEMU_OPTIONS="-nodefaults -display none -machine virt -accel qtest"
- ;;
- *qemu-system-tricore)
- export QEMU_OPTIONS="-nodefaults -display none -machine tricore_testboard -accel qtest"
- ;;
- *)
- export QEMU_OPTIONS="-nodefaults -display none -accel qtest"
- ;;
-esac
-
-if [ -z "$QEMU_IMG_PROG" ]; then
- if [ -x "$build_iotests/qemu-img" ]; then
- export QEMU_IMG_PROG="$build_iotests/qemu-img"
- elif [ -x "$build_root/qemu-img" ]; then
- export QEMU_IMG_PROG="$build_root/qemu-img"
- else
- _init_error "qemu-img not found"
- fi
-fi
-export QEMU_IMG_PROG="$(type -p "$QEMU_IMG_PROG")"
-
-if [ -z "$QEMU_IO_PROG" ]; then
- if [ -x "$build_iotests/qemu-io" ]; then
- export QEMU_IO_PROG="$build_iotests/qemu-io"
- elif [ -x "$build_root/qemu-io" ]; then
- export QEMU_IO_PROG="$build_root/qemu-io"
- else
- _init_error "qemu-io not found"
- fi
-fi
-export QEMU_IO_PROG="$(type -p "$QEMU_IO_PROG")"
-
-if [ -z $QEMU_NBD_PROG ]; then
- if [ -x "$build_iotests/qemu-nbd" ]; then
- export QEMU_NBD_PROG="$build_iotests/qemu-nbd"
- elif [ -x "$build_root/qemu-nbd" ]; then
- export QEMU_NBD_PROG="$build_root/qemu-nbd"
- else
- _init_error "qemu-nbd not found"
- fi
-fi
-export QEMU_NBD_PROG="$(type -p "$QEMU_NBD_PROG")"
-
-if [ -x "$build_iotests/socket_scm_helper" ]
-then
- export SOCKET_SCM_HELPER="$build_iotests/socket_scm_helper"
-fi
-
-python_usable=false
-if $PYTHON -c 'import sys; sys.exit(0 if sys.version_info >= (3,6) else 1)'
-then
- # Our python framework also requires virtio-blk
- if "$QEMU_PROG" -M none -device help | grep -q virtio-blk >/dev/null 2>&1
- then
- python_usable=true
- else
- python_unusable_because="Missing virtio-blk in QEMU binary"
- fi
-else
- python_unusable_because="Unsupported Python version"
-fi
-
-default_machine=$($QEMU_PROG -machine help | sed -n '/(default)/ s/ .*//p')
-default_alias_machine=$($QEMU_PROG -machine help | \
- sed -n "/(alias of $default_machine)/ { s/ .*//p; q; }")
-if [[ "$default_alias_machine" ]]; then
- default_machine="$default_alias_machine"
-fi
-
-export QEMU_DEFAULT_MACHINE="$default_machine"
-
-TIMESTAMP_FILE=check.time-$IMGPROTO-$IMGFMT
-
-_wallclock()
-{
- date "+%H %M %S" | awk '{ print $1*3600 + $2*60 + $3 }'
-}
-
-_wrapup()
-{
- if $showme
- then
- :
- elif $needwrap
- then
- if [ -f $TIMESTAMP_FILE -a -f $tmp.time ]
- then
- cat $TIMESTAMP_FILE $tmp.time \
- | awk '
- { t[$1] = $2 }
-END { if (NR > 0) {
- for (i in t) print i " " t[i]
- }
- }' \
- | sort -n >$tmp.out
- mv $tmp.out $TIMESTAMP_FILE
- fi
-
- if [ -f $tmp.expunged ]
- then
- notrun=$(wc -l <$tmp.expunged | sed -e 's/ *//g')
- try=$(expr $try - $notrun)
- list=$(echo "$list" | sed -f $tmp.expunged)
- fi
-
- echo "" >>check.log
- date >>check.log
- echo $list | fmt | sed -e 's/^/ /' >>check.log
- $interrupt && echo "Interrupted!" >>check.log
-
- if [ ! -z "$notrun" ]
- then
- echo "Not run:$notrun"
- echo "Not run:$notrun" >>check.log
- fi
- if [ ! -z "$casenotrun" ]
- then
- echo "Some cases not run in:$casenotrun"
- echo "Some cases not run in:$casenotrun" >>check.log
- fi
- if [ ! -z "$n_bad" -a $n_bad != 0 ]
- then
- echo "Failures:$bad"
- echo "Failed $n_bad of $try iotests"
- echo "Failures:$bad" | fmt >>check.log
- echo "Failed $n_bad of $try iotests" >>check.log
- else
- echo "Passed all $try iotests"
- echo "Passed all $try iotests" >>check.log
- fi
- needwrap=false
- fi
-
- if test -n "$STTY_RESTORE"; then
- stty $STTY_RESTORE
- fi
- rm -f "${TEST_DIR}"/*.out "${TEST_DIR}"/*.err "${TEST_DIR}"/*.time
- rm -f "${TEST_DIR}"/check.pid "${TEST_DIR}"/check.sts
- rm -f $tmp.*
-
- if $tmp_sock_dir
- then
- rm -rf "$SOCK_DIR"
- fi
-}
-
-trap "_wrapup; exit \$status" 0 1 2 3 15
-
-# Report the test start and results. For makecheck we want to pretty
-# print the whole report at the end of the execution.
-# args: $seq, $starttime, $lasttime
-_report_test_start()
-{
- if ! $makecheck; then
- if [ -n "$3" ]; then
- local lasttime=" (last: $3s)"
- fi
- printf "%-8s %-10s [%s] %4s%-14s\r" "$1" "..." "$2" "..." "$lasttime"
- fi
-}
-# args:$seq $status $starttime $lasttime $thistime $details
-_report_test_result()
-{
- local status lasttime thistime
- if $makecheck; then
- if [ -n "$2" ] && [ "$2" != "pass" ]; then
- status=" [$2]"
- fi
- printf " TEST iotest-$IMGFMT: %s%s\n" "$1" "$status"
- return
- fi
-
- if [ -n "$4" ]; then
- lasttime=" (last: $4s)"
- fi
- if [ -n "$5" ]; then
- thistime=" $5s"
- fi
- case "$2" in
- "pass") status=$(printf "\e[32m%-10s\e[0m" "$2") ;;
- "fail") status=$(printf "\e[1m\e[31m%-10s\e[0m" "$2") ;;
- "not run") status=$(printf "\e[33m%-10s\e[0m" "$2") ;;
- *) status=$(printf "%-10s" "$2") ;;
- esac
-
- printf "%-8s %s [%s] [%s] %4s%-14s %s\n" "$1" "$status" "$3" "$(date '+%T')" "$thistime" "$lasttime" "$6"
-}
-
-[ -f $TIMESTAMP_FILE ] || touch $TIMESTAMP_FILE
-
-FULL_IMGFMT_DETAILS=$(_full_imgfmt_details)
-FULL_HOST_DETAILS=$(_full_platform_details)
-
-if ! $makecheck; then
- _full_env_details
-fi
-
-seq="check"
-
-[ -n "$TESTS_REMAINING_LOG" ] && echo $list > $TESTS_REMAINING_LOG
-
-for seq in $list
-do
- err=false # error flag
- printdiff=false # show diff to reference output?
- status="" # test result summary
- results="" # test result details
- thistime="" # time the test took
-
- if [ -n "$TESTS_REMAINING_LOG" ] ; then
- sed -e "s/$seq//" -e 's/ / /' -e 's/^ *//' $TESTS_REMAINING_LOG > $TESTS_REMAINING_LOG.tmp
- mv $TESTS_REMAINING_LOG.tmp $TESTS_REMAINING_LOG
- sync
- fi
-
- lasttime=$(sed -n -e "/^$seq /s/.* //p" <$TIMESTAMP_FILE)
- starttime=$(date "+%T")
- _report_test_start $seq $starttime $lasttime
-
- if $showme
- then
- status="not run"
- elif [ -f expunged ] && $expunge && egrep "^$seq([ ]|\$)" expunged >/dev/null
- then
- status="not run"
- results="expunged"
- rm -f $seq.out.bad
- echo "/^$seq\$/d" >>$tmp.expunged
- elif [ ! -f "$source_iotests/$seq" ]
- then
- status="not run"
- results="no such test?"
- echo "/^$seq\$/d" >>$tmp.expunged
- else
- # really going to try and run this one
- #
- rm -f $seq.out.bad
- rm -f core $seq.notrun
- rm -f $seq.casenotrun
-
- start=$(_wallclock)
-
- if [ "$(head -n 1 "$source_iotests/$seq")" == "#!/usr/bin/env python3" ]; then
- if $python_usable; then
- run_command="$PYTHON $seq"
- else
- run_command="false"
- echo "$python_unusable_because" > $seq.notrun
- fi
- else
- run_command="./$seq"
- fi
- export OUTPUT_DIR=$PWD
- if $debug; then
- (cd "$source_iotests";
- MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(($RANDOM % 255 + 1))} \
- $run_command -d 2>&1 | tee $tmp.out)
- else
- (cd "$source_iotests";
- MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(($RANDOM % 255 + 1))} \
- $run_command >$tmp.out 2>&1)
- fi
- sts=$?
- stop=$(_wallclock)
-
- if [ -f core ]
- then
- mv core $seq.core
- status="fail"
- results="[dumped core] $seq.core"
- err=true
- fi
-
- if [ -f $seq.notrun ]
- then
- # overwrites timestamp output
- status="not run"
- results="$(cat $seq.notrun)"
- else
- if [ $sts -ne 0 ]
- then
- status="fail"
- results=$(printf %s "[failed, exit status $sts]")
- err=true
- fi
-
- reference="$source_iotests/$seq.out"
- reference_machine="$source_iotests/$seq.$QEMU_DEFAULT_MACHINE.out"
- if [ -f "$reference_machine" ]; then
- reference="$reference_machine"
- fi
-
- reference_format="$source_iotests/$seq.out.$IMGFMT"
- if [ -f "$reference_format" ]; then
- reference="$reference_format"
- fi
-
- if [ "$CACHEMODE" = "none" ]; then
- [ -f "$source_iotests/$seq.out.nocache" ] && reference="$source_iotests/$seq.out.nocache"
- fi
-
- if [ ! -f "$reference" ]
- then
- status="fail"
- results="no qualified output"
- err=true
- else
- if diff -w "$reference" $tmp.out >/dev/null 2>&1
- then
- if ! $err; then
- status="pass"
- thistime=$(expr $stop - $start)
- echo "$seq $thistime" >>$tmp.time
- fi
- else
- mv $tmp.out $seq.out.bad
- status="fail"
- results="output mismatch (see $seq.out.bad)"
- printdiff=true
- err=true
- fi
- fi
- fi
- if [ -f $seq.casenotrun ]
- then
- cat $seq.casenotrun
- casenotrun="$casenotrun $seq"
- fi
- fi
-
- # come here for each test, except when $showme is true
- #
- _report_test_result $seq "$status" "$starttime" "$lasttime" "$thistime" "$results"
- case "$status" in
- "pass")
- try=$(expr $try + 1)
- ;;
- "fail")
- try=$(expr $try + 1)
- if $makecheck; then
- _full_env_details
- fi
- if $printdiff; then
- $diff -w "$reference" "$PWD"/$seq.out.bad
- fi
- bad="$bad $seq"
- n_bad=$(expr $n_bad + 1)
- quick=false
- ;;
- "not run")
- notrun="$notrun $seq"
- ;;
- esac
-
- seq="after_$seq"
-done
-interrupt=false
-status=$(expr $n_bad)
-exit
+import sys
+import os
+from testfinder import find_tests, TestFinder
+from testenv import TestEnv
+from testrunner import TestRunner
+
+if __name__ == '__main__':
+ if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
+ print('Usage: ./check [options] [testlist]')
+ print()
+ TestFinder.argparser.print_help()
+ print()
+ TestEnv.argparser.print_help()
+ print()
+ TestRunner.argparser.print_help()
+ exit()
+
+ env = TestEnv(sys.argv[1:])
+ tests, remaining_argv = find_tests(env.remaining_argv,
+ test_dir=env.source_iotests)
+
+ with TestRunner(remaining_argv, env) as tr:
+ assert not tr.remaining_argv
+ tr.run_tests([os.path.join(env.source_iotests, t) for t in tests])
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
deleted file mode 100644
index 435dccd5af..0000000000
--- a/tests/qemu-iotests/group
+++ /dev/null
@@ -1,299 +0,0 @@
-#
-# QA groups control file
-# Defines test groups
-#
-# Some notes about the groups:
-#
-# - do not start group names with a digit
-#
-# - quick : Tests in this group should finish within some few seconds.
-#
-# - img : Tests in this group can be used to excercise the qemu-img tool.
-#
-# - auto : Tests in this group are used during "make check" and should be
-# runnable in any case. That means they should run with every QEMU binary
-# (also non-x86), with every QEMU configuration (i.e. must not fail if
-# an optional feature is not compiled in - but reporting a "skip" is ok),
-# work at least with the qcow2 file format, work with all kind of host
-# filesystems and users (e.g. "nobody" or "root") and must not take too
-# much memory and disk space (since CI pipelines tend to fail otherwise).
-#
-
-#
-# test-group association ... one line per test
-#
-001 rw auto quick
-002 rw auto quick
-003 rw auto
-004 rw auto quick
-005 img auto quick
-# 006 was removed, do not reuse
-007 snapshot auto
-008 rw auto quick
-009 rw auto quick
-010 rw auto quick
-011 rw auto quick
-012 auto quick
-013 rw auto
-014 rw
-015 rw snapshot
-# 016 was removed, do not reuse
-017 rw backing auto quick
-018 rw backing auto quick
-019 rw backing auto quick
-020 rw backing auto quick
-021 io auto quick
-022 rw snapshot auto
-023 rw
-024 rw backing auto quick
-025 rw auto quick
-026 rw blkdbg
-027 rw auto quick
-028 rw backing quick
-029 rw auto quick
-030 rw auto backing
-031 rw auto quick
-032 rw auto quick
-033 rw auto quick
-034 rw auto backing quick
-035 rw auto quick
-036 rw auto quick
-037 rw auto backing quick
-038 rw auto backing quick
-039 rw auto quick
-040 rw auto
-041 rw auto backing
-042 rw auto quick
-043 rw auto backing
-044 rw
-045 rw quick
-046 rw auto aio quick
-047 rw auto quick
-048 img auto quick
-049 rw auto
-050 rw auto backing quick
-051 rw
-052 rw auto backing quick
-053 rw auto quick
-054 rw auto quick
-055 rw
-056 rw backing
-057 rw
-058 rw quick
-059 rw quick
-060 rw auto quick
-061 rw auto
-062 rw auto quick
-063 rw auto quick
-064 rw quick
-065 rw quick
-066 rw auto quick
-067 rw quick
-068 rw quick
-069 rw auto quick
-070 rw quick
-071 rw auto quick
-072 rw auto quick
-073 rw auto quick
-074 rw auto quick
-075 rw quick
-076 io
-077 rw quick
-078 rw quick
-079 rw auto
-080 rw auto
-081 rw quick
-082 rw quick
-083 rw
-084 img quick
-085 rw
-086 rw auto quick
-087 rw quick
-088 rw quick
-089 rw auto quick
-090 rw auto quick
-091 rw migration
-092 rw quick
-093 throttle
-094 rw quick
-095 rw quick
-096 rw quick
-097 rw auto backing
-098 rw auto backing quick
-099 rw auto quick
-# 100 was removed, do not reuse
-101 rw quick
-102 rw quick
-103 rw auto quick
-104 rw auto
-105 rw auto quick
-106 rw quick
-107 rw auto quick
-108 rw auto quick
-109 rw
-110 rw auto backing quick
-111 rw auto quick
-112 rw
-113 rw quick
-114 rw auto quick
-115 rw
-116 rw quick
-117 rw auto
-118 rw
-119 rw quick
-120 rw auto quick
-121 rw
-122 rw
-123 rw quick
-124 rw backing
-125 rw
-126 rw auto backing
-127 rw auto backing quick
-128 rw quick
-129 rw quick
-130 rw quick
-131 rw quick
-132 rw quick
-133 auto quick
-134 rw auto quick
-135 rw
-136 rw
-137 rw auto
-138 rw auto quick
-139 rw quick
-140 rw auto quick
-141 rw auto quick
-142
-143 auto quick
-144 rw quick
-145 quick
-146 quick
-147 img
-148 rw quick
-149 rw sudo
-150 rw auto quick
-151 rw
-152 rw quick
-153 rw quick
-154 rw auto backing quick
-155 rw
-156 rw auto quick
-157 quick
-158 rw auto quick
-159 rw auto quick
-160 rw quick
-161 rw auto quick
-162 quick
-163 rw
-165 rw quick
-169 rw quick migration
-170 rw auto quick
-171 rw quick
-172 auto
-173 rw
-174 auto
-175 quick
-176 rw auto backing
-177 rw auto quick
-178 img
-179 rw auto quick
-181 rw auto migration
-182 rw quick
-183 rw migration
-184 rw auto quick
-185 rw
-186 rw auto
-187 rw auto
-188 rw quick
-189 rw
-190 rw auto quick
-191 rw auto
-192 rw auto quick
-194 rw migration quick
-195 rw auto quick
-196 rw quick migration
-197 rw quick
-198 rw
-199 rw migration
-200 rw
-201 rw migration
-202 rw quick
-203 rw auto migration
-204 rw quick
-205 rw quick
-206 rw
-207 rw
-208 rw quick
-209 rw quick
-210 rw
-211 rw quick
-212 rw quick
-213 rw quick
-214 rw auto
-215 rw quick
-216 rw quick
-217 rw auto quick
-218 rw quick
-219 rw
-220 rw auto
-221 rw quick
-222 rw quick
-223 rw quick
-224 rw quick
-225 rw quick
-226 auto quick
-227 quick
-228 rw quick
-229 auto quick
-231 quick
-232 quick
-233 quick
-234 quick migration
-235 quick
-236 quick
-237 rw quick
-238 quick
-239 rw quick
-240 quick
-241 rw quick
-242 rw quick
-243 rw quick
-244 rw auto quick
-245 rw
-246 rw quick
-247 rw quick
-248 rw quick
-249 rw auto quick
-250 rw quick
-251 rw auto quick
-252 rw auto backing quick
-253 rw quick
-254 rw backing quick
-255 rw quick
-256 rw auto quick
-257 rw
-258 rw quick
-259 rw auto quick
-260 rw quick
-261 rw
-262 rw quick migration
-263 rw quick
-264 rw
-265 rw auto quick
-266 rw quick
-267 rw auto quick snapshot
-268 rw auto quick
-270 rw backing quick
-272 rw
-273 backing quick
-277 rw quick
-279 rw backing quick
-280 rw migration quick
-281 rw quick
-282 rw img quick
-283 auto quick
-284 rw
-286 rw quick
-288 quick
-289 rw quick
-290 rw auto quick
--
2.21.0
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v3 10/10] iotests: rename 169 and 199
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
` (8 preceding siblings ...)
2020-04-21 7:36 ` [PATCH v3 09/10] iotests: rewrite check into python Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:36 ` Vladimir Sementsov-Ogievskiy
9 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:36 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, vsementsov, jsnow, qemu-devel, mreitz, den
Rename bitmaps migration tests to demonstrate new test naming
convention.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/{199 => migrate-bitmaps-postcopy-test} | 0
tests/qemu-iotests/{199.out => migrate-bitmaps-postcopy-test.out} | 0
tests/qemu-iotests/{169 => migrate-bitmaps-test} | 0
tests/qemu-iotests/{169.out => migrate-bitmaps-test.out} | 0
4 files changed, 0 insertions(+), 0 deletions(-)
rename tests/qemu-iotests/{199 => migrate-bitmaps-postcopy-test} (100%)
rename tests/qemu-iotests/{199.out => migrate-bitmaps-postcopy-test.out} (100%)
rename tests/qemu-iotests/{169 => migrate-bitmaps-test} (100%)
rename tests/qemu-iotests/{169.out => migrate-bitmaps-test.out} (100%)
diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/migrate-bitmaps-postcopy-test
similarity index 100%
rename from tests/qemu-iotests/199
rename to tests/qemu-iotests/migrate-bitmaps-postcopy-test
diff --git a/tests/qemu-iotests/199.out b/tests/qemu-iotests/migrate-bitmaps-postcopy-test.out
similarity index 100%
rename from tests/qemu-iotests/199.out
rename to tests/qemu-iotests/migrate-bitmaps-postcopy-test.out
diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/migrate-bitmaps-test
similarity index 100%
rename from tests/qemu-iotests/169
rename to tests/qemu-iotests/migrate-bitmaps-test
diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/migrate-bitmaps-test.out
similarity index 100%
rename from tests/qemu-iotests/169.out
rename to tests/qemu-iotests/migrate-bitmaps-test.out
--
2.21.0
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 09/10] iotests: rewrite check into python
2020-04-21 7:36 ` [PATCH v3 09/10] iotests: rewrite check into python Vladimir Sementsov-Ogievskiy
@ 2020-04-21 7:40 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 7:40 UTC (permalink / raw)
To: qemu-block; +Cc: kwolf, qemu-devel, mreitz, den, jsnow
21.04.2020 10:36, Vladimir Sementsov-Ogievskiy wrote:
> Just use classes introduced in previous three commits. Behavior
> difference is described in these three commits.
>
> Drop group file, as it becomes unused.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy<vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/check | 965 ++-------------------------------------
> tests/qemu-iotests/group | 299 ------------
> 2 files changed, 28 insertions(+), 1236 deletions(-)
> delete mode 100644 tests/qemu-iotests/group
>
> diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
> index 03016e1e91..4cb6fd5113 100755
> --- a/tests/qemu-iotests/check
> +++ b/tests/qemu-iotests/check
> @@ -1,7 +1,8 @@
> -#!/usr/bin/env bash
> +#!/usr/bin/env python3
> #
> -# Copyright (C) 2009 Red Hat, Inc.
> -# Copyright (c) 2000-2002,2006 Silicon Graphics, Inc. All Rights Reserved.
I've dropped old copyrights. If someone think that I should keep them here
or in a new testenv.py/testfinder.py/testrunner.py, please let me know.
> +# Configure environment and run group of tests in it.
> +#
> +# Copyright (c) 2020 Virtuozzo International GmbH
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running
2020-04-21 7:35 ` [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running Vladimir Sementsov-Ogievskiy
@ 2020-04-21 12:54 ` Eric Blake
2020-04-21 13:04 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2020-04-21 12:54 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 4/21/20 2:35 AM, Vladimir Sementsov-Ogievskiy wrote:
> Somehow, this doesn't work if you just call ./277. But check 277 works.
> We are going to new iotests running framework, which will consider test
> files as self-executable, so let's just change it now.
This is a rather vague explanation. I'd prefer something a bit more
forthcoming, maybe:
If you run './check 277', check includes common.config which adjusts
$PATH to include '.' first, and therefore finds nbd-fault-injector.py on
PATH. But if you run './277' directly, there is nothing to adjust PATH,
and if '.' is not already on your PATH by other means, the test fails
because the executable is not found. Adjust how we invoke the helper
executable to avoid needing a PATH search in the first place.
At any rate, while the commit message could be improved, the fix itself
is reasonable.
Reviewed-by: Eric Blake <eblake@redhat.com>
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/277 | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277
> index 04aa15a3d5..a4ef1a38c6 100755
> --- a/tests/qemu-iotests/277
> +++ b/tests/qemu-iotests/277
> @@ -40,7 +40,7 @@ def make_conf_file(event):
> def start_server_NBD(event):
> make_conf_file(event)
>
> - srv = subprocess.Popen(['nbd-fault-injector.py', '--classic-negotiation',
> + srv = subprocess.Popen(['./nbd-fault-injector.py', '--classic-negotiation',
> nbd_sock, conf_file], stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT, universal_newlines=True)
> line = srv.stdout.readline()
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 03/10] iotests/283: make executable
2020-04-21 7:35 ` [PATCH v3 03/10] iotests/283: make executable Vladimir Sementsov-Ogievskiy
@ 2020-04-21 12:55 ` Eric Blake
2020-05-14 6:17 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Eric Blake @ 2020-04-21 12:55 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 4/21/20 2:35 AM, Vladimir Sementsov-Ogievskiy wrote:
> All other test files are executable, except for this one. Fix that.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/283 | 0
> 1 file changed, 0 insertions(+), 0 deletions(-)
> mode change 100644 => 100755 tests/qemu-iotests/283
Reviewed-by: Eric Blake <eblake@redhat.com>
>
> diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283
> old mode 100644
> new mode 100755
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running
2020-04-21 12:54 ` Eric Blake
@ 2020-04-21 13:04 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 13:04 UTC (permalink / raw)
To: Eric Blake, qemu-block; +Cc: kwolf, den, jsnow, qemu-devel, mreitz
21.04.2020 15:54, Eric Blake wrote:
> On 4/21/20 2:35 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Somehow, this doesn't work if you just call ./277. But check 277 works.
>> We are going to new iotests running framework, which will consider test
>> files as self-executable, so let's just change it now.
>
> This is a rather vague explanation.
Yes, because I didn't understand why does it work with check:) But you make it clean.
> I'd prefer something a bit more forthcoming, maybe:
>
> If you run './check 277', check includes common.config which adjusts $PATH to include '.' first, and therefore finds nbd-fault-injector.py on PATH. But if you run './277' directly, there is nothing to adjust PATH, and if '.' is not already on your PATH by other means, the test fails because the executable is not found. Adjust how we invoke the helper executable to avoid needing a PATH search in the first place.
That's better of course.
>
>
> At any rate, while the commit message could be improved, the fix itself is reasonable.
>
> Reviewed-by: Eric Blake <eblake@redhat.com>
Thanks!
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc
2020-04-21 7:35 ` [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc Vladimir Sementsov-Ogievskiy
@ 2020-04-21 16:03 ` Kevin Wolf
2020-04-22 5:14 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2020-04-21 16:03 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> QEMU_VXHS_PROG is used only in common.rc. So, move it to common.rc,
> simplifying a bit further conversion of check into python
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
This feels inconsistent when every other QEMU_*_PROG stays in check. Is
QEMU_VXHS_PROG really so different?
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-21 7:35 ` [PATCH v3 06/10] iotests: add testfinder.py Vladimir Sementsov-Ogievskiy
@ 2020-04-21 16:56 ` Kevin Wolf
2020-04-22 5:35 ` Vladimir Sementsov-Ogievskiy
2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
0 siblings, 2 replies; 30+ messages in thread
From: Kevin Wolf @ 2020-04-21 16:56 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> Add python script with new logic of searching for tests:
>
> Current ./check behavior:
> - tests are named [0-9][0-9][0-9]
> - tests must be registered in group file (even if test doesn't belong
> to any group, like 142)
>
> Behavior of new test:
> - group file is dropped
> - tests are searched by file-name instead of group file, so it's not
> needed more to "register the test", just create it with name
> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
> recommended for new tests
I wonder if a tests/ subdirectory instead of the -test suffix would
organise things a bit better.
> - groups are parsed from '# group: ' line inside test files
> - optional file group.local may be used to define some additional
> groups for downstreams
> - 'disabled' group is used to temporary disable tests. So instead of
> commenting tests in old 'group' file you now can add them to
> disabled group with help of 'group.local' file
> - selecting test ranges like 5-15 are not supported more
Occasionally they were useful when something went wrong during the test
run and I only wanted to repeat the part after it happened. But it's a
rare case and we don't have a clear order any more with arbitrary test
names (which are an improvement otherwise), so we'll live with it.
> Benefits:
> - no rebase conflicts in group file on patch porting from branch to
> branch
> - no conflicts in upstream, when different series want to occupy same
> test number
> - meaningful names for test files
> For example, with digital number, when some person wants to add some
> test about block-stream, he most probably will just create a new
> test. But if there would be test-block-stream test already, he will
> at first look at it and may be just add a test-case into it.
> And anyway meaningful names are better.
>
> This commit just adds class, which is unused now, and will be used in
> further patches, to finally substitute ./check selecting tests logic.
Maybe mention here that the file can be executed standalone, even if
it'S not used by check yet.
> Still, the documentation changed like new behavior is already here.
> Let's live with this small inconsistency for the following few commits,
> until final change.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> docs/devel/testing.rst | 52 +++++++++-
> tests/qemu-iotests/testfinder.py | 167 +++++++++++++++++++++++++++++++
A little bit of bikeshedding: As this can be executed as a standalone
tool, would a name like findtests.py be better?
> 2 files changed, 218 insertions(+), 1 deletion(-)
> create mode 100755 tests/qemu-iotests/testfinder.py
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 770a987ea4..6c9d5b126b 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -153,7 +153,7 @@ check-block
> -----------
>
> ``make check-block`` runs a subset of the block layer iotests (the tests that
> -are in the "auto" group in ``tests/qemu-iotests/group``).
> +are in the "auto" group).
> See the "QEMU iotests" section below for more information.
>
> GCC gcov support
> @@ -267,6 +267,56 @@ another application on the host may have locked the file, possibly leading to a
> test failure. If using such devices are explicitly desired, consider adding
> ``locking=off`` option to disable image locking.
>
> +Test case groups
> +----------------
> +
> +Test may belong to some groups, you may define it in the comment inside the
> +test. By convention, test groups are listed in the second line of the test
> +file, after "#!/..." line, like this:
> +
> +.. code::
> +
> + #!/usr/bin/env python3
> + # group: auto quick
> + #
> + ...
> +
> +Additional way of defining groups is creating tests/qemu-iotests/group.local
> +file. This should be used only for downstream (this file should never appear
> +in upstream). This file may be used for defining some downstream test groups
> +or for temporary disable tests, like this:
> +
> +.. code::
> +
> + # groups for some company downstream process
> + #
> + # ci - tests to run on build
> + # down - our downstream tests, not for upstream
> + #
> + # Format of each line is:
> + # TEST_NAME TEST_GROUP [TEST_GROUP ]...
> +
> + 013 ci
> + 210 disabled
> + 215 disabled
> + our-ugly-workaround-test down ci
> +
> +The following groups are defined:
> +
> +- quick : Tests in this group should finish within some few seconds.
> +
> +- img : Tests in this group can be used to excercise the qemu-img tool.
> +
> +- auto : Tests in this group are used during "make check" and should be
> + runnable in any case. That means they should run with every QEMU binary
> + (also non-x86), with every QEMU configuration (i.e. must not fail if
> + an optional feature is not compiled in - but reporting a "skip" is ok),
> + work at least with the qcow2 file format, work with all kind of host
> + filesystems and users (e.g. "nobody" or "root") and must not take too
> + much memory and disk space (since CI pipelines tend to fail otherwise).
> +
> +- disabled : Tests in this group are disabled and ignored by check.
We have more groups than just these. We could either try and document
all of them here, or we could only keep auto/quick/disabled which have a
general meaning rather than defining an area of code that they test. If
we do the latter, I would clarify that this is not an exhaustive list,
and remove "img" from this section.
> .. _docker-ref:
>
> Docker based tests
> diff --git a/tests/qemu-iotests/testfinder.py b/tests/qemu-iotests/testfinder.py
> new file mode 100755
> index 0000000000..1da28564c0
> --- /dev/null
> +++ b/tests/qemu-iotests/testfinder.py
As this is a new code file, would you mind adding type hints from the
start?
> @@ -0,0 +1,167 @@
> +#!/usr/bin/env python3
> +#
> +# Parse command line options to define set of tests to run.
> +#
> +# Copyright (c) 2020 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import os
> +import sys
> +import glob
> +import argparse
> +import re
> +from collections import defaultdict
> +from contextlib import contextmanager
> +
> +
> +@contextmanager
> +def chdir(path):
> + if path is None:
> + yield
> + return
> +
> + saved_dir = os.getcwd()
> + os.chdir(path)
> + try:
> + yield
> + finally:
> + os.chdir(saved_dir)
> +
> +
> +class TestFinder:
> + @staticmethod
> + def create_argparser():
> + p = argparse.ArgumentParser(description="Select set of tests",
"a set of tests"?
> + add_help=False, usage=argparse.SUPPRESS)
> +
> + p.add_argument('-g', metavar='group1,...', dest='groups',
> + help='include tests from these groups')
> + p.add_argument('-x', metavar='group1,...', dest='exclude_groups',
> + help='exclude tests from these groups')
> + p.add_argument('tests', metavar='TEST_FILES', nargs='*',
> + help='tests to run')
> +
> + return p
> +
> + argparser = create_argparser.__func__()
> +
> + def __init__(self, test_dir=None):
> + self.groups = defaultdict(set)
> +
> + with chdir(test_dir):
> + self.all_tests = glob.glob('[0-9][0-9][0-9]')
> + self.all_tests += [f for f in glob.iglob('*-test')]
> +
> + for t in self.all_tests:
> + with open(t) as f:
> + for line in f:
> + if line.startswith('# group: '):
> + for g in line.split()[2:]:
> + self.groups[g].add(t)
> +
> + def add_group_file(self, fname):
> + with open(fname) as f:
> + for line in f:
> + line = line.strip()
> +
> + if (not line) or line[0] == '#':
> + continue
> +
> + words = line.split()
> + test_file = words[0]
> + groups = words[1:]
> +
> + if test_file not in self.all_tests:
> + print('Warning: {}: "{}" test is not found. '
> + 'Skip.'.format(fname, test_file))
You're using f strings in the later patches. Wouldn't it be more
consistent to use them here, too?
> + continue
> +
> + for g in groups:
> + self.groups[g].add(test_file)
> +
> + def find_tests(self, groups=None, exclude_groups=None, tests=None):
> + """
> + 1. Take all tests from @groups,
> + or just all tests if @groups is None and @tests is None
> + or nothing if @groups is None and @tests is not None
> + 2. Drop tests, which are in at least one of @exclude_groups or in
> + 'disabled' group (if 'disabled' is not listed in @groups)
> + 3. Add tests from @tests
> + """
> + if groups is None:
> + groups = []
> + if exclude_groups is None:
> + exclude_groups = []
> + if tests is None:
> + tests = []
> +
> + if groups:
> + res = set().union(*(self.groups[g] for g in groups))
> + elif tests:
> + res = set()
> + else:
> + res = set(self.all_tests)
> +
> + if 'disabled' not in groups and 'disabled' not in exclude_groups:
> + exclude_groups.append('disabled')
> +
> + res = res.difference(*(self.groups[g] for g in exclude_groups))
> +
> + # We want to add @tests. But for compatibility with old test names,
> + # we should convert any number < 100 to number padded by
> + # leading zeroes, like 1 -> 001 and 23 -> 023.
> + for t in tests:
> + if re.fullmatch(r'\d{1,2}', t):
> + res.add('0' * (3 - len(t)) + t)
> + else:
> + res.add(t)
> +
> + return sorted(res)
> +
> + def find_tests_argv(self, argv):
> + args, remaining = self.argparser.parse_known_args(argv)
> +
> + if args.groups is not None:
> + args.groups = args.groups.split(',')
> +
> + if args.exclude_groups is not None:
> + args.exclude_groups = args.exclude_groups.split(',')
> +
> + return self.find_tests(**vars(args)), remaining
> +
> +
> +def find_tests(argv, test_dir=None):
> + tf = TestFinder(test_dir)
> +
> + if test_dir is None:
> + group_local = 'group.local'
> + else:
> + group_local = os.path.join(test_dir, 'group.local')
> + if os.path.isfile(group_local):
> + tf.add_group_file(group_local)
> +
> + return tf.find_tests_argv(argv)
> +
> +
> +if __name__ == '__main__':
> + if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
> + TestFinder.argparser.print_help()
> + exit()
> +
> + tests, remaining_argv = find_tests(sys.argv[1:])
> + print('\n'.join(tests))
> + if remaining_argv:
> + print('\nUnhandled options: ', remaining_argv)
I think unhandled options shouldn't be considered an error and we
shouldn't even try to find and display any tests then. I'd either print
the help text or have an error message that refers to --help.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc
2020-04-21 16:03 ` Kevin Wolf
@ 2020-04-22 5:14 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-22 5:14 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
21.04.2020 19:03, Kevin Wolf wrote:
> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> QEMU_VXHS_PROG is used only in common.rc. So, move it to common.rc,
>> simplifying a bit further conversion of check into python
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>
> This feels inconsistent when every other QEMU_*_PROG stays in check. Is
> QEMU_VXHS_PROG really so different?
>
Hmm, I was just too lazy to understand set_prog_path logic :) If you think it worth it, I'll try.
+set_prog_path()
+{
+ p=$(command -v $1 2> /dev/null)
+ if [ -n "$p" -a -x "$p" ]; then
+ type -p "$p"
+ else
+ return 1
+ fi
+}
Aha. It just tries to get path to the command and check that it is executable.
So, in python, it probably should look like simply
p = shutil.which(command)
return p if os.access(p, os.X_OK) else None
OK, I'll add it in next version.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-21 16:56 ` Kevin Wolf
@ 2020-04-22 5:35 ` Vladimir Sementsov-Ogievskiy
2020-04-22 11:53 ` Kevin Wolf
2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-22 5:35 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
21.04.2020 19:56, Kevin Wolf wrote:
> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> Add python script with new logic of searching for tests:
>>
>> Current ./check behavior:
>> - tests are named [0-9][0-9][0-9]
>> - tests must be registered in group file (even if test doesn't belong
>> to any group, like 142)
>>
>> Behavior of new test:
>> - group file is dropped
>> - tests are searched by file-name instead of group file, so it's not
>> needed more to "register the test", just create it with name
>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>> recommended for new tests
>
> I wonder if a tests/ subdirectory instead of the -test suffix would
> organise things a bit better.
No objections.
I also thought about, may be, a tests/ subtree, so we'll have something like
tests/jobs/<mirror, stream, backup tests>
tests/formats/<qcow2 tests and others, or may be one more subdirectory level>
...
>
>> - groups are parsed from '# group: ' line inside test files
>> - optional file group.local may be used to define some additional
>> groups for downstreams
>> - 'disabled' group is used to temporary disable tests. So instead of
>> commenting tests in old 'group' file you now can add them to
>> disabled group with help of 'group.local' file
>> - selecting test ranges like 5-15 are not supported more
>
> Occasionally they were useful when something went wrong during the test
> run and I only wanted to repeat the part after it happened. But it's a
> rare case and we don't have a clear order any more with arbitrary test
> names (which are an improvement otherwise), so we'll live with it.
Yes, I've used it for same thing.
Actually, we still have the order, as I just sort iotests by name. I think,
we could add a parameter for testfinder (may be, as a separate step no in
these series), something like
--start-from TEST : parse all other arguments as usual, make sorted sequence
and than drop tests from the first one to TEST (not inclusive). This may be
used to rerun failed ./check command, starting from the middle of the process.
>
>> Benefits:
>> - no rebase conflicts in group file on patch porting from branch to
>> branch
>> - no conflicts in upstream, when different series want to occupy same
>> test number
>> - meaningful names for test files
>> For example, with digital number, when some person wants to add some
>> test about block-stream, he most probably will just create a new
>> test. But if there would be test-block-stream test already, he will
>> at first look at it and may be just add a test-case into it.
>> And anyway meaningful names are better.
>>
>> This commit just adds class, which is unused now, and will be used in
>> further patches, to finally substitute ./check selecting tests logic.
>
> Maybe mention here that the file can be executed standalone, even if
> it'S not used by check yet.
>
>> Still, the documentation changed like new behavior is already here.
>> Let's live with this small inconsistency for the following few commits,
>> until final change.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> docs/devel/testing.rst | 52 +++++++++-
>> tests/qemu-iotests/testfinder.py | 167 +++++++++++++++++++++++++++++++
>
> A little bit of bikeshedding: As this can be executed as a standalone
> tool, would a name like findtests.py be better?
Hmm. I named it by class name, considering possibility to execute is just for module testing. So for module users, it's just a module with class TestFinder, and it's called testfinder.. But I don't have strict opinion in it, findtests sound more human-friendly.
>
>> 2 files changed, 218 insertions(+), 1 deletion(-)
>> create mode 100755 tests/qemu-iotests/testfinder.py
>>
>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>> index 770a987ea4..6c9d5b126b 100644
>> --- a/docs/devel/testing.rst
>> +++ b/docs/devel/testing.rst
>> @@ -153,7 +153,7 @@ check-block
>> -----------
>>
>> ``make check-block`` runs a subset of the block layer iotests (the tests that
>> -are in the "auto" group in ``tests/qemu-iotests/group``).
>> +are in the "auto" group).
>> See the "QEMU iotests" section below for more information.
>>
>> GCC gcov support
>> @@ -267,6 +267,56 @@ another application on the host may have locked the file, possibly leading to a
>> test failure. If using such devices are explicitly desired, consider adding
>> ``locking=off`` option to disable image locking.
>>
>> +Test case groups
>> +----------------
>> +
>> +Test may belong to some groups, you may define it in the comment inside the
>> +test. By convention, test groups are listed in the second line of the test
>> +file, after "#!/..." line, like this:
>> +
>> +.. code::
>> +
>> + #!/usr/bin/env python3
>> + # group: auto quick
>> + #
>> + ...
>> +
>> +Additional way of defining groups is creating tests/qemu-iotests/group.local
>> +file. This should be used only for downstream (this file should never appear
>> +in upstream). This file may be used for defining some downstream test groups
>> +or for temporary disable tests, like this:
>> +
>> +.. code::
>> +
>> + # groups for some company downstream process
>> + #
>> + # ci - tests to run on build
>> + # down - our downstream tests, not for upstream
>> + #
>> + # Format of each line is:
>> + # TEST_NAME TEST_GROUP [TEST_GROUP ]...
>> +
>> + 013 ci
>> + 210 disabled
>> + 215 disabled
>> + our-ugly-workaround-test down ci
>> +
>> +The following groups are defined:
>> +
>> +- quick : Tests in this group should finish within some few seconds.
>> +
>> +- img : Tests in this group can be used to excercise the qemu-img tool.
>> +
>> +- auto : Tests in this group are used during "make check" and should be
>> + runnable in any case. That means they should run with every QEMU binary
>> + (also non-x86), with every QEMU configuration (i.e. must not fail if
>> + an optional feature is not compiled in - but reporting a "skip" is ok),
>> + work at least with the qcow2 file format, work with all kind of host
>> + filesystems and users (e.g. "nobody" or "root") and must not take too
>> + much memory and disk space (since CI pipelines tend to fail otherwise).
>> +
>> +- disabled : Tests in this group are disabled and ignored by check.
>
> We have more groups than just these. We could either try and document
> all of them here, or we could only keep auto/quick/disabled which have a
> general meaning rather than defining an area of code that they test. If
> we do the latter, I would clarify that this is not an exhaustive list,
> and remove "img" from this section.
OK. I'll drop img for now, documenting all the groups may be done in separate.
>
>> .. _docker-ref:
>>
>> Docker based tests
>> diff --git a/tests/qemu-iotests/testfinder.py b/tests/qemu-iotests/testfinder.py
>> new file mode 100755
>> index 0000000000..1da28564c0
>> --- /dev/null
>> +++ b/tests/qemu-iotests/testfinder.py
>
> As this is a new code file, would you mind adding type hints from the
> start?
I added it in one-two non-obvious places. Is there any general thought about using type-hints? Should modern coder use them absolutely everywhere?
>
>> @@ -0,0 +1,167 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Parse command line options to define set of tests to run.
>> +#
>> +# Copyright (c) 2020 Virtuozzo International GmbH
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>> +#
>> +
>> +import os
>> +import sys
>> +import glob
>> +import argparse
>> +import re
>> +from collections import defaultdict
>> +from contextlib import contextmanager
>> +
>> +
>> +@contextmanager
>> +def chdir(path):
>> + if path is None:
>> + yield
>> + return
>> +
>> + saved_dir = os.getcwd()
>> + os.chdir(path)
>> + try:
>> + yield
>> + finally:
>> + os.chdir(saved_dir)
>> +
>> +
>> +class TestFinder:
>> + @staticmethod
>> + def create_argparser():
>> + p = argparse.ArgumentParser(description="Select set of tests",
>
> "a set of tests"?
>
>> + add_help=False, usage=argparse.SUPPRESS)
>> +
>> + p.add_argument('-g', metavar='group1,...', dest='groups',
>> + help='include tests from these groups')
>> + p.add_argument('-x', metavar='group1,...', dest='exclude_groups',
>> + help='exclude tests from these groups')
>> + p.add_argument('tests', metavar='TEST_FILES', nargs='*',
>> + help='tests to run')
>> +
>> + return p
>> +
>> + argparser = create_argparser.__func__()
>> +
>> + def __init__(self, test_dir=None):
>> + self.groups = defaultdict(set)
>> +
>> + with chdir(test_dir):
>> + self.all_tests = glob.glob('[0-9][0-9][0-9]')
>> + self.all_tests += [f for f in glob.iglob('*-test')]
>> +
>> + for t in self.all_tests:
>> + with open(t) as f:
>> + for line in f:
>> + if line.startswith('# group: '):
>> + for g in line.split()[2:]:
>> + self.groups[g].add(t)
>> +
>> + def add_group_file(self, fname):
>> + with open(fname) as f:
>> + for line in f:
>> + line = line.strip()
>> +
>> + if (not line) or line[0] == '#':
>> + continue
>> +
>> + words = line.split()
>> + test_file = words[0]
>> + groups = words[1:]
>> +
>> + if test_file not in self.all_tests:
>> + print('Warning: {}: "{}" test is not found. '
>> + 'Skip.'.format(fname, test_file))
>
> You're using f strings in the later patches. Wouldn't it be more
> consistent to use them here, too?
Yes, thanks. I just wrote this code before learning f-strings)
>
>> + continue
>> +
>> + for g in groups:
>> + self.groups[g].add(test_file)
>> +
>> + def find_tests(self, groups=None, exclude_groups=None, tests=None):
>> + """
>> + 1. Take all tests from @groups,
>> + or just all tests if @groups is None and @tests is None
>> + or nothing if @groups is None and @tests is not None
>> + 2. Drop tests, which are in at least one of @exclude_groups or in
>> + 'disabled' group (if 'disabled' is not listed in @groups)
>> + 3. Add tests from @tests
>> + """
>> + if groups is None:
>> + groups = []
>> + if exclude_groups is None:
>> + exclude_groups = []
>> + if tests is None:
>> + tests = []
>> +
>> + if groups:
>> + res = set().union(*(self.groups[g] for g in groups))
>> + elif tests:
>> + res = set()
>> + else:
>> + res = set(self.all_tests)
>> +
>> + if 'disabled' not in groups and 'disabled' not in exclude_groups:
>> + exclude_groups.append('disabled')
>> +
>> + res = res.difference(*(self.groups[g] for g in exclude_groups))
>> +
>> + # We want to add @tests. But for compatibility with old test names,
>> + # we should convert any number < 100 to number padded by
>> + # leading zeroes, like 1 -> 001 and 23 -> 023.
>> + for t in tests:
>> + if re.fullmatch(r'\d{1,2}', t):
>> + res.add('0' * (3 - len(t)) + t)
>> + else:
>> + res.add(t)
>> +
>> + return sorted(res)
>> +
>> + def find_tests_argv(self, argv):
>> + args, remaining = self.argparser.parse_known_args(argv)
>> +
>> + if args.groups is not None:
>> + args.groups = args.groups.split(',')
>> +
>> + if args.exclude_groups is not None:
>> + args.exclude_groups = args.exclude_groups.split(',')
>> +
>> + return self.find_tests(**vars(args)), remaining
>> +
>> +
>> +def find_tests(argv, test_dir=None):
>> + tf = TestFinder(test_dir)
>> +
>> + if test_dir is None:
>> + group_local = 'group.local'
>> + else:
>> + group_local = os.path.join(test_dir, 'group.local')
>> + if os.path.isfile(group_local):
>> + tf.add_group_file(group_local)
>> +
>> + return tf.find_tests_argv(argv)
>> +
>> +
>> +if __name__ == '__main__':
>> + if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
>> + TestFinder.argparser.print_help()
>> + exit()
>> +
>> + tests, remaining_argv = find_tests(sys.argv[1:])
>> + print('\n'.join(tests))
>> + if remaining_argv:
>> + print('\nUnhandled options: ', remaining_argv)
>
> I think unhandled options shouldn't be considered an error and we
> shouldn't even try to find and display any tests then. I'd either print
> the help text or have an error message that refers to --help.
>
As I considered running this script as executable for testing purposes, it's
good to know, which options are not handled..
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-22 5:35 ` Vladimir Sementsov-Ogievskiy
@ 2020-04-22 11:53 ` Kevin Wolf
2020-04-22 12:49 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2020-04-22 11:53 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
Am 22.04.2020 um 07:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 21.04.2020 19:56, Kevin Wolf wrote:
> > Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > Add python script with new logic of searching for tests:
> > >
> > > Current ./check behavior:
> > > - tests are named [0-9][0-9][0-9]
> > > - tests must be registered in group file (even if test doesn't belong
> > > to any group, like 142)
> > >
> > > Behavior of new test:
> > > - group file is dropped
> > > - tests are searched by file-name instead of group file, so it's not
> > > needed more to "register the test", just create it with name
> > > *-test. Old names like [0-9][0-9][0-9] are supported too, but not
> > > recommended for new tests
> >
> > I wonder if a tests/ subdirectory instead of the -test suffix would
> > organise things a bit better.
>
> No objections.
>
> I also thought about, may be, a tests/ subtree, so we'll have something like
>
> tests/jobs/<mirror, stream, backup tests>
> tests/formats/<qcow2 tests and others, or may be one more subdirectory level>
> ...
I thought of that, too, but then decided not to mention it because I
feel it might be hard to decide where a test belongs. Many tests are
qcow2 and mirror tests at the same time, every commit test is also a
backing file test etc.
This is essentially why we have groups and each test can be in more
than one group.
On the other hand, I assume that for some tests it would be quite clear
where they belong. So if we're prepared to have some tests directly in
tests/ and only some others in subdirectories, we can still do that.
> > > - groups are parsed from '# group: ' line inside test files
> > > - optional file group.local may be used to define some additional
> > > groups for downstreams
> > > - 'disabled' group is used to temporary disable tests. So instead of
> > > commenting tests in old 'group' file you now can add them to
> > > disabled group with help of 'group.local' file
> > > - selecting test ranges like 5-15 are not supported more
> >
> > Occasionally they were useful when something went wrong during the test
> > run and I only wanted to repeat the part after it happened. But it's a
> > rare case and we don't have a clear order any more with arbitrary test
> > names (which are an improvement otherwise), so we'll live with it.
>
> Yes, I've used it for same thing.
>
> Actually, we still have the order, as I just sort iotests by name. I think,
> we could add a parameter for testfinder (may be, as a separate step no in
> these series), something like
>
> --start-from TEST : parse all other arguments as usual, make sorted sequence
> and than drop tests from the first one to TEST (not inclusive). This may be
> used to rerun failed ./check command, starting from the middle of the process.
True, this would be a good replacement. I don't think it's a requirement
to have it in this series, but I won't complain if you implement it. :-)
> >
> > > Benefits:
> > > - no rebase conflicts in group file on patch porting from branch to
> > > branch
> > > - no conflicts in upstream, when different series want to occupy same
> > > test number
> > > - meaningful names for test files
> > > For example, with digital number, when some person wants to add some
> > > test about block-stream, he most probably will just create a new
> > > test. But if there would be test-block-stream test already, he will
> > > at first look at it and may be just add a test-case into it.
> > > And anyway meaningful names are better.
> > >
> > > This commit just adds class, which is unused now, and will be used in
> > > further patches, to finally substitute ./check selecting tests logic.
> >
> > Maybe mention here that the file can be executed standalone, even if
> > it'S not used by check yet.
> >
> > > Still, the documentation changed like new behavior is already here.
> > > Let's live with this small inconsistency for the following few commits,
> > > until final change.
> > >
> > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > > ---
> > > docs/devel/testing.rst | 52 +++++++++-
> > > tests/qemu-iotests/testfinder.py | 167 +++++++++++++++++++++++++++++++
> >
> > A little bit of bikeshedding: As this can be executed as a standalone
> > tool, would a name like findtests.py be better?
>
> Hmm. I named it by class name, considering possibility to execute is
> just for module testing. So for module users, it's just a module with
> class TestFinder, and it's called testfinder.. But I don't have strict
> opinion in it, findtests sound more human-friendly.
It was just a thought. If you prefer testfinder.py, I'm fine with it.
> >
> > > 2 files changed, 218 insertions(+), 1 deletion(-)
> > > create mode 100755 tests/qemu-iotests/testfinder.py
> > >
> > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > > index 770a987ea4..6c9d5b126b 100644
> > > --- a/docs/devel/testing.rst
> > > +++ b/docs/devel/testing.rst
> > > @@ -153,7 +153,7 @@ check-block
> > > -----------
> > > ``make check-block`` runs a subset of the block layer iotests (the tests that
> > > -are in the "auto" group in ``tests/qemu-iotests/group``).
> > > +are in the "auto" group).
> > > See the "QEMU iotests" section below for more information.
> > > GCC gcov support
> > > @@ -267,6 +267,56 @@ another application on the host may have locked the file, possibly leading to a
> > > test failure. If using such devices are explicitly desired, consider adding
> > > ``locking=off`` option to disable image locking.
> > > +Test case groups
> > > +----------------
> > > +
> > > +Test may belong to some groups, you may define it in the comment inside the
> > > +test. By convention, test groups are listed in the second line of the test
> > > +file, after "#!/..." line, like this:
> > > +
> > > +.. code::
> > > +
> > > + #!/usr/bin/env python3
> > > + # group: auto quick
> > > + #
> > > + ...
> > > +
> > > +Additional way of defining groups is creating tests/qemu-iotests/group.local
> > > +file. This should be used only for downstream (this file should never appear
> > > +in upstream). This file may be used for defining some downstream test groups
> > > +or for temporary disable tests, like this:
> > > +
> > > +.. code::
> > > +
> > > + # groups for some company downstream process
> > > + #
> > > + # ci - tests to run on build
> > > + # down - our downstream tests, not for upstream
> > > + #
> > > + # Format of each line is:
> > > + # TEST_NAME TEST_GROUP [TEST_GROUP ]...
> > > +
> > > + 013 ci
> > > + 210 disabled
> > > + 215 disabled
> > > + our-ugly-workaround-test down ci
> > > +
> > > +The following groups are defined:
> > > +
> > > +- quick : Tests in this group should finish within some few seconds.
> > > +
> > > +- img : Tests in this group can be used to excercise the qemu-img tool.
> > > +
> > > +- auto : Tests in this group are used during "make check" and should be
> > > + runnable in any case. That means they should run with every QEMU binary
> > > + (also non-x86), with every QEMU configuration (i.e. must not fail if
> > > + an optional feature is not compiled in - but reporting a "skip" is ok),
> > > + work at least with the qcow2 file format, work with all kind of host
> > > + filesystems and users (e.g. "nobody" or "root") and must not take too
> > > + much memory and disk space (since CI pipelines tend to fail otherwise).
> > > +
> > > +- disabled : Tests in this group are disabled and ignored by check.
> >
> > We have more groups than just these. We could either try and document
> > all of them here, or we could only keep auto/quick/disabled which have a
> > general meaning rather than defining an area of code that they test. If
> > we do the latter, I would clarify that this is not an exhaustive list,
> > and remove "img" from this section.
>
> OK. I'll drop img for now, documenting all the groups may be done in separate.
>
> >
> > > .. _docker-ref:
> > > Docker based tests
> > > diff --git a/tests/qemu-iotests/testfinder.py b/tests/qemu-iotests/testfinder.py
> > > new file mode 100755
> > > index 0000000000..1da28564c0
> > > --- /dev/null
> > > +++ b/tests/qemu-iotests/testfinder.py
> >
> > As this is a new code file, would you mind adding type hints from the
> > start?
>
> I added it in one-two non-obvious places. Is there any general thought
> about using type-hints? Should modern coder use them absolutely
> everywhere?
Type checkers only work if things are consistently annotated. For
example, if a function definition doesn't have type hints, the whole
code in this function stays completely unchecked even if it contains an
obvious error like passing a string literal to another function that is
annotated to take an int.
So I think we should add type hints everywhere.
> >
> > > @@ -0,0 +1,167 @@
> > > +#!/usr/bin/env python3
> > > +#
> > > +# Parse command line options to define set of tests to run.
> > > +#
> > > +# Copyright (c) 2020 Virtuozzo International GmbH
> > > +#
> > > +# This program is free software; you can redistribute it and/or modify
> > > +# it under the terms of the GNU General Public License as published by
> > > +# the Free Software Foundation; either version 2 of the License, or
> > > +# (at your option) any later version.
> > > +#
> > > +# This program is distributed in the hope that it will be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > > +# GNU General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU General Public License
> > > +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> > > +#
> > > +
> > > +import os
> > > +import sys
> > > +import glob
> > > +import argparse
> > > +import re
> > > +from collections import defaultdict
> > > +from contextlib import contextmanager
> > > +
> > > +
> > > +@contextmanager
> > > +def chdir(path):
> > > + if path is None:
> > > + yield
> > > + return
> > > +
> > > + saved_dir = os.getcwd()
> > > + os.chdir(path)
> > > + try:
> > > + yield
> > > + finally:
> > > + os.chdir(saved_dir)
> > > +
> > > +
> > > +class TestFinder:
> > > + @staticmethod
> > > + def create_argparser():
> > > + p = argparse.ArgumentParser(description="Select set of tests",
> >
> > "a set of tests"?
> >
> > > + add_help=False, usage=argparse.SUPPRESS)
> > > +
> > > + p.add_argument('-g', metavar='group1,...', dest='groups',
> > > + help='include tests from these groups')
> > > + p.add_argument('-x', metavar='group1,...', dest='exclude_groups',
> > > + help='exclude tests from these groups')
> > > + p.add_argument('tests', metavar='TEST_FILES', nargs='*',
> > > + help='tests to run')
> > > +
> > > + return p
> > > +
> > > + argparser = create_argparser.__func__()
> > > +
> > > + def __init__(self, test_dir=None):
> > > + self.groups = defaultdict(set)
> > > +
> > > + with chdir(test_dir):
> > > + self.all_tests = glob.glob('[0-9][0-9][0-9]')
> > > + self.all_tests += [f for f in glob.iglob('*-test')]
> > > +
> > > + for t in self.all_tests:
> > > + with open(t) as f:
> > > + for line in f:
> > > + if line.startswith('# group: '):
> > > + for g in line.split()[2:]:
> > > + self.groups[g].add(t)
> > > +
> > > + def add_group_file(self, fname):
> > > + with open(fname) as f:
> > > + for line in f:
> > > + line = line.strip()
> > > +
> > > + if (not line) or line[0] == '#':
> > > + continue
> > > +
> > > + words = line.split()
> > > + test_file = words[0]
> > > + groups = words[1:]
> > > +
> > > + if test_file not in self.all_tests:
> > > + print('Warning: {}: "{}" test is not found. '
> > > + 'Skip.'.format(fname, test_file))
> >
> > You're using f strings in the later patches. Wouldn't it be more
> > consistent to use them here, too?
>
> Yes, thanks. I just wrote this code before learning f-strings)
>
> >
> > > + continue
> > > +
> > > + for g in groups:
> > > + self.groups[g].add(test_file)
> > > +
> > > + def find_tests(self, groups=None, exclude_groups=None, tests=None):
> > > + """
> > > + 1. Take all tests from @groups,
> > > + or just all tests if @groups is None and @tests is None
> > > + or nothing if @groups is None and @tests is not None
> > > + 2. Drop tests, which are in at least one of @exclude_groups or in
> > > + 'disabled' group (if 'disabled' is not listed in @groups)
> > > + 3. Add tests from @tests
> > > + """
> > > + if groups is None:
> > > + groups = []
> > > + if exclude_groups is None:
> > > + exclude_groups = []
> > > + if tests is None:
> > > + tests = []
> > > +
> > > + if groups:
> > > + res = set().union(*(self.groups[g] for g in groups))
> > > + elif tests:
> > > + res = set()
> > > + else:
> > > + res = set(self.all_tests)
> > > +
> > > + if 'disabled' not in groups and 'disabled' not in exclude_groups:
> > > + exclude_groups.append('disabled')
> > > +
> > > + res = res.difference(*(self.groups[g] for g in exclude_groups))
> > > +
> > > + # We want to add @tests. But for compatibility with old test names,
> > > + # we should convert any number < 100 to number padded by
> > > + # leading zeroes, like 1 -> 001 and 23 -> 023.
> > > + for t in tests:
> > > + if re.fullmatch(r'\d{1,2}', t):
> > > + res.add('0' * (3 - len(t)) + t)
> > > + else:
> > > + res.add(t)
> > > +
> > > + return sorted(res)
> > > +
> > > + def find_tests_argv(self, argv):
> > > + args, remaining = self.argparser.parse_known_args(argv)
> > > +
> > > + if args.groups is not None:
> > > + args.groups = args.groups.split(',')
> > > +
> > > + if args.exclude_groups is not None:
> > > + args.exclude_groups = args.exclude_groups.split(',')
> > > +
> > > + return self.find_tests(**vars(args)), remaining
> > > +
> > > +
> > > +def find_tests(argv, test_dir=None):
> > > + tf = TestFinder(test_dir)
> > > +
> > > + if test_dir is None:
> > > + group_local = 'group.local'
> > > + else:
> > > + group_local = os.path.join(test_dir, 'group.local')
> > > + if os.path.isfile(group_local):
> > > + tf.add_group_file(group_local)
> > > +
> > > + return tf.find_tests_argv(argv)
> > > +
> > > +
> > > +if __name__ == '__main__':
> > > + if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
> > > + TestFinder.argparser.print_help()
> > > + exit()
> > > +
> > > + tests, remaining_argv = find_tests(sys.argv[1:])
> > > + print('\n'.join(tests))
> > > + if remaining_argv:
> > > + print('\nUnhandled options: ', remaining_argv)
> >
> > I think unhandled options shouldn't be considered an error and we
> > shouldn't even try to find and display any tests then. I'd either print
> > the help text or have an error message that refers to --help.
>
> As I considered running this script as executable for testing purposes, it's
> good to know, which options are not handled..
Yes, that makes sense. I just think it should be an error and not just
an additional hint at the end.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-22 11:53 ` Kevin Wolf
@ 2020-04-22 12:49 ` Vladimir Sementsov-Ogievskiy
2020-04-22 13:06 ` Kevin Wolf
0 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-22 12:49 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
22.04.2020 14:53, Kevin Wolf wrote:
> Am 22.04.2020 um 07:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 21.04.2020 19:56, Kevin Wolf wrote:
>>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> Add python script with new logic of searching for tests:
>>>>
>>>> Current ./check behavior:
>>>> - tests are named [0-9][0-9][0-9]
>>>> - tests must be registered in group file (even if test doesn't belong
>>>> to any group, like 142)
>>>>
>>>> Behavior of new test:
>>>> - group file is dropped
>>>> - tests are searched by file-name instead of group file, so it's not
>>>> needed more to "register the test", just create it with name
>>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>>> recommended for new tests
>>>
>>> I wonder if a tests/ subdirectory instead of the -test suffix would
>>> organise things a bit better.
>>
>> No objections.
>>
>> I also thought about, may be, a tests/ subtree, so we'll have something like
>>
>> tests/jobs/<mirror, stream, backup tests>
>> tests/formats/<qcow2 tests and others, or may be one more subdirectory level>
>> ...
>
> I thought of that, too, but then decided not to mention it because I
> feel it might be hard to decide where a test belongs. Many tests are
> qcow2 and mirror tests at the same time, every commit test is also a
> backing file test etc.
>
> This is essentially why we have groups and each test can be in more
> than one group.
>
> On the other hand, I assume that for some tests it would be quite clear
> where they belong. So if we're prepared to have some tests directly in
> tests/ and only some others in subdirectories, we can still do that.
>
>>>> - groups are parsed from '# group: ' line inside test files
>>>> - optional file group.local may be used to define some additional
>>>> groups for downstreams
>>>> - 'disabled' group is used to temporary disable tests. So instead of
>>>> commenting tests in old 'group' file you now can add them to
>>>> disabled group with help of 'group.local' file
>>>> - selecting test ranges like 5-15 are not supported more
>>>
>>> Occasionally they were useful when something went wrong during the test
>>> run and I only wanted to repeat the part after it happened. But it's a
>>> rare case and we don't have a clear order any more with arbitrary test
>>> names (which are an improvement otherwise), so we'll live with it.
>>
>> Yes, I've used it for same thing.
>>
>> Actually, we still have the order, as I just sort iotests by name. I think,
>> we could add a parameter for testfinder (may be, as a separate step no in
>> these series), something like
>>
>> --start-from TEST : parse all other arguments as usual, make sorted sequence
>> and than drop tests from the first one to TEST (not inclusive). This may be
>> used to rerun failed ./check command, starting from the middle of the process.
>
> True, this would be a good replacement. I don't think it's a requirement
> to have it in this series, but I won't complain if you implement it. :-)
>
>>>
>>>> Benefits:
>>>> - no rebase conflicts in group file on patch porting from branch to
>>>> branch
>>>> - no conflicts in upstream, when different series want to occupy same
>>>> test number
>>>> - meaningful names for test files
>>>> For example, with digital number, when some person wants to add some
>>>> test about block-stream, he most probably will just create a new
>>>> test. But if there would be test-block-stream test already, he will
>>>> at first look at it and may be just add a test-case into it.
>>>> And anyway meaningful names are better.
>>>>
>>>> This commit just adds class, which is unused now, and will be used in
>>>> further patches, to finally substitute ./check selecting tests logic.
>>>
>>> Maybe mention here that the file can be executed standalone, even if
>>> it'S not used by check yet.
>>>
>>>> Still, the documentation changed like new behavior is already here.
>>>> Let's live with this small inconsistency for the following few commits,
>>>> until final change.
>>>>
>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>> ---
>>>> docs/devel/testing.rst | 52 +++++++++-
>>>> tests/qemu-iotests/testfinder.py | 167 +++++++++++++++++++++++++++++++
>>>
>>> A little bit of bikeshedding: As this can be executed as a standalone
>>> tool, would a name like findtests.py be better?
>>
>> Hmm. I named it by class name, considering possibility to execute is
>> just for module testing. So for module users, it's just a module with
>> class TestFinder, and it's called testfinder.. But I don't have strict
>> opinion in it, findtests sound more human-friendly.
>
> It was just a thought. If you prefer testfinder.py, I'm fine with it.
>
>>>
>>>> 2 files changed, 218 insertions(+), 1 deletion(-)
>>>> create mode 100755 tests/qemu-iotests/testfinder.py
>>>>
>>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>>>> index 770a987ea4..6c9d5b126b 100644
>>>> --- a/docs/devel/testing.rst
>>>> +++ b/docs/devel/testing.rst
>>>> @@ -153,7 +153,7 @@ check-block
>>>> -----------
>>>> ``make check-block`` runs a subset of the block layer iotests (the tests that
>>>> -are in the "auto" group in ``tests/qemu-iotests/group``).
>>>> +are in the "auto" group).
>>>> See the "QEMU iotests" section below for more information.
>>>> GCC gcov support
>>>> @@ -267,6 +267,56 @@ another application on the host may have locked the file, possibly leading to a
>>>> test failure. If using such devices are explicitly desired, consider adding
>>>> ``locking=off`` option to disable image locking.
>>>> +Test case groups
>>>> +----------------
>>>> +
>>>> +Test may belong to some groups, you may define it in the comment inside the
>>>> +test. By convention, test groups are listed in the second line of the test
>>>> +file, after "#!/..." line, like this:
>>>> +
>>>> +.. code::
>>>> +
>>>> + #!/usr/bin/env python3
>>>> + # group: auto quick
>>>> + #
>>>> + ...
>>>> +
>>>> +Additional way of defining groups is creating tests/qemu-iotests/group.local
>>>> +file. This should be used only for downstream (this file should never appear
>>>> +in upstream). This file may be used for defining some downstream test groups
>>>> +or for temporary disable tests, like this:
>>>> +
>>>> +.. code::
>>>> +
>>>> + # groups for some company downstream process
>>>> + #
>>>> + # ci - tests to run on build
>>>> + # down - our downstream tests, not for upstream
>>>> + #
>>>> + # Format of each line is:
>>>> + # TEST_NAME TEST_GROUP [TEST_GROUP ]...
>>>> +
>>>> + 013 ci
>>>> + 210 disabled
>>>> + 215 disabled
>>>> + our-ugly-workaround-test down ci
>>>> +
>>>> +The following groups are defined:
>>>> +
>>>> +- quick : Tests in this group should finish within some few seconds.
>>>> +
>>>> +- img : Tests in this group can be used to excercise the qemu-img tool.
>>>> +
>>>> +- auto : Tests in this group are used during "make check" and should be
>>>> + runnable in any case. That means they should run with every QEMU binary
>>>> + (also non-x86), with every QEMU configuration (i.e. must not fail if
>>>> + an optional feature is not compiled in - but reporting a "skip" is ok),
>>>> + work at least with the qcow2 file format, work with all kind of host
>>>> + filesystems and users (e.g. "nobody" or "root") and must not take too
>>>> + much memory and disk space (since CI pipelines tend to fail otherwise).
>>>> +
>>>> +- disabled : Tests in this group are disabled and ignored by check.
>>>
>>> We have more groups than just these. We could either try and document
>>> all of them here, or we could only keep auto/quick/disabled which have a
>>> general meaning rather than defining an area of code that they test. If
>>> we do the latter, I would clarify that this is not an exhaustive list,
>>> and remove "img" from this section.
>>
>> OK. I'll drop img for now, documenting all the groups may be done in separate.
>>
>>>
>>>> .. _docker-ref:
>>>> Docker based tests
>>>> diff --git a/tests/qemu-iotests/testfinder.py b/tests/qemu-iotests/testfinder.py
>>>> new file mode 100755
>>>> index 0000000000..1da28564c0
>>>> --- /dev/null
>>>> +++ b/tests/qemu-iotests/testfinder.py
>>>
>>> As this is a new code file, would you mind adding type hints from the
>>> start?
>>
>> I added it in one-two non-obvious places. Is there any general thought
>> about using type-hints? Should modern coder use them absolutely
>> everywhere?
>
> Type checkers only work if things are consistently annotated. For
> example, if a function definition doesn't have type hints, the whole
> code in this function stays completely unchecked even if it contains an
> obvious error like passing a string literal to another function that is
> annotated to take an int.
>
> So I think we should add type hints everywhere.
>
>>>
>>>> @@ -0,0 +1,167 @@
>>>> +#!/usr/bin/env python3
>>>> +#
>>>> +# Parse command line options to define set of tests to run.
>>>> +#
>>>> +# Copyright (c) 2020 Virtuozzo International GmbH
>>>> +#
>>>> +# This program is free software; you can redistribute it and/or modify
>>>> +# it under the terms of the GNU General Public License as published by
>>>> +# the Free Software Foundation; either version 2 of the License, or
>>>> +# (at your option) any later version.
>>>> +#
>>>> +# This program is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>>> +# GNU General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU General Public License
>>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>>>> +#
>>>> +
>>>> +import os
>>>> +import sys
>>>> +import glob
>>>> +import argparse
>>>> +import re
>>>> +from collections import defaultdict
>>>> +from contextlib import contextmanager
>>>> +
>>>> +
>>>> +@contextmanager
>>>> +def chdir(path):
>>>> + if path is None:
>>>> + yield
>>>> + return
>>>> +
>>>> + saved_dir = os.getcwd()
>>>> + os.chdir(path)
>>>> + try:
>>>> + yield
>>>> + finally:
>>>> + os.chdir(saved_dir)
>>>> +
>>>> +
>>>> +class TestFinder:
>>>> + @staticmethod
>>>> + def create_argparser():
>>>> + p = argparse.ArgumentParser(description="Select set of tests",
>>>
>>> "a set of tests"?
>>>
>>>> + add_help=False, usage=argparse.SUPPRESS)
>>>> +
>>>> + p.add_argument('-g', metavar='group1,...', dest='groups',
>>>> + help='include tests from these groups')
>>>> + p.add_argument('-x', metavar='group1,...', dest='exclude_groups',
>>>> + help='exclude tests from these groups')
>>>> + p.add_argument('tests', metavar='TEST_FILES', nargs='*',
>>>> + help='tests to run')
>>>> +
>>>> + return p
>>>> +
>>>> + argparser = create_argparser.__func__()
>>>> +
>>>> + def __init__(self, test_dir=None):
>>>> + self.groups = defaultdict(set)
>>>> +
>>>> + with chdir(test_dir):
>>>> + self.all_tests = glob.glob('[0-9][0-9][0-9]')
>>>> + self.all_tests += [f for f in glob.iglob('*-test')]
>>>> +
>>>> + for t in self.all_tests:
>>>> + with open(t) as f:
>>>> + for line in f:
>>>> + if line.startswith('# group: '):
>>>> + for g in line.split()[2:]:
>>>> + self.groups[g].add(t)
>>>> +
>>>> + def add_group_file(self, fname):
>>>> + with open(fname) as f:
>>>> + for line in f:
>>>> + line = line.strip()
>>>> +
>>>> + if (not line) or line[0] == '#':
>>>> + continue
>>>> +
>>>> + words = line.split()
>>>> + test_file = words[0]
>>>> + groups = words[1:]
>>>> +
>>>> + if test_file not in self.all_tests:
>>>> + print('Warning: {}: "{}" test is not found. '
>>>> + 'Skip.'.format(fname, test_file))
>>>
>>> You're using f strings in the later patches. Wouldn't it be more
>>> consistent to use them here, too?
>>
>> Yes, thanks. I just wrote this code before learning f-strings)
>>
>>>
>>>> + continue
>>>> +
>>>> + for g in groups:
>>>> + self.groups[g].add(test_file)
>>>> +
>>>> + def find_tests(self, groups=None, exclude_groups=None, tests=None):
>>>> + """
>>>> + 1. Take all tests from @groups,
>>>> + or just all tests if @groups is None and @tests is None
>>>> + or nothing if @groups is None and @tests is not None
>>>> + 2. Drop tests, which are in at least one of @exclude_groups or in
>>>> + 'disabled' group (if 'disabled' is not listed in @groups)
>>>> + 3. Add tests from @tests
>>>> + """
>>>> + if groups is None:
>>>> + groups = []
>>>> + if exclude_groups is None:
>>>> + exclude_groups = []
>>>> + if tests is None:
>>>> + tests = []
>>>> +
>>>> + if groups:
>>>> + res = set().union(*(self.groups[g] for g in groups))
>>>> + elif tests:
>>>> + res = set()
>>>> + else:
>>>> + res = set(self.all_tests)
>>>> +
>>>> + if 'disabled' not in groups and 'disabled' not in exclude_groups:
>>>> + exclude_groups.append('disabled')
>>>> +
>>>> + res = res.difference(*(self.groups[g] for g in exclude_groups))
>>>> +
>>>> + # We want to add @tests. But for compatibility with old test names,
>>>> + # we should convert any number < 100 to number padded by
>>>> + # leading zeroes, like 1 -> 001 and 23 -> 023.
>>>> + for t in tests:
>>>> + if re.fullmatch(r'\d{1,2}', t):
>>>> + res.add('0' * (3 - len(t)) + t)
>>>> + else:
>>>> + res.add(t)
>>>> +
>>>> + return sorted(res)
>>>> +
>>>> + def find_tests_argv(self, argv):
>>>> + args, remaining = self.argparser.parse_known_args(argv)
>>>> +
>>>> + if args.groups is not None:
>>>> + args.groups = args.groups.split(',')
>>>> +
>>>> + if args.exclude_groups is not None:
>>>> + args.exclude_groups = args.exclude_groups.split(',')
>>>> +
>>>> + return self.find_tests(**vars(args)), remaining
>>>> +
>>>> +
>>>> +def find_tests(argv, test_dir=None):
>>>> + tf = TestFinder(test_dir)
>>>> +
>>>> + if test_dir is None:
>>>> + group_local = 'group.local'
>>>> + else:
>>>> + group_local = os.path.join(test_dir, 'group.local')
>>>> + if os.path.isfile(group_local):
>>>> + tf.add_group_file(group_local)
>>>> +
>>>> + return tf.find_tests_argv(argv)
>>>> +
>>>> +
>>>> +if __name__ == '__main__':
>>>> + if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
>>>> + TestFinder.argparser.print_help()
>>>> + exit()
>>>> +
>>>> + tests, remaining_argv = find_tests(sys.argv[1:])
>>>> + print('\n'.join(tests))
>>>> + if remaining_argv:
>>>> + print('\nUnhandled options: ', remaining_argv)
>>>
>>> I think unhandled options shouldn't be considered an error and we
>>> shouldn't even try to find and display any tests then. I'd either print
>>> the help text or have an error message that refers to --help.
>>
>> As I considered running this script as executable for testing purposes, it's
>> good to know, which options are not handled..
>
> Yes, that makes sense. I just think it should be an error and not just
> an additional hint at the end.
>
It's not and error, as usual case will leave some arguments for TestEnv and TestRunner.
Assume you run ./check with some arguments.. And it works bad. You assume that problem is in testfinder.py. Then, you just take the entire list of options and call ./testfinder.py with them. And it shows, how it parses its arguments, and what is reminded. Seems correct and shouldn't be an error.
Still, maybe better to print unhandled options first, to be more noticeable.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-22 12:49 ` Vladimir Sementsov-Ogievskiy
@ 2020-04-22 13:06 ` Kevin Wolf
0 siblings, 0 replies; 30+ messages in thread
From: Kevin Wolf @ 2020-04-22 13:06 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
Am 22.04.2020 um 14:49 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 22.04.2020 14:53, Kevin Wolf wrote:
> > Am 22.04.2020 um 07:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > 21.04.2020 19:56, Kevin Wolf wrote:
> > > > Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > > > +if __name__ == '__main__':
> > > > > + if len(sys.argv) == 2 and sys.argv[1] in ['-h', '--help']:
> > > > > + TestFinder.argparser.print_help()
> > > > > + exit()
> > > > > +
> > > > > + tests, remaining_argv = find_tests(sys.argv[1:])
> > > > > + print('\n'.join(tests))
> > > > > + if remaining_argv:
> > > > > + print('\nUnhandled options: ', remaining_argv)
> > > >
> > > > I think unhandled options shouldn't be considered an error and we
> > > > shouldn't even try to find and display any tests then. I'd either print
> > > > the help text or have an error message that refers to --help.
> > >
> > > As I considered running this script as executable for testing purposes, it's
> > > good to know, which options are not handled..
> >
> > Yes, that makes sense. I just think it should be an error and not just
> > an additional hint at the end.
> >
>
> It's not and error, as usual case will leave some arguments for
> TestEnv and TestRunner. Assume you run ./check with some arguments..
>
> And it works bad. You assume that problem is in testfinder.py. Then,
> you just take the entire list of options and call ./testfinder.py with
> them. And it shows, how it parses its arguments, and what is reminded.
> Seems correct and shouldn't be an error.
Hm, I didn't consider this use case. Fair enough then.
> Still, maybe better to print unhandled options first, to be more
> noticeable.
Actually, I think it's more noticeable at the end when you execute the
script standalone, especially when it prints a long list.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-04-21 16:56 ` Kevin Wolf
2020-04-22 5:35 ` Vladimir Sementsov-Ogievskiy
@ 2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
2020-05-08 8:49 ` Kevin Wolf
2020-05-13 21:58 ` John Snow
1 sibling, 2 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-05-07 17:43 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
21.04.2020 19:56, Kevin Wolf wrote:
> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> Add python script with new logic of searching for tests:
>>
>> Current ./check behavior:
>> - tests are named [0-9][0-9][0-9]
>> - tests must be registered in group file (even if test doesn't belong
>> to any group, like 142)
>>
>> Behavior of new test:
>> - group file is dropped
>> - tests are searched by file-name instead of group file, so it's not
>> needed more to "register the test", just create it with name
>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>> recommended for new tests
>
> I wonder if a tests/ subdirectory instead of the -test suffix would
> organise things a bit better.
>
It will make more difficult to import iotests.py.. Calling common.rc from
bash tests will need to be modified too.
So, we'll need additional line in all python tests:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
which doesn't seem to be good practice.. So, instead we'd better call tests with PYTHONPATH set appropriately..
and modify bash tests to do
. ../common.rc
. ../common.filter
<etc>
or again, better to export BASH_TEST_LIB directory.
Is it worth doing?
I think, there are two variants:
1) keep as is: all in one directory, add *-test notation
2) go further and restructure to something like:
iotests/<only check script and its dependencies here>
iotests/tests/<all-tests-here>
iotests/lib/python/iotests.py
iotests/lib/bash/<common.rc and friends here>
And then, check script will export PYTHONPATH and BASH_TEST_LIB variables.
In case of [2], I definitely prefer to split directory tree refactoring to the separate series. So this series just rewrite check into python and get rid of group file.
What do you prefer?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
@ 2020-05-08 8:49 ` Kevin Wolf
2020-05-08 9:42 ` Vladimir Sementsov-Ogievskiy
2020-05-13 21:58 ` John Snow
1 sibling, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2020-05-08 8:49 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
Am 07.05.2020 um 19:43 hat Vladimir Sementsov-Ogievskiy geschrieben:
> 21.04.2020 19:56, Kevin Wolf wrote:
> > Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > Add python script with new logic of searching for tests:
> > >
> > > Current ./check behavior:
> > > - tests are named [0-9][0-9][0-9]
> > > - tests must be registered in group file (even if test doesn't belong
> > > to any group, like 142)
> > >
> > > Behavior of new test:
> > > - group file is dropped
> > > - tests are searched by file-name instead of group file, so it's not
> > > needed more to "register the test", just create it with name
> > > *-test. Old names like [0-9][0-9][0-9] are supported too, but not
> > > recommended for new tests
> >
> > I wonder if a tests/ subdirectory instead of the -test suffix would
> > organise things a bit better.
> >
>
> It will make more difficult to import iotests.py.. Calling common.rc from
> bash tests will need to be modified too.
>
> So, we'll need additional line in all python tests:
>
> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
Hm, yes, this is nasty.
Would it be any better have a wrapper script that just imports the
actual test so that the import path would still contain the main
qemu-iotests/ directory?
> which doesn't seem to be good practice.. So, instead we'd better call
> tests with PYTHONPATH set appropriately..
This is another option, especially if we do want to create a lib/.
> and modify bash tests to do
> . ../common.rc
> . ../common.filter
> <etc>
>
> or again, better to export BASH_TEST_LIB directory.
I don't think this is necessary because the working directory wouldn't
change, so bash scripts should just keep working as before.
> Is it worth doing?
>
> I think, there are two variants:
>
> 1) keep as is: all in one directory, add *-test notation
I think it would make it rather hard to find the files that belong to
the test harness implementation between all the tests. Currently, 'ls'
in the qemu-iotests directory is still kind of usable because all the
test cases are at the start and everything that comes later is not a
test.
> 2) go further and restructure to something like:
>
> iotests/<only check script and its dependencies here>
> iotests/tests/<all-tests-here>
> iotests/lib/python/iotests.py
> iotests/lib/bash/<common.rc and friends here>
>
> And then, check script will export PYTHONPATH and BASH_TEST_LIB
> variables.
I think leaving everything except named tests where it is, but setting
PYTHONPATH or having a wrapper script, is still a third option that's
worth considering. It sounds like the most attrative option to me.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-08 8:49 ` Kevin Wolf
@ 2020-05-08 9:42 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-05-08 9:42 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-block, qemu-devel, mreitz, den, jsnow
08.05.2020 11:49, Kevin Wolf wrote:
> Am 07.05.2020 um 19:43 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> 21.04.2020 19:56, Kevin Wolf wrote:
>>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> Add python script with new logic of searching for tests:
>>>>
>>>> Current ./check behavior:
>>>> - tests are named [0-9][0-9][0-9]
>>>> - tests must be registered in group file (even if test doesn't belong
>>>> to any group, like 142)
>>>>
>>>> Behavior of new test:
>>>> - group file is dropped
>>>> - tests are searched by file-name instead of group file, so it's not
>>>> needed more to "register the test", just create it with name
>>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>>> recommended for new tests
>>>
>>> I wonder if a tests/ subdirectory instead of the -test suffix would
>>> organise things a bit better.
>>>
>>
>> It will make more difficult to import iotests.py.. Calling common.rc from
>> bash tests will need to be modified too.
>>
>> So, we'll need additional line in all python tests:
>>
>> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>
> Hm, yes, this is nasty.
>
> Would it be any better have a wrapper script that just imports the
> actual test so that the import path would still contain the main
> qemu-iotests/ directory?
I think, that better to keep all tests self-executable without any additional preparations, just set all interface environment variables and run test (or use check to automate environment initialization). In this way we can support any executable, not even limited to python and bash (I doubt that we need more, but keeping test interface full-defined by environment variables seems a good thing).
>
>> which doesn't seem to be good practice.. So, instead we'd better call
>> tests with PYTHONPATH set appropriately..
>
> This is another option, especially if we do want to create a lib/.
>
>> and modify bash tests to do
>> . ../common.rc
>> . ../common.filter
>> <etc>
>>
>> or again, better to export BASH_TEST_LIB directory.
>
> I don't think this is necessary because the working directory wouldn't
> change, so bash scripts should just keep working as before.
Hmm, right. Probably same works for python tests as well?
>
>> Is it worth doing?
>>
>> I think, there are two variants:
>>
>> 1) keep as is: all in one directory, add *-test notation
>
> I think it would make it rather hard to find the files that belong to
> the test harness implementation between all the tests. Currently, 'ls'
> in the qemu-iotests directory is still kind of usable because all the
> test cases are at the start and everything that comes later is not a
> test.
>
>> 2) go further and restructure to something like:
>>
>> iotests/<only check script and its dependencies here>
>> iotests/tests/<all-tests-here>
>> iotests/lib/python/iotests.py
>> iotests/lib/bash/<common.rc and friends here>
>>
>> And then, check script will export PYTHONPATH and BASH_TEST_LIB
>> variables.
>
> I think leaving everything except named tests where it is, but setting
> PYTHONPATH or having a wrapper script, is still a third option that's
> worth considering. It sounds like the most attrative option to me.
>
Works for me too. OK, let's go this way.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
2020-05-08 8:49 ` Kevin Wolf
@ 2020-05-13 21:58 ` John Snow
2020-05-14 4:54 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 30+ messages in thread
From: John Snow @ 2020-05-13 21:58 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Kevin Wolf
Cc: den, qemu-devel, qemu-block, mreitz
On 5/7/20 1:43 PM, Vladimir Sementsov-Ogievskiy wrote:
> 21.04.2020 19:56, Kevin Wolf wrote:
>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>> Add python script with new logic of searching for tests:
>>>
>>> Current ./check behavior:
>>> - tests are named [0-9][0-9][0-9]
>>> - tests must be registered in group file (even if test doesn't belong
>>> to any group, like 142)
>>>
>>> Behavior of new test:
>>> - group file is dropped
>>> - tests are searched by file-name instead of group file, so it's not
>>> needed more to "register the test", just create it with name
>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>> recommended for new tests
>>
>> I wonder if a tests/ subdirectory instead of the -test suffix would
>> organise things a bit better.
>>
>
> It will make more difficult to import iotests.py.. Calling common.rc from
> bash tests will need to be modified too.
>
> So, we'll need additional line in all python tests:
>
> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>
> which doesn't seem to be good practice.. So, instead we'd better call
> tests with PYTHONPATH set appropriately..
>
Just chiming in to say that it's largely bad practice because it
confuses pylint, mypy and friends -- if we want to keep pushing our CI
code analysis for python in that direction, this will be a barrier.
Using PYTHONPATH is better, because it isolates the script itself from
the environment, but requires you to now always set PYTHONPATH to
execute any of the individual iotests.
Not actually a big deal, because iotests already expect a large number
of environment variables to be set. It's not really a huge net loss in
convenience, I think.
looks like that's the direction you're headed in anyway based on
discussion, so that's good.
--js
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-13 21:58 ` John Snow
@ 2020-05-14 4:54 ` Vladimir Sementsov-Ogievskiy
2020-05-14 5:06 ` John Snow
0 siblings, 1 reply; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-05-14 4:54 UTC (permalink / raw)
To: John Snow, Kevin Wolf; +Cc: den, qemu-devel, qemu-block, mreitz
14.05.2020 00:58, John Snow wrote:
>
>
> On 5/7/20 1:43 PM, Vladimir Sementsov-Ogievskiy wrote:
>> 21.04.2020 19:56, Kevin Wolf wrote:
>>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>> Add python script with new logic of searching for tests:
>>>>
>>>> Current ./check behavior:
>>>> - tests are named [0-9][0-9][0-9]
>>>> - tests must be registered in group file (even if test doesn't belong
>>>> to any group, like 142)
>>>>
>>>> Behavior of new test:
>>>> - group file is dropped
>>>> - tests are searched by file-name instead of group file, so it's not
>>>> needed more to "register the test", just create it with name
>>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>>> recommended for new tests
>>>
>>> I wonder if a tests/ subdirectory instead of the -test suffix would
>>> organise things a bit better.
>>>
>>
>> It will make more difficult to import iotests.py.. Calling common.rc from
>> bash tests will need to be modified too.
>>
>> So, we'll need additional line in all python tests:
>>
>> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>>
>> which doesn't seem to be good practice.. So, instead we'd better call
>> tests with PYTHONPATH set appropriately..
>>
>
> Just chiming in to say that it's largely bad practice because it
> confuses pylint, mypy and friends -- if we want to keep pushing our CI
> code analysis for python in that direction, this will be a barrier.
>
> Using PYTHONPATH is better, because it isolates the script itself from
> the environment, but requires you to now always set PYTHONPATH to
> execute any of the individual iotests.
>
> Not actually a big deal, because iotests already expect a large number
> of environment variables to be set. It's not really a huge net loss in
> convenience, I think.
>
> looks like that's the direction you're headed in anyway based on
> discussion, so that's good.
>
Hm, does PYTHONPATH-way works good with mypy and friends? Probably, it should
be set when checking the code? So, actually developers will have to set
PYTHONPATH by hand to always contain some directories within qemu source tree?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-14 4:54 ` Vladimir Sementsov-Ogievskiy
@ 2020-05-14 5:06 ` John Snow
2020-05-14 5:31 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 30+ messages in thread
From: John Snow @ 2020-05-14 5:06 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Kevin Wolf
Cc: den, qemu-devel, qemu-block, mreitz
On 5/14/20 12:54 AM, Vladimir Sementsov-Ogievskiy wrote:
> 14.05.2020 00:58, John Snow wrote:
>>
>>
>> On 5/7/20 1:43 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> 21.04.2020 19:56, Kevin Wolf wrote:
>>>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>>> Add python script with new logic of searching for tests:
>>>>>
>>>>> Current ./check behavior:
>>>>> - tests are named [0-9][0-9][0-9]
>>>>> - tests must be registered in group file (even if test doesn't
>>>>> belong
>>>>> to any group, like 142)
>>>>>
>>>>> Behavior of new test:
>>>>> - group file is dropped
>>>>> - tests are searched by file-name instead of group file, so it's
>>>>> not
>>>>> needed more to "register the test", just create it with name
>>>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>>>> recommended for new tests
>>>>
>>>> I wonder if a tests/ subdirectory instead of the -test suffix would
>>>> organise things a bit better.
>>>>
>>>
>>> It will make more difficult to import iotests.py.. Calling common.rc
>>> from
>>> bash tests will need to be modified too.
>>>
>>> So, we'll need additional line in all python tests:
>>>
>>> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>>>
>>> which doesn't seem to be good practice.. So, instead we'd better call
>>> tests with PYTHONPATH set appropriately..
>>>
>>
>> Just chiming in to say that it's largely bad practice because it
>> confuses pylint, mypy and friends -- if we want to keep pushing our CI
>> code analysis for python in that direction, this will be a barrier.
>>
>> Using PYTHONPATH is better, because it isolates the script itself from
>> the environment, but requires you to now always set PYTHONPATH to
>> execute any of the individual iotests.
>>
>> Not actually a big deal, because iotests already expect a large number
>> of environment variables to be set. It's not really a huge net loss in
>> convenience, I think.
>>
>> looks like that's the direction you're headed in anyway based on
>> discussion, so that's good.
>>
>
> Hm, does PYTHONPATH-way works good with mypy and friends? Probably, it
> should
> be set when checking the code? So, actually developers will have to set
> PYTHONPATH by hand to always contain some directories within qemu source
> tree?
>
pylint respects PYTHONPATH but mypy doesn't. mypy uses MYPYPATH, but I
wouldn't worry about accommodating it. It's a fussy tool and we're only
ever going to run it from very specific environments.
You don't need to worry too much about what environment variables these
tools take; it's only worth noting that "sys.path" hacks tend to make
these tools harder to use.
As for setting PYTHONPATH by hand ... There are a few places in the QEMU
tree where we set PYTHONPATH already, and the individual iotests already
don't work if they're not launched by `check`, because they're missing a
ton of environment variables.
It's not going to be too bad to set PYTHONPATH in the launcher script,
is it?
(Or are we replacing the top-level script with a python one?)
Really, the same is true of pylint, too. It's only annoying to deal with
sys.path hacking because it can't be worked around in those CQA tools.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 06/10] iotests: add testfinder.py
2020-05-14 5:06 ` John Snow
@ 2020-05-14 5:31 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 30+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-05-14 5:31 UTC (permalink / raw)
To: John Snow, Kevin Wolf; +Cc: den, qemu-devel, qemu-block, mreitz
14.05.2020 08:06, John Snow wrote:
>
>
> On 5/14/20 12:54 AM, Vladimir Sementsov-Ogievskiy wrote:
>> 14.05.2020 00:58, John Snow wrote:
>>>
>>>
>>> On 5/7/20 1:43 PM, Vladimir Sementsov-Ogievskiy wrote:
>>>> 21.04.2020 19:56, Kevin Wolf wrote:
>>>>> Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>>>>> Add python script with new logic of searching for tests:
>>>>>>
>>>>>> Current ./check behavior:
>>>>>> - tests are named [0-9][0-9][0-9]
>>>>>> - tests must be registered in group file (even if test doesn't
>>>>>> belong
>>>>>> to any group, like 142)
>>>>>>
>>>>>> Behavior of new test:
>>>>>> - group file is dropped
>>>>>> - tests are searched by file-name instead of group file, so it's
>>>>>> not
>>>>>> needed more to "register the test", just create it with name
>>>>>> *-test. Old names like [0-9][0-9][0-9] are supported too, but not
>>>>>> recommended for new tests
>>>>>
>>>>> I wonder if a tests/ subdirectory instead of the -test suffix would
>>>>> organise things a bit better.
>>>>>
>>>>
>>>> It will make more difficult to import iotests.py.. Calling common.rc
>>>> from
>>>> bash tests will need to be modified too.
>>>>
>>>> So, we'll need additional line in all python tests:
>>>>
>>>> sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>>>>
>>>> which doesn't seem to be good practice.. So, instead we'd better call
>>>> tests with PYTHONPATH set appropriately..
>>>>
>>>
>>> Just chiming in to say that it's largely bad practice because it
>>> confuses pylint, mypy and friends -- if we want to keep pushing our CI
>>> code analysis for python in that direction, this will be a barrier.
>>>
>>> Using PYTHONPATH is better, because it isolates the script itself from
>>> the environment, but requires you to now always set PYTHONPATH to
>>> execute any of the individual iotests.
>>>
>>> Not actually a big deal, because iotests already expect a large number
>>> of environment variables to be set. It's not really a huge net loss in
>>> convenience, I think.
>>>
>>> looks like that's the direction you're headed in anyway based on
>>> discussion, so that's good.
>>>
>>
>> Hm, does PYTHONPATH-way works good with mypy and friends? Probably, it
>> should
>> be set when checking the code? So, actually developers will have to set
>> PYTHONPATH by hand to always contain some directories within qemu source
>> tree?
>>
>
> pylint respects PYTHONPATH but mypy doesn't. mypy uses MYPYPATH, but I
> wouldn't worry about accommodating it. It's a fussy tool and we're only
> ever going to run it from very specific environments.
>
Hmm, recently I installed dense-analysis/ale plugin into my vim which does mypy checking (among other things).. And most probably, I'll have to set these variables to keep it working. But it's not a big problem.
> You don't need to worry too much about what environment variables these
> tools take; it's only worth noting that "sys.path" hacks tend to make
> these tools harder to use.
>
>
> As for setting PYTHONPATH by hand ... There are a few places in the QEMU
> tree where we set PYTHONPATH already, and the individual iotests already
> don't work if they're not launched by `check`, because they're missing a
> ton of environment variables.
>
> It's not going to be too bad to set PYTHONPATH in the launcher script,
> is it?
>
> (Or are we replacing the top-level script with a python one?)
Yes we do, bright future is near:) But it's not a problem to set PYTHONPATH in it. Anyway, we run all tests as executables, so passing PYTHONPATH is a valid thing to do.
>
>
>
>
> Really, the same is true of pylint, too. It's only annoying to deal with
> sys.path hacking because it can't be worked around in those CQA tools.
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v3 03/10] iotests/283: make executable
2020-04-21 7:35 ` [PATCH v3 03/10] iotests/283: make executable Vladimir Sementsov-Ogievskiy
2020-04-21 12:55 ` Eric Blake
@ 2020-05-14 6:17 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-05-14 6:17 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 4/21/20 9:35 AM, Vladimir Sementsov-Ogievskiy wrote:
> All other test files are executable, except for this one. Fix that.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/283 | 0
> 1 file changed, 0 insertions(+), 0 deletions(-)
> mode change 100644 => 100755 tests/qemu-iotests/283
>
> diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283
> old mode 100644
> new mode 100755
>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2020-05-14 6:23 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-21 7:35 [PATCH v3 00/10] Rework iotests/check Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 01/10] iotests/277: use dot slash for nbd-fault-injector.py running Vladimir Sementsov-Ogievskiy
2020-04-21 12:54 ` Eric Blake
2020-04-21 13:04 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 02/10] iotests: fix some whitespaces in test output files Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 03/10] iotests/283: make executable Vladimir Sementsov-Ogievskiy
2020-04-21 12:55 ` Eric Blake
2020-05-14 6:17 ` Philippe Mathieu-Daudé
2020-04-21 7:35 ` [PATCH v3 04/10] iotests/check: move QEMU_VXHS_PROG to common.rc Vladimir Sementsov-Ogievskiy
2020-04-21 16:03 ` Kevin Wolf
2020-04-22 5:14 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 05/10] iotests: define group in each iotest Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 06/10] iotests: add testfinder.py Vladimir Sementsov-Ogievskiy
2020-04-21 16:56 ` Kevin Wolf
2020-04-22 5:35 ` Vladimir Sementsov-Ogievskiy
2020-04-22 11:53 ` Kevin Wolf
2020-04-22 12:49 ` Vladimir Sementsov-Ogievskiy
2020-04-22 13:06 ` Kevin Wolf
2020-05-07 17:43 ` Vladimir Sementsov-Ogievskiy
2020-05-08 8:49 ` Kevin Wolf
2020-05-08 9:42 ` Vladimir Sementsov-Ogievskiy
2020-05-13 21:58 ` John Snow
2020-05-14 4:54 ` Vladimir Sementsov-Ogievskiy
2020-05-14 5:06 ` John Snow
2020-05-14 5:31 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 07/10] iotests: add testenv.py Vladimir Sementsov-Ogievskiy
2020-04-21 7:35 ` [PATCH v3 08/10] iotests: add testrunner.py Vladimir Sementsov-Ogievskiy
2020-04-21 7:36 ` [PATCH v3 09/10] iotests: rewrite check into python Vladimir Sementsov-Ogievskiy
2020-04-21 7:40 ` Vladimir Sementsov-Ogievskiy
2020-04-21 7:36 ` [PATCH v3 10/10] iotests: rename 169 and 199 Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.