* [PATCH v2 01/17] iotests.py: img_info_log(): rename imgopts argument
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 02/17] iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd' Vladimir Sementsov-Ogievskiy
` (16 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
We are going to support IMGOPTS environment variable like in bash
tests. Corresponding global variable in iotests.py should be called
imgopts. So to not interfere with function argument, rename it in
advance.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/210 | 8 ++++----
tests/qemu-iotests/iotests.py | 5 +++--
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/tests/qemu-iotests/210 b/tests/qemu-iotests/210
index 5a62ed4dd1..79b4967225 100755
--- a/tests/qemu-iotests/210
+++ b/tests/qemu-iotests/210
@@ -62,7 +62,7 @@ with iotests.FilePath('t.luks') as disk_path, \
'driver=luks,file.driver=file,file.filename=%s,key-secret=keysec0' % (disk_path),
filter_path=disk_path,
extra_args=['--object', 'secret,id=keysec0,data=foo'],
- imgopts=True)
+ use_image_opts=True)
#
# Successful image creation (with non-default options)
@@ -96,7 +96,7 @@ with iotests.FilePath('t.luks') as disk_path, \
'driver=luks,file.driver=file,file.filename=%s,key-secret=keysec0' % (disk_path),
filter_path=disk_path,
extra_args=['--object', 'secret,id=keysec0,data=foo'],
- imgopts=True)
+ use_image_opts=True)
#
# Invalid BlockdevRef
@@ -132,7 +132,7 @@ with iotests.FilePath('t.luks') as disk_path, \
'driver=luks,file.driver=file,file.filename=%s,key-secret=keysec0' % (disk_path),
filter_path=disk_path,
extra_args=['--object', 'secret,id=keysec0,data=foo'],
- imgopts=True)
+ use_image_opts=True)
#
# Invalid sizes
@@ -176,4 +176,4 @@ with iotests.FilePath('t.luks') as disk_path, \
'driver=luks,file.driver=file,file.filename=%s,key-secret=keysec0' % (disk_path),
filter_path=disk_path,
extra_args=['--object', 'secret,id=keysec0,data=foo'],
- imgopts=True)
+ use_image_opts=True)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 89663dac06..0d99dd841f 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -187,9 +187,10 @@ def qemu_img_log(*args):
log(result, filters=[filter_testfiles])
return result
-def img_info_log(filename, filter_path=None, imgopts=False, extra_args=()):
+def img_info_log(filename, filter_path=None, use_image_opts=False,
+ extra_args=()):
args = ['info']
- if imgopts:
+ if use_image_opts:
args.append('--image-opts')
else:
args += ['-f', imgfmt]
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 02/17] iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 01/17] iotests.py: img_info_log(): rename imgopts argument Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 10:40 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 03/17] iotests: drop qemu_img_verbose() helper Vladimir Sementsov-Ogievskiy
` (15 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
Adding support of IMGOPTS (like in bash tests) allows user to pass a
lot of different options. Still, some may require additional logic.
Now we want compression_type option, so add some smart logic around it:
ignore compression_type=zstd in IMGOPTS, if test want qcow2 in
compatibility mode. As well, ignore compression_type for non-qcow2
formats.
Note that we may instead add support only to qemu_img_create(), but
that works bad:
1. We'll have to update a lot of tests to use qemu_img_create instead
of qemu_img('create'). (still, we may want do it anyway, but no
reason to create a dependancy between task of supporting IMGOPTS and
updating a lot of tests)
2. Some tests use qemu_img_pipe('create', ..) - even more work on
updating
3. Even if we update all tests to go through qemu_img_create, we'll
need a way to avoid creating new tests using qemu_img*('create') -
add assertions.. That doesn't seem good.
So, let's add support of IMGOPTS to most generic
qemu_img_pipe_and_status().
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/iotests.py | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 0d99dd841f..03cc00366e 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -16,6 +16,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
+import argparse
import atexit
import bz2
from collections import OrderedDict
@@ -121,11 +122,35 @@ def qemu_tool_pipe_and_status(tool: str, args: Sequence[str],
{-subp.returncode}: {cmd}\n')
return (output, subp.returncode)
+def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
+ if not args or args[0] != 'create':
+ return list(args)
+ args = args[1:]
+
+ p = argparse.ArgumentParser(allow_abbrev=False)
+ p.add_argument('-f')
+ parsed, remaining = p.parse_known_args(args)
+
+ result = ['create']
+ if parsed.f is not None:
+ result += ['-f', parsed.f]
+
+ # IMGOPTS most probably contain options specific for the selected format,
+ # like extended_l2 or compression_type for qcow2. Test may want to create
+ # additional images in other formats that doesn't support these options.
+ # So, use IMGOPTS only for images created in imgfmt format.
+ if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
+ result += ['-o', os.environ['IMGOPTS']]
+
+ result += remaining
+
+ return result
+
def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:
"""
Run qemu-img and return both its output and its exit code
"""
- full_args = qemu_img_args + list(args)
+ full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
return qemu_tool_pipe_and_status('qemu-img', full_args)
def qemu_img(*args: str) -> int:
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 02/17] iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
2021-07-20 11:38 ` [PATCH v2 02/17] iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd' Vladimir Sementsov-Ogievskiy
@ 2021-09-13 10:40 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 10:40 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> Adding support of IMGOPTS (like in bash tests) allows user to pass a
> lot of different options. Still, some may require additional logic.
>
> Now we want compression_type option, so add some smart logic around it:
> ignore compression_type=zstd in IMGOPTS, if test want qcow2 in
> compatibility mode. As well, ignore compression_type for non-qcow2
> formats.
>
> Note that we may instead add support only to qemu_img_create(), but
> that works bad:
>
> 1. We'll have to update a lot of tests to use qemu_img_create instead
> of qemu_img('create'). (still, we may want do it anyway, but no
> reason to create a dependancy between task of supporting IMGOPTS and
> updating a lot of tests)
>
> 2. Some tests use qemu_img_pipe('create', ..) - even more work on
> updating
>
> 3. Even if we update all tests to go through qemu_img_create, we'll
> need a way to avoid creating new tests using qemu_img*('create') -
> add assertions.. That doesn't seem good.
>
> So, let's add support of IMGOPTS to most generic
> qemu_img_pipe_and_status().
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/iotests.py | 27 ++++++++++++++++++++++++++-
> 1 file changed, 26 insertions(+), 1 deletion(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 03/17] iotests: drop qemu_img_verbose() helper
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 01/17] iotests.py: img_info_log(): rename imgopts argument Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 02/17] iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd' Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 10:49 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img Vladimir Sementsov-Ogievskiy
` (14 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
qemu_img_verbose() has a drawback of not going through generic
qemu_img_pipe_and_status(). qemu_img_verbose() is not very popular, so
update the only two users to qemu_img_log() and drop qemu_img_verbose()
at all.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/044 | 5 +++--
tests/qemu-iotests/044.out | 1 +
tests/qemu-iotests/209 | 7 ++++---
tests/qemu-iotests/209.out | 2 ++
tests/qemu-iotests/iotests.py | 8 --------
5 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 64b18eb7c8..714329eb16 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -24,7 +24,7 @@ import os
import qcow2
from qcow2 import QcowHeader
import iotests
-from iotests import qemu_img, qemu_img_verbose, qemu_io
+from iotests import qemu_img, qemu_img_log, qemu_io
import struct
import subprocess
import sys
@@ -112,9 +112,10 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
def test_grow_refcount_table(self):
qemu_io('-c', 'write 3800M 1M', test_img)
- qemu_img_verbose('check' , test_img)
+ qemu_img_log('check' , test_img)
pass
if __name__ == '__main__':
+ iotests.activate_logging()
iotests.main(supported_fmts=['qcow2'],
supported_protocols=['file'])
diff --git a/tests/qemu-iotests/044.out b/tests/qemu-iotests/044.out
index 703cf3dee1..ff663b17d7 100644
--- a/tests/qemu-iotests/044.out
+++ b/tests/qemu-iotests/044.out
@@ -1,6 +1,7 @@
No errors were found on the image.
7292415/33554432 = 21.73% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 4296217088
+
.
----------------------------------------------------------------------
Ran 1 tests
diff --git a/tests/qemu-iotests/209 b/tests/qemu-iotests/209
index ff7efea11b..f6ad08ec42 100755
--- a/tests/qemu-iotests/209
+++ b/tests/qemu-iotests/209
@@ -20,8 +20,8 @@
#
import iotests
-from iotests import qemu_img_create, qemu_io, qemu_img_verbose, qemu_nbd, \
- file_path
+from iotests import qemu_img_create, qemu_io, qemu_img_log, qemu_nbd, \
+ file_path, log
iotests.script_initialize(supported_fmts=['qcow2'])
@@ -33,4 +33,5 @@ qemu_img_create('-f', iotests.imgfmt, disk, '1M')
qemu_io('-f', iotests.imgfmt, '-c', 'write 0 512K', disk)
qemu_nbd('-k', nbd_sock, '-x', 'exp', '-f', iotests.imgfmt, disk)
-qemu_img_verbose('map', '-f', 'raw', '--output=json', nbd_uri)
+qemu_img_log('map', '-f', 'raw', '--output=json', nbd_uri)
+log('done.') # avoid new line at the end of output file
diff --git a/tests/qemu-iotests/209.out b/tests/qemu-iotests/209.out
index f27be3fa7b..515906ac7a 100644
--- a/tests/qemu-iotests/209.out
+++ b/tests/qemu-iotests/209.out
@@ -1,2 +1,4 @@
[{ "start": 0, "length": 524288, "depth": 0, "present": true, "zero": false, "data": true, "offset": 0},
{ "start": 524288, "length": 524288, "depth": 0, "present": true, "zero": true, "data": false, "offset": 524288}]
+
+done.
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 03cc00366e..def6ae2475 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -195,14 +195,6 @@ def qemu_img_measure(*args):
def qemu_img_check(*args):
return json.loads(qemu_img_pipe("check", "--output", "json", *args))
-def qemu_img_verbose(*args):
- '''Run qemu-img without suppressing its output and return the exit code'''
- exitcode = subprocess.call(qemu_img_args + list(args))
- if exitcode < 0:
- sys.stderr.write('qemu-img received signal %i: %s\n'
- % (-exitcode, ' '.join(qemu_img_args + list(args))))
- return exitcode
-
def qemu_img_pipe(*args: str) -> str:
'''Run qemu-img and return its output'''
return qemu_img_pipe_and_status(*args)[0]
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 03/17] iotests: drop qemu_img_verbose() helper
2021-07-20 11:38 ` [PATCH v2 03/17] iotests: drop qemu_img_verbose() helper Vladimir Sementsov-Ogievskiy
@ 2021-09-13 10:49 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 10:49 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> qemu_img_verbose() has a drawback of not going through generic
> qemu_img_pipe_and_status(). qemu_img_verbose() is not very popular, so
> update the only two users to qemu_img_log() and drop qemu_img_verbose()
> at all.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/044 | 5 +++--
> tests/qemu-iotests/044.out | 1 +
> tests/qemu-iotests/209 | 7 ++++---
> tests/qemu-iotests/209.out | 2 ++
> tests/qemu-iotests/iotests.py | 8 --------
> 5 files changed, 10 insertions(+), 13 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 03/17] iotests: drop qemu_img_verbose() helper Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 11:16 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 05/17] iotest 303: explicit compression type Vladimir Sementsov-Ogievskiy
` (13 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
Move the logic to more generic qemu_img_pipe_and_status(). Also behave
better when we have several -o options. And reuse argument parser of
course.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/iotests.py | 36 +++++++++++++++++------------------
1 file changed, 17 insertions(+), 19 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index def6ae2475..484f616270 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -128,9 +128,13 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
args = args[1:]
p = argparse.ArgumentParser(allow_abbrev=False)
+ # -o option may be specified several times
+ p.add_argument('-o', action='append', default=[])
p.add_argument('-f')
parsed, remaining = p.parse_known_args(args)
+ opts_list = parsed.o
+
result = ['create']
if parsed.f is not None:
result += ['-f', parsed.f]
@@ -139,8 +143,18 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
# like extended_l2 or compression_type for qcow2. Test may want to create
# additional images in other formats that doesn't support these options.
# So, use IMGOPTS only for images created in imgfmt format.
- if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
- result += ['-o', os.environ['IMGOPTS']]
+ imgopts = os.environ.get('IMGOPTS')
+ if imgopts and parsed.f == imgfmt:
+ opts_list.insert(0, imgopts)
+
+ # default luks support
+ if parsed.f == 'luks' and \
+ all('key-secret' not in opts for opts in opts_list):
+ result += ['--object', luks_default_secret_object]
+ opts_list.append(luks_default_key_secret_opt)
+
+ for opts in opts_list:
+ result += ['-o', opts]
result += remaining
@@ -171,23 +185,7 @@ def ordered_qmp(qmsg, conv_keys=True):
return qmsg
def qemu_img_create(*args):
- args = list(args)
-
- # default luks support
- if '-f' in args and args[args.index('-f') + 1] == 'luks':
- if '-o' in args:
- i = args.index('-o')
- if 'key-secret' not in args[i + 1]:
- args[i + 1].append(luks_default_key_secret_opt)
- args.insert(i + 2, '--object')
- args.insert(i + 3, luks_default_secret_object)
- else:
- args = ['-o', luks_default_key_secret_opt,
- '--object', luks_default_secret_object] + args
-
- args.insert(0, 'create')
-
- return qemu_img(*args)
+ return qemu_img('create', *args)
def qemu_img_measure(*args):
return json.loads(qemu_img_pipe("measure", "--output", "json", *args))
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img
2021-07-20 11:38 ` [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img Vladimir Sementsov-Ogievskiy
@ 2021-09-13 11:16 ` Hanna Reitz
2021-09-13 13:47 ` Vladimir Sementsov-Ogievskiy
2021-09-14 7:44 ` Vladimir Sementsov-Ogievskiy
0 siblings, 2 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 11:16 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> Move the logic to more generic qemu_img_pipe_and_status(). Also behave
> better when we have several -o options. And reuse argument parser of
> course.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/iotests.py | 36 +++++++++++++++++------------------
> 1 file changed, 17 insertions(+), 19 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index def6ae2475..484f616270 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -128,9 +128,13 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
> args = args[1:]
>
> p = argparse.ArgumentParser(allow_abbrev=False)
> + # -o option may be specified several times
> + p.add_argument('-o', action='append', default=[])
> p.add_argument('-f')
> parsed, remaining = p.parse_known_args(args)
>
> + opts_list = parsed.o
> +
> result = ['create']
> if parsed.f is not None:
> result += ['-f', parsed.f]
> @@ -139,8 +143,18 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
> # like extended_l2 or compression_type for qcow2. Test may want to create
> # additional images in other formats that doesn't support these options.
> # So, use IMGOPTS only for images created in imgfmt format.
> - if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
> - result += ['-o', os.environ['IMGOPTS']]
> + imgopts = os.environ.get('IMGOPTS')
> + if imgopts and parsed.f == imgfmt:
> + opts_list.insert(0, imgopts)
Hm. Yes, IMGOPTS should come first, so it has lower priority. That
means that patch 2 should have inserted IMGOPTS at the beginning of the
parameter list, though, right?
Hanna
> +
> + # default luks support
> + if parsed.f == 'luks' and \
> + all('key-secret' not in opts for opts in opts_list):
> + result += ['--object', luks_default_secret_object]
> + opts_list.append(luks_default_key_secret_opt)
> +
> + for opts in opts_list:
> + result += ['-o', opts]
>
> result += remaining
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img
2021-09-13 11:16 ` Hanna Reitz
@ 2021-09-13 13:47 ` Vladimir Sementsov-Ogievskiy
2021-09-14 7:44 ` Vladimir Sementsov-Ogievskiy
1 sibling, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-09-13 13:47 UTC (permalink / raw)
To: Hanna Reitz, qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow
13.09.2021 14:16, Hanna Reitz wrote:
> On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
>> Move the logic to more generic qemu_img_pipe_and_status(). Also behave
>> better when we have several -o options. And reuse argument parser of
>> course.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/iotests.py | 36 +++++++++++++++++------------------
>> 1 file changed, 17 insertions(+), 19 deletions(-)
>
> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index def6ae2475..484f616270 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -128,9 +128,13 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
>> args = args[1:]
>> p = argparse.ArgumentParser(allow_abbrev=False)
>> + # -o option may be specified several times
>> + p.add_argument('-o', action='append', default=[])
>> p.add_argument('-f')
>> parsed, remaining = p.parse_known_args(args)
>> + opts_list = parsed.o
>> +
>> result = ['create']
>> if parsed.f is not None:
>> result += ['-f', parsed.f]
>> @@ -139,8 +143,18 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
>> # like extended_l2 or compression_type for qcow2. Test may want to create
>> # additional images in other formats that doesn't support these options.
>> # So, use IMGOPTS only for images created in imgfmt format.
>> - if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
>> - result += ['-o', os.environ['IMGOPTS']]
>> + imgopts = os.environ.get('IMGOPTS')
>> + if imgopts and parsed.f == imgfmt:
>> + opts_list.insert(0, imgopts)
>
> Hm. Yes, IMGOPTS should come first, so it has lower priority. That means that patch 2 should have inserted IMGOPTS at the beginning of the parameter list, though, right?
Agree
>
>> +
>> + # default luks support
>> + if parsed.f == 'luks' and \
>> + all('key-secret' not in opts for opts in opts_list):
>> + result += ['--object', luks_default_secret_object]
>> + opts_list.append(luks_default_key_secret_opt)
>> +
>> + for opts in opts_list:
>> + result += ['-o', opts]
>> result += remaining
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img
2021-09-13 11:16 ` Hanna Reitz
2021-09-13 13:47 ` Vladimir Sementsov-Ogievskiy
@ 2021-09-14 7:44 ` Vladimir Sementsov-Ogievskiy
2021-09-14 8:02 ` Hanna Reitz
1 sibling, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-09-14 7:44 UTC (permalink / raw)
To: Hanna Reitz, qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow
13.09.2021 14:16, Hanna Reitz wrote:
> On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
>> Move the logic to more generic qemu_img_pipe_and_status(). Also behave
>> better when we have several -o options. And reuse argument parser of
>> course.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/iotests.py | 36 +++++++++++++++++------------------
>> 1 file changed, 17 insertions(+), 19 deletions(-)
>
> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index def6ae2475..484f616270 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -128,9 +128,13 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
>> args = args[1:]
>> p = argparse.ArgumentParser(allow_abbrev=False)
>> + # -o option may be specified several times
>> + p.add_argument('-o', action='append', default=[])
>> p.add_argument('-f')
>> parsed, remaining = p.parse_known_args(args)
>> + opts_list = parsed.o
>> +
>> result = ['create']
>> if parsed.f is not None:
>> result += ['-f', parsed.f]
>> @@ -139,8 +143,18 @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
>> # like extended_l2 or compression_type for qcow2. Test may want to create
>> # additional images in other formats that doesn't support these options.
>> # So, use IMGOPTS only for images created in imgfmt format.
>> - if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
>> - result += ['-o', os.environ['IMGOPTS']]
>> + imgopts = os.environ.get('IMGOPTS')
>> + if imgopts and parsed.f == imgfmt:
>> + opts_list.insert(0, imgopts)
>
> Hm. Yes, IMGOPTS should come first, so it has lower priority. That means that patch 2 should have inserted IMGOPTS at the beginning of the parameter list, though, right?
Now looking at it closer: no, patch 2 is OK.
It only put into result:
1. 'create'
2. -f <fmt>
3. -o <IMGOPTS>
4. remaining
and that's correct
>
>> +
>> + # default luks support
>> + if parsed.f == 'luks' and \
>> + all('key-secret' not in opts for opts in opts_list):
>> + result += ['--object', luks_default_secret_object]
>> + opts_list.append(luks_default_key_secret_opt)
>> +
>> + for opts in opts_list:
>> + result += ['-o', opts]
>> result += remaining
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img
2021-09-14 7:44 ` Vladimir Sementsov-Ogievskiy
@ 2021-09-14 8:02 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-14 8:02 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 14.09.21 09:44, Vladimir Sementsov-Ogievskiy wrote:
> 13.09.2021 14:16, Hanna Reitz wrote:
>> On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
>>> Move the logic to more generic qemu_img_pipe_and_status(). Also behave
>>> better when we have several -o options. And reuse argument parser of
>>> course.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> tests/qemu-iotests/iotests.py | 36
>>> +++++++++++++++++------------------
>>> 1 file changed, 17 insertions(+), 19 deletions(-)
>>
>> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
>>
>>> diff --git a/tests/qemu-iotests/iotests.py
>>> b/tests/qemu-iotests/iotests.py
>>> index def6ae2475..484f616270 100644
>>> --- a/tests/qemu-iotests/iotests.py
>>> +++ b/tests/qemu-iotests/iotests.py
>>> @@ -128,9 +128,13 @@ def qemu_img_create_prepare_args(args:
>>> List[str]) -> List[str]:
>>> args = args[1:]
>>> p = argparse.ArgumentParser(allow_abbrev=False)
>>> + # -o option may be specified several times
>>> + p.add_argument('-o', action='append', default=[])
>>> p.add_argument('-f')
>>> parsed, remaining = p.parse_known_args(args)
>>> + opts_list = parsed.o
>>> +
>>> result = ['create']
>>> if parsed.f is not None:
>>> result += ['-f', parsed.f]
>>> @@ -139,8 +143,18 @@ def qemu_img_create_prepare_args(args:
>>> List[str]) -> List[str]:
>>> # like extended_l2 or compression_type for qcow2. Test may
>>> want to create
>>> # additional images in other formats that doesn't support
>>> these options.
>>> # So, use IMGOPTS only for images created in imgfmt format.
>>> - if parsed.f == imgfmt and 'IMGOPTS' in os.environ:
>>> - result += ['-o', os.environ['IMGOPTS']]
>>> + imgopts = os.environ.get('IMGOPTS')
>>> + if imgopts and parsed.f == imgfmt:
>>> + opts_list.insert(0, imgopts)
>>
>> Hm. Yes, IMGOPTS should come first, so it has lower priority. That
>> means that patch 2 should have inserted IMGOPTS at the beginning of
>> the parameter list, though, right?
>
> Now looking at it closer: no, patch 2 is OK.
>
> It only put into result:
>
> 1. 'create'
> 2. -f <fmt>
> 3. -o <IMGOPTS>
> 4. remaining
>
> and that's correct
Ah, right!
Hanna
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 05/17] iotest 303: explicit compression type
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 04/17] iotests.py: rewrite default luks support in qemu_img Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 06/17] iotest 065: " Vladimir Sementsov-Ogievskiy
` (12 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
The test prints qcow2 header fields which depends on chosen compression
type. So, let's be explicit in what compression type we want and
independent of IMGOPTS. Test both existing compression types.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/303 | 25 ++++++++++++++++---------
tests/qemu-iotests/303.out | 30 +++++++++++++++++++++++++++++-
2 files changed, 45 insertions(+), 10 deletions(-)
diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303
index 425544c064..9dee2bdfb8 100755
--- a/tests/qemu-iotests/303
+++ b/tests/qemu-iotests/303
@@ -53,12 +53,19 @@ def add_bitmap(num, begin, end, disabled):
log('')
-qemu_img_create('-f', iotests.imgfmt, disk, '10M')
-
-add_bitmap(1, 0, 6, False)
-add_bitmap(2, 6, 8, True)
-dump = ['./qcow2.py', disk, 'dump-header']
-subprocess.run(dump)
-# Dump the metadata in JSON format
-dump.append('-j')
-subprocess.run(dump)
+def test(compression_type: str, json_output: bool) -> None:
+ qemu_img_create('-f', iotests.imgfmt,
+ '-o', f'compression_type={compression_type}',
+ disk, '10M')
+ add_bitmap(1, 0, 6, False)
+ add_bitmap(2, 6, 8, True)
+
+ cmd = ['./qcow2.py', disk, 'dump-header']
+ if json_output:
+ cmd.append('-j')
+
+ subprocess.run(cmd)
+
+
+test('zlib', False)
+test('zstd', True)
diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out
index 7c16998587..b3c70827b7 100644
--- a/tests/qemu-iotests/303.out
+++ b/tests/qemu-iotests/303.out
@@ -80,6 +80,34 @@ extra_data_size 0
Bitmap table type size offset
0 all-zeroes 0 0
+Add bitmap 1
+wrote 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 1048576
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 2097152
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 3145728
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 4194304
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 5242880
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+
+Add bitmap 2
+wrote 1048576/1048576 bytes at offset 6291456
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+wrote 1048576/1048576 bytes at offset 7340032
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+
{
"magic": 1363560955,
"version": 3,
@@ -94,7 +122,7 @@ Bitmap table type size offset
"refcount_table_clusters": 1,
"nb_snapshots": 0,
"snapshot_offset": 0,
- "incompatible_features": 0,
+ "incompatible_features": 8,
"compatible_features": 0,
"autoclear_features": 1,
"refcount_order": 4,
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 06/17] iotest 065: explicit compression type
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 05/17] iotest 303: explicit compression type Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 11:19 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 07/17] iotests.py: filter out successful output of qemu-img crate Vladimir Sementsov-Ogievskiy
` (11 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
The test checks different options. It of course fails if set
IMGOPTS='compression_type=zstd'. So, let's be explicit in what
compression type we want and independent of IMGOPTS. Test both existing
compression types.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/065 | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 3c2ca27627..4b3c5c6c8c 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -88,7 +88,7 @@ class TestQMP(TestImageInfoSpecific):
class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
- img_options = 'compat=0.10'
+ img_options = 'compat=0.10,compression_type=zlib'
json_compare = { 'compat': '0.10', 'refcount-bits': 16,
'compression-type': 'zlib' }
human_compare = [ 'compat: 0.10', 'compression type: zlib',
@@ -96,17 +96,17 @@ class TestQCow2(TestQemuImgInfo):
class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''
- img_options = 'compat=1.1,lazy_refcounts=off'
+ img_options = 'compat=1.1,lazy_refcounts=off,compression_type=zstd'
json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
'refcount-bits': 16, 'corrupt': False,
- 'compression-type': 'zlib', 'extended-l2': False }
- human_compare = [ 'compat: 1.1', 'compression type: zlib',
+ 'compression-type': 'zstd', 'extended-l2': False }
+ human_compare = [ 'compat: 1.1', 'compression type: zstd',
'lazy refcounts: false', 'refcount bits: 16',
'corrupt: false', 'extended l2: false' ]
class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''
- img_options = 'compat=1.1,lazy_refcounts=on'
+ img_options = 'compat=1.1,lazy_refcounts=on,compression_type=zlib'
json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
'refcount-bits': 16, 'corrupt': False,
'compression-type': 'zlib', 'extended-l2': False }
@@ -117,7 +117,7 @@ class TestQCow3Lazy(TestQemuImgInfo):
class TestQCow3NotLazyQMP(TestQMP):
'''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
with lazy refcounts enabled'''
- img_options = 'compat=1.1,lazy_refcounts=off'
+ img_options = 'compat=1.1,lazy_refcounts=off,compression_type=zlib'
qemu_options = 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False,
'refcount-bits': 16, 'corrupt': False,
@@ -127,11 +127,11 @@ class TestQCow3NotLazyQMP(TestQMP):
class TestQCow3LazyQMP(TestQMP):
'''Testing a qcow2 version 3 image with lazy refcounts enabled, opening
with lazy refcounts disabled'''
- img_options = 'compat=1.1,lazy_refcounts=on'
+ img_options = 'compat=1.1,lazy_refcounts=on,compression_type=zstd'
qemu_options = 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True,
'refcount-bits': 16, 'corrupt': False,
- 'compression-type': 'zlib', 'extended-l2': False }
+ 'compression-type': 'zstd', 'extended-l2': False }
TestImageInfoSpecific = None
TestQemuImgInfo = None
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 06/17] iotest 065: explicit compression type
2021-07-20 11:38 ` [PATCH v2 06/17] iotest 065: " Vladimir Sementsov-Ogievskiy
@ 2021-09-13 11:19 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 11:19 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> The test checks different options. It of course fails if set
> IMGOPTS='compression_type=zstd'. So, let's be explicit in what
> compression type we want and independent of IMGOPTS. Test both existing
> compression types.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/065 | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 07/17] iotests.py: filter out successful output of qemu-img crate
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 06/17] iotest 065: " Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 11:46 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 08/17] iotests.py: filter compression type out Vladimir Sementsov-Ogievskiy
` (10 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
The only "feature" of this "Formatting ..." line is that we have to
update it every time we add new option. Let's drop it.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/255.out | 4 ----
tests/qemu-iotests/274.out | 29 -----------------------------
tests/qemu-iotests/280.out | 1 -
tests/qemu-iotests/iotests.py | 10 ++++++++--
4 files changed, 8 insertions(+), 36 deletions(-)
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
index 33b7f22de3..11a05a5213 100644
--- a/tests/qemu-iotests/255.out
+++ b/tests/qemu-iotests/255.out
@@ -3,9 +3,7 @@ Finishing a commit job with background reads
=== Create backing chain and start VM ===
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
=== Start background read requests ===
@@ -23,9 +21,7 @@ Closing the VM while a job is being cancelled
=== Create images and start VM ===
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
wrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/qemu-iotests/274.out b/tests/qemu-iotests/274.out
index 16a95a4850..1d2928e14d 100644
--- a/tests/qemu-iotests/274.out
+++ b/tests/qemu-iotests/274.out
@@ -1,9 +1,6 @@
== Commit tests ==
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 2097152/2097152 bytes at offset 0
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -66,11 +63,8 @@ read 1048576/1048576 bytes at offset 1048576
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
=== Testing HMP commit (top -> mid) ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 2097152/2097152 bytes at offset 0
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -98,11 +92,8 @@ read 1048576/1048576 bytes at offset 1048576
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
=== Testing QMP active commit (top -> mid) ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 2097152/2097152 bytes at offset 0
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -136,11 +127,8 @@ read 1048576/1048576 bytes at offset 1048576
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
=== Testing qemu-img commit (top -> base) ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 2097152/2097152 bytes at offset 0
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -166,11 +154,8 @@ read 1048576/1048576 bytes at offset 1048576
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
=== Testing QMP active commit (top -> base) ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-mid', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1048576 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=2097152 backing_file=TEST_DIR/PID-mid backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 2097152/2097152 bytes at offset 0
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -205,9 +190,7 @@ read 1048576/1048576 bytes at offset 1048576
== Resize tests ==
=== preallocation=off ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 5368709120
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -224,9 +207,7 @@ read 65536/65536 bytes at offset 5368709120
{ "start": 1073741824, "length": 7516192768, "depth": 0, "present": true, "zero": true, "data": false}]
=== preallocation=metadata ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=34359738368 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=32212254720 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 33285996544
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -248,9 +229,7 @@ read 65536/65536 bytes at offset 33285996544
{ "start": 34896609280, "length": 536870912, "depth": 0, "present": true, "zero": true, "data": false, "offset": 2685075456}]
=== preallocation=falloc ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10485760 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=5242880 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 9437184
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -267,9 +246,7 @@ read 65536/65536 bytes at offset 9437184
{ "start": 5242880, "length": 10485760, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
=== preallocation=full ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=16777216 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=8388608 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 11534336
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -286,9 +263,7 @@ read 65536/65536 bytes at offset 11534336
{ "start": 8388608, "length": 4194304, "depth": 0, "present": true, "zero": false, "data": true, "offset": 327680}]
=== preallocation=off ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=393216 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=259072 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 259072
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -306,9 +281,7 @@ read 65536/65536 bytes at offset 259072
{ "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
=== preallocation=off ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=409600 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=262144 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 344064
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -325,9 +298,7 @@ read 65536/65536 bytes at offset 344064
{ "start": 262144, "length": 262144, "depth": 0, "present": true, "zero": true, "data": false}]
=== preallocation=off ===
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=524288 lazy_refcounts=off refcount_bits=16
-Formatting 'TEST_DIR/PID-top', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=262144 backing_file=TEST_DIR/PID-base backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 446464
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/qemu-iotests/280.out b/tests/qemu-iotests/280.out
index 09a0f1a7cb..e39164c579 100644
--- a/tests/qemu-iotests/280.out
+++ b/tests/qemu-iotests/280.out
@@ -1,4 +1,3 @@
-Formatting 'TEST_DIR/PID-base', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=67108864 lazy_refcounts=off refcount_bits=16
=== Launch VM ===
Enabling migration QMP events on VM...
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 484f616270..857676a329 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -108,7 +108,9 @@ def unarchive_sample_image(sample, fname):
def qemu_tool_pipe_and_status(tool: str, args: Sequence[str],
- connect_stderr: bool = True) -> Tuple[str, int]:
+ connect_stderr: bool = True,
+ drop_successful_output: bool = False) \
+ -> Tuple[str, int]:
"""
Run a tool and return both its output and its exit code
"""
@@ -120,6 +122,8 @@ def qemu_tool_pipe_and_status(tool: str, args: Sequence[str],
cmd = ' '.join(args)
sys.stderr.write(f'{tool} received signal \
{-subp.returncode}: {cmd}\n')
+ if drop_successful_output and subp.returncode == 0:
+ output = ''
return (output, subp.returncode)
def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
@@ -164,8 +168,10 @@ def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:
"""
Run qemu-img and return both its output and its exit code
"""
+ is_create = bool(args and args[0] == 'create')
full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
- return qemu_tool_pipe_and_status('qemu-img', full_args)
+ return qemu_tool_pipe_and_status('qemu-img', full_args,
+ drop_successful_output=is_create)
def qemu_img(*args: str) -> int:
'''Run qemu-img and return the exit code'''
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 07/17] iotests.py: filter out successful output of qemu-img crate
2021-07-20 11:38 ` [PATCH v2 07/17] iotests.py: filter out successful output of qemu-img crate Vladimir Sementsov-Ogievskiy
@ 2021-09-13 11:46 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 11:46 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
Subject: s/crate/create/
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> The only "feature" of this "Formatting ..." line is that we have to
> update it every time we add new option. Let's drop it.
Sounds good to me.
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/255.out | 4 ----
> tests/qemu-iotests/274.out | 29 -----------------------------
> tests/qemu-iotests/280.out | 1 -
> tests/qemu-iotests/iotests.py | 10 ++++++++--
> 4 files changed, 8 insertions(+), 36 deletions(-)
Grepping like so:
$ (for f in $(ag -l 'Formatting' | grep '\.out' | sed -e 's/\.out.*//');
do \
echo -n "$f "; \
head -n 1 $f; \
done) | grep python
yields also 149, 237, and 296 as tests whose reference output needs to
be adjusted.
(Although 149 just fails for me altogether, seemingly for the same
reason that makes 210 fail.)
Hanna
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 08/17] iotests.py: filter compression type out
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 07/17] iotests.py: filter out successful output of qemu-img crate Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 12:00 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 09/17] iotest 302: use img_info_log() helper Vladimir Sementsov-Ogievskiy
` (9 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
We want iotests pass with both the default zlib compression and with
IMGOPTS='compression_type=zstd'.
Actually the only test that is interested in real compression type in
test output is 287 (test for qcow2 compression type) and it's in bash.
So for now we can safely filter out compression type in all qcow2
tests.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/206.out | 10 +++++-----
tests/qemu-iotests/242.out | 10 +++++-----
tests/qemu-iotests/274.out | 10 +++++-----
tests/qemu-iotests/iotests.py | 2 ++
4 files changed, 17 insertions(+), 15 deletions(-)
diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out
index b68c443867..253209eca9 100644
--- a/tests/qemu-iotests/206.out
+++ b/tests/qemu-iotests/206.out
@@ -18,7 +18,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -42,7 +42,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -66,7 +66,7 @@ virtual size: 32 MiB (33554432 bytes)
cluster_size: 2097152
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 1
corrupt: false
@@ -92,7 +92,7 @@ backing file: TEST_IMG.base
backing file format: IMGFMT
Format specific information:
compat: 0.10
- compression type: zlib
+ compression type: COMPRESSION_TYPE
refcount bits: 16
=== Successful image creation (encrypted) ===
@@ -109,7 +109,7 @@ encrypted: yes
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
encrypt:
diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
index 3759c99284..ce231424a7 100644
--- a/tests/qemu-iotests/242.out
+++ b/tests/qemu-iotests/242.out
@@ -12,7 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -34,7 +34,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
bitmaps:
[0]:
@@ -68,7 +68,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
bitmaps:
[0]:
@@ -110,7 +110,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
bitmaps:
[0]:
@@ -161,7 +161,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
bitmaps:
[0]:
diff --git a/tests/qemu-iotests/274.out b/tests/qemu-iotests/274.out
index 1d2928e14d..1ce40d839a 100644
--- a/tests/qemu-iotests/274.out
+++ b/tests/qemu-iotests/274.out
@@ -50,7 +50,7 @@ backing file: TEST_DIR/PID-base
backing file format: IMGFMT
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -79,7 +79,7 @@ backing file: TEST_DIR/PID-base
backing file format: IMGFMT
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -114,7 +114,7 @@ backing file: TEST_DIR/PID-base
backing file format: IMGFMT
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -141,7 +141,7 @@ virtual size: 2 MiB (2097152 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -176,7 +176,7 @@ backing file: TEST_DIR/PID-base
backing file format: IMGFMT
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 857676a329..43c1e723a6 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -454,6 +454,8 @@ def filter_img_info(output, filename):
'uuid: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX',
line)
line = re.sub('cid: [0-9]+', 'cid: XXXXXXXXXX', line)
+ line = re.sub('(compression type: )(zlib|zstd)', r'\1COMPRESSION_TYPE',
+ line)
lines.append(line)
return '\n'.join(lines)
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 08/17] iotests.py: filter compression type out
2021-07-20 11:38 ` [PATCH v2 08/17] iotests.py: filter compression type out Vladimir Sementsov-Ogievskiy
@ 2021-09-13 12:00 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 12:00 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> We want iotests pass with both the default zlib compression and with
> IMGOPTS='compression_type=zstd'.
>
> Actually the only test that is interested in real compression type in
> test output is 287 (test for qcow2 compression type) and it's in bash.
> So for now we can safely filter out compression type in all qcow2
> tests.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/206.out | 10 +++++-----
> tests/qemu-iotests/242.out | 10 +++++-----
> tests/qemu-iotests/274.out | 10 +++++-----
> tests/qemu-iotests/iotests.py | 2 ++
> 4 files changed, 17 insertions(+), 15 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 09/17] iotest 302: use img_info_log() helper
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 08/17] iotests.py: filter compression type out Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 10/17] qcow2: simple case support for downgrading of qcow2 images with zstd Vladimir Sementsov-Ogievskiy
` (8 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
Instead of qemu_img_log("info", ..) use generic helper img_info_log().
img_info_log() has smarter logic. For example it use filter_img_info()
to filter output, which in turns filter a compression type. So it will
help us in future when we implement a possibility to use zstd
compression by default (with help of some runtime config file or maybe
build option). For now to test you should recompile qemu with a small
addition into block/qcow2.c before
"if (qcow2_opts->has_compression_type":
if (!qcow2_opts->has_compression_type && version >= 3) {
qcow2_opts->has_compression_type = true;
qcow2_opts->compression_type = QCOW2_COMPRESSION_TYPE_ZSTD;
}
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/302 | 4 +++-
tests/qemu-iotests/302.out | 7 +++----
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302
index 5695af4914..a6d79e727b 100755
--- a/tests/qemu-iotests/302
+++ b/tests/qemu-iotests/302
@@ -34,6 +34,7 @@ from iotests import (
qemu_img_measure,
qemu_io,
qemu_nbd_popen,
+ img_info_log,
)
iotests.script_initialize(supported_fmts=["qcow2"])
@@ -88,6 +89,7 @@ with tarfile.open(tar_file, "w") as tar:
tar_file):
iotests.log("=== Target image info ===")
+ # Not img_info_log as it enforces imgfmt, but now we print info on raw
qemu_img_log("info", nbd_uri)
qemu_img(
@@ -99,7 +101,7 @@ with tarfile.open(tar_file, "w") as tar:
nbd_uri)
iotests.log("=== Converted image info ===")
- qemu_img_log("info", nbd_uri)
+ img_info_log(nbd_uri)
iotests.log("=== Converted image check ===")
qemu_img_log("check", nbd_uri)
diff --git a/tests/qemu-iotests/302.out b/tests/qemu-iotests/302.out
index e2f6077e83..3e7c281b91 100644
--- a/tests/qemu-iotests/302.out
+++ b/tests/qemu-iotests/302.out
@@ -6,14 +6,13 @@ virtual size: 448 KiB (458752 bytes)
disk size: unavailable
=== Converted image info ===
-image: nbd+unix:///exp?socket=SOCK_DIR/PID-nbd-sock
-file format: qcow2
+image: TEST_IMG
+file format: IMGFMT
virtual size: 1 GiB (1073741824 bytes)
-disk size: unavailable
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 10/17] qcow2: simple case support for downgrading of qcow2 images with zstd
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (8 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 09/17] iotest 302: use img_info_log() helper Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 11/17] iotests/common.rc: introduce _qcow2_dump_header helper Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
If image doesn't have any compressed cluster we can easily switch to
zlib compression, which may allow to downgrade the image.
That's mostly needed to support IMGOPTS='compression_type=zstd' in some
iotests which do qcow2 downgrade.
While being here also fix checkpatch complain against '#' in printf
formatting.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 56 insertions(+), 2 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 9f1b6461c8..a841fa4c72 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -5250,6 +5250,38 @@ static int qcow2_load_vmstate(BlockDriverState *bs, QEMUIOVector *qiov,
qiov->size, qiov, 0, 0);
}
+static int qcow2_has_compressed_clusters(BlockDriverState *bs)
+{
+ int64_t offset = 0;
+ int64_t bytes = bdrv_getlength(bs);
+
+ if (bytes < 0) {
+ return bytes;
+ }
+
+ while (bytes != 0) {
+ int ret;
+ QCow2SubclusterType type;
+ unsigned int cur_bytes = MIN(INT_MAX, bytes);
+ uint64_t host_offset;
+
+ ret = qcow2_get_host_offset(bs, offset, &cur_bytes, &host_offset,
+ &type);
+ if (ret < 0) {
+ return ret;
+ }
+
+ if (type == QCOW2_SUBCLUSTER_COMPRESSED) {
+ return 1;
+ }
+
+ offset += cur_bytes;
+ bytes -= cur_bytes;
+ }
+
+ return 0;
+}
+
/*
* Downgrades an image's version. To achieve this, any incompatible features
* have to be removed.
@@ -5307,9 +5339,10 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
* the first place; if that happens nonetheless, returning -ENOTSUP is the
* best thing to do anyway */
- if (s->incompatible_features) {
+ if (s->incompatible_features & ~QCOW2_INCOMPAT_COMPRESSION) {
error_setg(errp, "Cannot downgrade an image with incompatible features "
- "%#" PRIx64 " set", s->incompatible_features);
+ "0x%" PRIx64 " set",
+ s->incompatible_features & ~QCOW2_INCOMPAT_COMPRESSION);
return -ENOTSUP;
}
@@ -5327,6 +5360,27 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,
return ret;
}
+ if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION) {
+ ret = qcow2_has_compressed_clusters(bs);
+ if (ret < 0) {
+ error_setg(errp, "Failed to check block status");
+ return -EINVAL;
+ }
+ if (ret) {
+ error_setg(errp, "Cannot downgrade an image with zstd compression "
+ "type and existing compressed clusters");
+ return -ENOTSUP;
+ }
+ /*
+ * No compressed clusters for now, so just chose default zlib
+ * compression.
+ */
+ s->incompatible_features &= ~QCOW2_INCOMPAT_COMPRESSION;
+ s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+ }
+
+ assert(s->incompatible_features == 0);
+
s->qcow_version = target_version;
ret = qcow2_update_header(bs);
if (ret < 0) {
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 11/17] iotests/common.rc: introduce _qcow2_dump_header helper
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (9 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 10/17] qcow2: simple case support for downgrading of qcow2 images with zstd Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 12/17] iotests: massive use _qcow2_dump_header Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
We'll use it in tests instead of explicit qcow2.py. Then we are going
to add some filtering in _qcow2_dump_header.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/common.rc | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 609d82de89..8ffeb8692f 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -990,5 +990,15 @@ _require_one_device_of()
_notrun "$* not available"
}
+_qcow2_dump_header()
+{
+ img="$1"
+ if [ -z "$img" ]; then
+ img="$TEST_IMG"
+ fi
+
+ $PYTHON qcow2.py "$img" dump-header
+}
+
# make sure this script returns success
true
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 12/17] iotests: massive use _qcow2_dump_header
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (10 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 11/17] iotests/common.rc: introduce _qcow2_dump_header helper Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 13/17] iotest 39: " Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
We are going to add filtering in _qcow2_dump_header and want all tests
use it.
The patch is generated by commands:
cd tests/qemu-iotests
sed -ie 's/$PYTHON qcow2.py "$TEST_IMG" dump-header\($\| \)/_qcow2_dump_header\1/' ??? tests/*
(the difficulty is to avoid converting dump-header-exts)
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/031 | 6 +++---
tests/qemu-iotests/036 | 6 +++---
tests/qemu-iotests/039 | 20 ++++++++++----------
tests/qemu-iotests/060 | 20 ++++++++++----------
tests/qemu-iotests/061 | 36 ++++++++++++++++++------------------
tests/qemu-iotests/137 | 2 +-
tests/qemu-iotests/287 | 8 ++++----
7 files changed, 49 insertions(+), 49 deletions(-)
diff --git a/tests/qemu-iotests/031 b/tests/qemu-iotests/031
index 58b57a0ef2..648112f796 100755
--- a/tests/qemu-iotests/031
+++ b/tests/qemu-iotests/031
@@ -58,21 +58,21 @@ for compat in "compat=0.10" "compat=1.1"; do
echo
_make_test_img -o $compat 64M
$PYTHON qcow2.py "$TEST_IMG" add-header-ext 0x12345678 "This is a test header extension"
- $PYTHON qcow2.py "$TEST_IMG" dump-header
+ _qcow2_dump_header
_check_test_img
echo
echo === Rewrite header with no backing file ===
echo
$QEMU_IMG rebase -u -b "" "$TEST_IMG"
- $PYTHON qcow2.py "$TEST_IMG" dump-header
+ _qcow2_dump_header
_check_test_img
echo
echo === Add a backing file and format ===
echo
$QEMU_IMG rebase -u -b "/some/backing/file/path" -F host_device "$TEST_IMG"
- $PYTHON qcow2.py "$TEST_IMG" dump-header
+ _qcow2_dump_header
done
# success, all done
diff --git a/tests/qemu-iotests/036 b/tests/qemu-iotests/036
index 5e567012a8..f703605e44 100755
--- a/tests/qemu-iotests/036
+++ b/tests/qemu-iotests/036
@@ -58,7 +58,7 @@ $PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 63
# Without feature table
$PYTHON qcow2.py "$TEST_IMG" del-header-ext 0x6803f857
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep features
+_qcow2_dump_header | grep features
$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
_img_info
@@ -107,7 +107,7 @@ echo === Create image with unknown autoclear feature bit ===
echo
_make_test_img 64M
$PYTHON qcow2.py "$TEST_IMG" set-feature-bit autoclear 63
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep features
+_qcow2_dump_header | grep features
$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
echo
@@ -115,7 +115,7 @@ echo === Repair image ===
echo
_check_test_img -r all
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep features
+_qcow2_dump_header | grep features
$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
# success, all done
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index 12b2c7fa7b..8e783a8380 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -59,7 +59,7 @@ _make_test_img -o "compat=1.1,lazy_refcounts=on" $size
$QEMU_IO -c "write -P 0x5a 0 512" "$TEST_IMG" | _filter_qemu_io
# The dirty bit must not be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img
echo
@@ -73,7 +73,7 @@ $QEMU_IO -c "write -P 0x5a 0 512" \
| _filter_qemu_io
# The dirty bit must be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img
echo
@@ -82,7 +82,7 @@ echo "== Read-only access must still work =="
$QEMU_IO -r -c "read -P 0x5a 0 512" "$TEST_IMG" | _filter_qemu_io
# The dirty bit must be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "== Repairing the image file must succeed =="
@@ -90,7 +90,7 @@ echo "== Repairing the image file must succeed =="
_check_test_img -r all
# The dirty bit must not be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "== Data should still be accessible after repair =="
@@ -108,12 +108,12 @@ $QEMU_IO -c "write -P 0x5a 0 512" \
| _filter_qemu_io
# The dirty bit must be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
$QEMU_IO -c "write 0 512" "$TEST_IMG" | _filter_qemu_io
# The dirty bit must not be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "== Creating an image file with lazy_refcounts=off =="
@@ -126,7 +126,7 @@ $QEMU_IO -c "write -P 0x5a 0 512" \
| _filter_qemu_io
# The dirty bit must not be set since lazy_refcounts=off
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img
echo
@@ -141,7 +141,7 @@ $QEMU_IO -c "write 0 512" "$TEST_IMG" | _filter_qemu_io
$QEMU_IMG commit "$TEST_IMG"
# The dirty bit must not be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
$PYTHON qcow2.py "$TEST_IMG".base dump-header | grep incompatible_features
_check_test_img
@@ -159,7 +159,7 @@ $QEMU_IO -c "reopen -o lazy-refcounts=on" \
| _filter_qemu_io
# The dirty bit must be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img
_make_test_img -o "compat=1.1,lazy_refcounts=on" $size
@@ -171,7 +171,7 @@ $QEMU_IO -c "reopen -o lazy-refcounts=off" \
| _filter_qemu_io
# The dirty bit must not be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img
diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060
index db26c6b246..d1e3204d4e 100755
--- a/tests/qemu-iotests/060
+++ b/tests/qemu-iotests/060
@@ -80,13 +80,13 @@ poke_file "$TEST_IMG" "$l1_offset" "\x80\x00\x00\x00\x00\x03\x00\x00"
_check_test_img
# The corrupt bit should not be set anyway
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
# Try to write something, thereby forcing the corrupt bit to be set
$QEMU_IO -c "$OPEN_RW" -c "write -P 0x2a 0 512" | _filter_qemu_io
# The corrupt bit must now be set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
# This information should be available through qemu-img info
_img_info --format-specific
@@ -114,19 +114,19 @@ poke_file "$TEST_IMG" "$(($rb_offset+8))" "\x00\x01"
# Redirect new data cluster onto refcount block
poke_file "$TEST_IMG" "$l2_offset" "\x80\x00\x00\x00\x00\x02\x00\x00"
_check_test_img
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
$QEMU_IO -c "$OPEN_RW" -c "write -P 0x2a 0 512" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
# Try to fix it
_check_test_img -r all
# The corrupt bit should be cleared
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
# Look if it's really really fixed
$QEMU_IO -c "$OPEN_RW" -c "write -P 0x2a 0 512" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "=== Testing cluster data reference into inactive L2 table ==="
@@ -139,13 +139,13 @@ $QEMU_IO -c "$OPEN_RW" -c "write -P 2 0 512" | _filter_qemu_io
poke_file "$TEST_IMG" "$l2_offset_after_snapshot" \
"\x80\x00\x00\x00\x00\x04\x00\x00"
_check_test_img
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
$QEMU_IO -c "$OPEN_RW" -c "write -P 3 0 512" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
_check_test_img -r all
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
$QEMU_IO -c "$OPEN_RW" -c "write -P 4 0 512" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
# Check data
$QEMU_IO -c "$OPEN_RO" -c "read -P 4 0 512" | _filter_qemu_io
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index 9507c223bd..70edf1a163 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -55,9 +55,9 @@ echo "=== Testing version downgrade with zero expansion ==="
echo
_make_test_img -o "compat=1.1,lazy_refcounts=on" 64M
$QEMU_IO -c "write -z 0 128k" "$TEST_IMG" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IO -c "read -P 0 0 128k" "$TEST_IMG" | _filter_qemu_io
_check_test_img
@@ -68,10 +68,10 @@ _make_test_img -o "compat=1.1,lazy_refcounts=on" 64M
$QEMU_IO -c "write -z 0 128k" "$TEST_IMG" | _filter_qemu_io
$QEMU_IO -c "write -z 32M 128k" "$TEST_IMG" | _filter_qemu_io
$QEMU_IO -c map "$TEST_IMG" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "compat=0.10" --image-opts \
driver=qcow2,file.filename=$TEST_IMG,l2-cache-entry-size=4096
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IO -c "read -P 0 0 128k" "$TEST_IMG" | _filter_qemu_io
$QEMU_IO -c "read -P 0 32M 128k" "$TEST_IMG" | _filter_qemu_io
$QEMU_IO -c map "$TEST_IMG" | _filter_qemu_io
@@ -84,9 +84,9 @@ _make_test_img -o "compat=1.1,lazy_refcounts=on" 64M
_NO_VALGRIND \
$QEMU_IO -c "write -P 0x2a 0 128k" -c flush \
-c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IO -c "read -P 0x2a 0 128k" "$TEST_IMG" | _filter_qemu_io
_check_test_img
@@ -96,9 +96,9 @@ echo
_make_test_img -o "compat=1.1" 64M
$PYTHON qcow2.py "$TEST_IMG" set-feature-bit compatible 42
$PYTHON qcow2.py "$TEST_IMG" set-feature-bit autoclear 42
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG"
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
_check_test_img
echo
@@ -106,9 +106,9 @@ echo "=== Testing version upgrade and resize ==="
echo
_make_test_img -o "compat=0.10" 64M
$QEMU_IO -c "write -P 0x2a 42M 64k" "$TEST_IMG" | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "compat=1.1,lazy_refcounts=on,size=128M" "$TEST_IMG"
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IO -c "read -P 0x2a 42M 64k" "$TEST_IMG" | _filter_qemu_io
_check_test_img
@@ -120,29 +120,29 @@ $QEMU_IO -c "write -P 0x2a 24M 64k" "$TEST_IMG" | _filter_qemu_io
$QEMU_IMG snapshot -c foo "$TEST_IMG"
$QEMU_IMG resize "$TEST_IMG" 64M &&
echo "unexpected pass"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
$QEMU_IMG amend -o "compat=1.1,size=128M" "$TEST_IMG" ||
echo "unexpected fail"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
$QEMU_IMG snapshot -c bar "$TEST_IMG"
$QEMU_IMG resize --shrink "$TEST_IMG" 64M ||
echo "unexpected fail"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" &&
echo "unexpected pass"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
$QEMU_IMG snapshot -a bar "$TEST_IMG" ||
echo "unexpected fail"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
$QEMU_IMG snapshot -d bar "$TEST_IMG"
$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" ||
echo "unexpected fail"
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)'
+_qcow2_dump_header | grep '^\(version\|size\|nb_snap\)'
_check_test_img
@@ -154,9 +154,9 @@ _make_test_img -o "compat=1.1,lazy_refcounts=on" 64M
_NO_VALGRIND \
$QEMU_IO -c "write -P 0x2a 0 128k" -c flush \
-c "sigraise $(kill -l KILL)" "$TEST_IMG" 2>&1 | _filter_qemu_io
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IMG amend -o "lazy_refcounts=off" "$TEST_IMG"
-$PYTHON qcow2.py "$TEST_IMG" dump-header
+_qcow2_dump_header
$QEMU_IO -c "read -P 0x2a 0 128k" "$TEST_IMG" | _filter_qemu_io
_check_test_img
diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137
index 4680d5df3d..52ee135184 100755
--- a/tests/qemu-iotests/137
+++ b/tests/qemu-iotests/137
@@ -140,7 +140,7 @@ $QEMU_IO \
# The dirty bit must not be set
# (Filter the external data file bit)
-if $PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features \
+if _qcow2_dump_header | grep incompatible_features \
| grep -q '\<0\>'
then
echo 'ERROR: Dirty bit set'
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
index 22ce9ff0e4..6716419da4 100755
--- a/tests/qemu-iotests/287
+++ b/tests/qemu-iotests/287
@@ -61,13 +61,13 @@ echo
echo "=== Testing compression type incompatible bit setting for zlib ==="
echo
_make_test_img -o compression_type=zlib 64M
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "=== Testing compression type incompatible bit setting for zstd ==="
echo
_make_test_img -o compression_type=zstd 64M
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
echo
echo "=== Testing zlib with incompatible bit set ==="
@@ -75,7 +75,7 @@ echo
_make_test_img -o compression_type=zlib 64M
$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
# to make sure the bit was actually set
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
echo "Error: The image opened successfully. The image must not be opened."
@@ -87,7 +87,7 @@ echo
_make_test_img -o compression_type=zstd 64M
$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
# to make sure the bit was actually unset
-$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+_qcow2_dump_header | grep incompatible_features
if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
echo "Error: The image opened successfully. The image must not be opened."
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 13/17] iotest 39: use _qcow2_dump_header
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (11 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 12/17] iotests: massive use _qcow2_dump_header Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 14/17] iotests: bash tests: filter compression type Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
_qcow2_dump_header has filter for compression type, so this change
makes test pass with IMGOPTS='compression_type=zstd'.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/039 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index 8e783a8380..00d379cde2 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -142,7 +142,7 @@ $QEMU_IMG commit "$TEST_IMG"
# The dirty bit must not be set
_qcow2_dump_header | grep incompatible_features
-$PYTHON qcow2.py "$TEST_IMG".base dump-header | grep incompatible_features
+_qcow2_dump_header "$TEST_IMG".base | grep incompatible_features
_check_test_img
TEST_IMG="$TEST_IMG".base _check_test_img
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 14/17] iotests: bash tests: filter compression type
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (12 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 13/17] iotest 39: " Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 12:25 ` Hanna Reitz
2021-07-20 11:38 ` [PATCH v2 15/17] iotests 60: more accurate set dirty bit in qcow2 header Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
We want iotests pass with both the default zlib compression and with
IMGOPTS='compression_type=zstd'.
Actually the only test that is interested in real compression type in
test output is 287 (test for qcow2 compression type), so implement
specific option for it.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/060.out | 2 +-
tests/qemu-iotests/061.out | 12 ++++++------
tests/qemu-iotests/082.out | 14 +++++++-------
tests/qemu-iotests/198.out | 4 ++--
tests/qemu-iotests/287 | 8 ++++----
tests/qemu-iotests/common.filter | 8 ++++++++
tests/qemu-iotests/common.rc | 14 +++++++++++++-
7 files changed, 41 insertions(+), 21 deletions(-)
diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index b74540bafb..329977d9b9 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -17,7 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: true
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 7ecbd4dea8..139fc68177 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -525,7 +525,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -552,7 +552,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file: foo
@@ -567,7 +567,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file raw: false
@@ -583,7 +583,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -597,7 +597,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -612,7 +612,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index 077ed0f2c7..d0dd333117 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -17,7 +17,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 4096
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
@@ -31,7 +31,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 8192
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
@@ -329,7 +329,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 4096
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
@@ -342,7 +342,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 8192
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
@@ -639,7 +639,7 @@ virtual size: 128 MiB (134217728 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
@@ -652,7 +652,7 @@ virtual size: 130 MiB (136314880 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -665,7 +665,7 @@ virtual size: 132 MiB (138412032 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
- compression type: zlib
+ compression type: COMPRESSION_TYPE
lazy refcounts: true
refcount bits: 16
corrupt: false
diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out
index 3952708444..805494916f 100644
--- a/tests/qemu-iotests/198.out
+++ b/tests/qemu-iotests/198.out
@@ -36,7 +36,7 @@ image: json:{ /* filtered */ }
file format: IMGFMT
virtual size: 16 MiB (16777216 bytes)
Format specific information:
- compression type: zlib
+ compression type: COMPRESSION_TYPE
encrypt:
ivgen alg: plain64
hash alg: sha256
@@ -81,7 +81,7 @@ virtual size: 16 MiB (16777216 bytes)
backing file: TEST_DIR/t.IMGFMT.base
backing file format: IMGFMT
Format specific information:
- compression type: zlib
+ compression type: COMPRESSION_TYPE
encrypt:
ivgen alg: plain64
hash alg: sha256
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
index 6716419da4..aab03fb973 100755
--- a/tests/qemu-iotests/287
+++ b/tests/qemu-iotests/287
@@ -61,13 +61,13 @@ echo
echo "=== Testing compression type incompatible bit setting for zlib ==="
echo
_make_test_img -o compression_type=zlib 64M
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
echo
echo "=== Testing compression type incompatible bit setting for zstd ==="
echo
_make_test_img -o compression_type=zstd 64M
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
echo
echo "=== Testing zlib with incompatible bit set ==="
@@ -75,7 +75,7 @@ echo
_make_test_img -o compression_type=zlib 64M
$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
# to make sure the bit was actually set
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
echo "Error: The image opened successfully. The image must not be opened."
@@ -87,7 +87,7 @@ echo
_make_test_img -o compression_type=zstd 64M
$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
# to make sure the bit was actually unset
-_qcow2_dump_header | grep incompatible_features
+_qcow2_dump_header --no-filter-compression | grep incompatible_features
if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
echo "Error: The image opened successfully. The image must not be opened."
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 268b749e2f..447fec74df 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -247,6 +247,7 @@ _filter_img_info()
-e "/block_state_zero: \\(on\\|off\\)/d" \
-e "/log_size: [0-9]\\+/d" \
-e "s/iters: [0-9]\\+/iters: 1024/" \
+ -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
-e "s/uuid: [-a-f0-9]\\+/uuid: 00000000-0000-0000-0000-000000000000/" | \
while IFS='' read -r line; do
if [[ $format_specific == 1 ]]; then
@@ -332,5 +333,12 @@ for fname in fnames:
sys.stdout.write(result)'
}
+_filter_qcow2_compression_type_bit()
+{
+ $SED -e 's/\(incompatible_features\s\+\)\[3\(, \)\?/\1[/' \
+ -e 's/\(incompatible_features.*\), 3\]/\1]/' \
+ -e 's/\(incompatible_features.*\), 3\(,.*\)/\1\2/'
+}
+
# make sure this script returns success
true
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 8ffeb8692f..2a0375bac9 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -693,6 +693,7 @@ _img_info()
-e "s#$TEST_DIR#TEST_DIR#g" \
-e "s#$SOCK_DIR/fuse-#TEST_DIR/#g" \
-e "s#$IMGFMT#IMGFMT#g" \
+ -e 's/\(compression type: \)\(zlib\|zstd\)/\1COMPRESSION_TYPE/' \
-e "/^disk size:/ D" \
-e "/actual-size/ D" | \
while IFS='' read -r line; do
@@ -992,12 +993,23 @@ _require_one_device_of()
_qcow2_dump_header()
{
+ if [[ "$1" == "--no-filter-compression" ]]; then
+ local filter_compression=0
+ shift
+ else
+ local filter_compression=1
+ fi
+
img="$1"
if [ -z "$img" ]; then
img="$TEST_IMG"
fi
- $PYTHON qcow2.py "$img" dump-header
+ if [[ $filter_compression == 0 ]]; then
+ $PYTHON qcow2.py "$img" dump-header
+ else
+ $PYTHON qcow2.py "$img" dump-header | _filter_qcow2_compression_type_bit
+ fi
}
# make sure this script returns success
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 14/17] iotests: bash tests: filter compression type
2021-07-20 11:38 ` [PATCH v2 14/17] iotests: bash tests: filter compression type Vladimir Sementsov-Ogievskiy
@ 2021-09-13 12:25 ` Hanna Reitz
0 siblings, 0 replies; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 12:25 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> We want iotests pass with both the default zlib compression and with
> IMGOPTS='compression_type=zstd'.
>
> Actually the only test that is interested in real compression type in
> test output is 287 (test for qcow2 compression type), so implement
> specific option for it.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/060.out | 2 +-
> tests/qemu-iotests/061.out | 12 ++++++------
> tests/qemu-iotests/082.out | 14 +++++++-------
> tests/qemu-iotests/198.out | 4 ++--
> tests/qemu-iotests/287 | 8 ++++----
> tests/qemu-iotests/common.filter | 8 ++++++++
> tests/qemu-iotests/common.rc | 14 +++++++++++++-
> 7 files changed, 41 insertions(+), 21 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v2 15/17] iotests 60: more accurate set dirty bit in qcow2 header
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (13 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 14/17] iotests: bash tests: filter compression type Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 16/17] iotest 214: explicit compression type Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
Don't touch other incompatible bits, like compression-type. This makes
the test pass with IMGOPTS='compression_type=zstd'.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/060 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060
index d1e3204d4e..df87d600f7 100755
--- a/tests/qemu-iotests/060
+++ b/tests/qemu-iotests/060
@@ -326,7 +326,7 @@ _make_test_img 64M
# Let the refblock appear unaligned
poke_file "$TEST_IMG" "$rt_offset" "\x00\x00\x00\x00\xff\xff\x2a\x00"
# Mark the image dirty, thus forcing an automatic check when opening it
-poke_file "$TEST_IMG" 72 "\x00\x00\x00\x00\x00\x00\x00\x01"
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 0
# Open the image (qemu should refuse to do so)
$QEMU_IO -c close "$TEST_IMG" 2>&1 | _filter_testdir | _filter_imgfmt
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 16/17] iotest 214: explicit compression type
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (14 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 15/17] iotests 60: more accurate set dirty bit in qcow2 header Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-07-20 11:38 ` [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS Vladimir Sementsov-Ogievskiy
2021-09-01 13:14 ` [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
The test-case "Corrupted size field in compressed cluster descriptor"
heavily depends on zlib compression type. So, make it explicit. This
way test passes with IMGOPTS='compression_type=zstd'.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/214 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214
index 0889089d81..c66e246ba2 100755
--- a/tests/qemu-iotests/214
+++ b/tests/qemu-iotests/214
@@ -51,7 +51,7 @@ echo
# The L2 entries of the two compressed clusters are located at
# 0x800000 and 0x800008, their original values are 0x4008000000a00000
# and 0x4008000000a00802 (5 sectors for compressed data each).
-_make_test_img 8M -o cluster_size=2M
+_make_test_img 8M -o cluster_size=2M,compression_type=zlib
$QEMU_IO -c "write -c -P 0x11 0 2M" -c "write -c -P 0x11 2M 2M" "$TEST_IMG" \
2>&1 | _filter_qemu_io | _filter_testdir
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (15 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 16/17] iotest 214: explicit compression type Vladimir Sementsov-Ogievskiy
@ 2021-07-20 11:38 ` Vladimir Sementsov-Ogievskiy
2021-09-13 12:43 ` Hanna Reitz
2021-09-01 13:14 ` [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
17 siblings, 1 reply; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-07-20 11:38 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow, vsementsov
compression_type can't be used if we want to create image with
compat=0.10. So, skip these tests, not many of them.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/031 | 5 +++--
tests/qemu-iotests/051 | 5 +++--
tests/qemu-iotests/061 | 6 +++++-
tests/qemu-iotests/112 | 3 ++-
tests/qemu-iotests/290 | 2 +-
5 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/tests/qemu-iotests/031 b/tests/qemu-iotests/031
index 648112f796..ee587b1606 100755
--- a/tests/qemu-iotests/031
+++ b/tests/qemu-iotests/031
@@ -42,8 +42,9 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
_supported_fmt qcow2
_supported_proto file fuse
# We want to test compat=0.10, which does not support external data
-# files or refcount widths other than 16
-_unsupported_imgopts data_file 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
+# files or refcount widths other than 16 or compression type
+_unsupported_imgopts data_file compression_type \
+ 'refcount_bits=\([^1]\|.\([^6]\|$\)\)'
CLUSTER_SIZE=65536
diff --git a/tests/qemu-iotests/051 b/tests/qemu-iotests/051
index 7bf29343d7..4832000fd5 100755
--- a/tests/qemu-iotests/051
+++ b/tests/qemu-iotests/051
@@ -41,8 +41,9 @@ _supported_fmt qcow2
_supported_proto file
# A compat=0.10 image is created in this test which does not support anything
# other than refcount_bits=16;
-# it also will not support an external data file
-_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file
+# it also will not support an external data file and compression type
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file \
+ compression_type
_require_drivers nbd
do_run_qemu()
diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061
index 70edf1a163..513fbec14c 100755
--- a/tests/qemu-iotests/061
+++ b/tests/qemu-iotests/061
@@ -48,7 +48,11 @@ _supported_os Linux
# not work with it;
# we have explicit tests for various cluster sizes, the remaining tests
# require the default 64k cluster
-_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file cluster_size
+# we don't have explicit tests for zstd qcow2 compression type, as zstd may be
+# not compiled in. And we can't create compat images with comression type
+# extension
+_unsupported_imgopts 'refcount_bits=\([^1]\|.\([^6]\|$\)\)' data_file \
+ cluster_size compression_type
echo
echo "=== Testing version downgrade with zero expansion ==="
diff --git a/tests/qemu-iotests/112 b/tests/qemu-iotests/112
index 07ac74fb2c..5333212993 100755
--- a/tests/qemu-iotests/112
+++ b/tests/qemu-iotests/112
@@ -43,7 +43,8 @@ _supported_proto file fuse
# This test will set refcount_bits on its own which would conflict with the
# manual setting; compat will be overridden as well;
# and external data files do not work well with our refcount testing
-_unsupported_imgopts refcount_bits 'compat=0.10' data_file
+# also, compression type is not supported with compat=0.10 used in test
+_unsupported_imgopts refcount_bits 'compat=0.10' data_file compression_type
print_refcount_bits()
{
diff --git a/tests/qemu-iotests/290 b/tests/qemu-iotests/290
index ed80da2685..776b59de1b 100755
--- a/tests/qemu-iotests/290
+++ b/tests/qemu-iotests/290
@@ -41,7 +41,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
_supported_fmt qcow2
_supported_proto file fuse
_supported_os Linux
-_unsupported_imgopts 'compat=0.10' refcount_bits data_file
+_unsupported_imgopts 'compat=0.10' refcount_bits data_file compression_type
echo
echo "### Test 'qemu-io -c discard' on a QCOW2 image without a backing file"
--
2.29.2
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS
2021-07-20 11:38 ` [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS Vladimir Sementsov-Ogievskiy
@ 2021-09-13 12:43 ` Hanna Reitz
2021-09-13 15:55 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 31+ messages in thread
From: Hanna Reitz @ 2021-09-13 12:43 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: kwolf, den, jsnow, qemu-devel, mreitz
On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
> compression_type can't be used if we want to create image with
> compat=0.10. So, skip these tests, not many of them.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/031 | 5 +++--
> tests/qemu-iotests/051 | 5 +++--
> tests/qemu-iotests/061 | 6 +++++-
> tests/qemu-iotests/112 | 3 ++-
> tests/qemu-iotests/290 | 2 +-
> 5 files changed, 14 insertions(+), 7 deletions(-)
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS
2021-09-13 12:43 ` Hanna Reitz
@ 2021-09-13 15:55 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-09-13 15:55 UTC (permalink / raw)
To: Hanna Reitz, qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow
13.09.2021 15:43, Hanna Reitz wrote:
> On 20.07.21 13:38, Vladimir Sementsov-Ogievskiy wrote:
>> compression_type can't be used if we want to create image with
>> compat=0.10. So, skip these tests, not many of them.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/031 | 5 +++--
>> tests/qemu-iotests/051 | 5 +++--
>> tests/qemu-iotests/061 | 6 +++++-
>> tests/qemu-iotests/112 | 3 ++-
>> tests/qemu-iotests/290 | 2 +-
>> 5 files changed, 14 insertions(+), 7 deletions(-)
>
> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
>
Thanks a lot for reviewing! I'll resend tomorrow.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v2 00/17] iotests: support zstd
2021-07-20 11:38 [PATCH v2 00/17] iotests: support zstd Vladimir Sementsov-Ogievskiy
` (16 preceding siblings ...)
2021-07-20 11:38 ` [PATCH v2 17/17] iotests: declare lack of support for compresion_type in IMGOPTS Vladimir Sementsov-Ogievskiy
@ 2021-09-01 13:14 ` Vladimir Sementsov-Ogievskiy
17 siblings, 0 replies; 31+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-09-01 13:14 UTC (permalink / raw)
To: qemu-block; +Cc: qemu-devel, mreitz, kwolf, den, jsnow
Ping.
This applies to current master with a small obvious conflict in tests/qemu-iotests/common.filter
20.07.2021 14:38, Vladimir Sementsov-Ogievskiy wrote:
> These series makes tests pass with
>
> IMGOPTS='compression_type=zstd'
>
> Also, python iotests start to support IMGOPTS (they didn't before).
>
> v2:
> 01: add Max's r-b
> 02: simplify a lot: just add separate -o for IMGOPTS.
> also, don't bother with catching compat=0.10
> 03,04: new
> 05: add Max's r-b
> 06: one more explict compression_type=zlib
> 07: new
> 08: reduced to only update filter_img_info
> 09: adjust commit message, add comment, add Max's r-b
> 10: s/ = 0/ &= ~QCOW2_INCOMPAT_COMPRESSION/
> add Max's r-b
> 11-13: add Max's r-b
> 14: one more regex for "2, 3, 4" case
> 15-16: add Max's r-b
> 17: new
>
> Vladimir Sementsov-Ogievskiy (17):
> iotests.py: img_info_log(): rename imgopts argument
> iotests.py: qemu_img*("create"): support
> IMGOPTS='compression_type=zstd'
> iotests: drop qemu_img_verbose() helper
> iotests.py: rewrite default luks support in qemu_img
> iotest 303: explicit compression type
> iotest 065: explicit compression type
> iotests.py: filter out successful output of qemu-img crate
> iotests.py: filter compression type out
> iotest 302: use img_info_log() helper
> qcow2: simple case support for downgrading of qcow2 images with zstd
> iotests/common.rc: introduce _qcow2_dump_header helper
> iotests: massive use _qcow2_dump_header
> iotest 39: use _qcow2_dump_header
> iotests: bash tests: filter compression type
> iotests 60: more accurate set dirty bit in qcow2 header
> iotest 214: explicit compression type
> iotests: declare lack of support for compresion_type in IMGOPTS
>
> block/qcow2.c | 58 +++++++++++++++++++++-
> tests/qemu-iotests/031 | 11 +++--
> tests/qemu-iotests/036 | 6 +--
> tests/qemu-iotests/039 | 22 ++++-----
> tests/qemu-iotests/044 | 5 +-
> tests/qemu-iotests/044.out | 1 +
> tests/qemu-iotests/051 | 5 +-
> tests/qemu-iotests/060 | 22 ++++-----
> tests/qemu-iotests/060.out | 2 +-
> tests/qemu-iotests/061 | 42 ++++++++--------
> tests/qemu-iotests/061.out | 12 ++---
> tests/qemu-iotests/065 | 16 +++---
> tests/qemu-iotests/082.out | 14 +++---
> tests/qemu-iotests/112 | 3 +-
> tests/qemu-iotests/137 | 2 +-
> tests/qemu-iotests/198.out | 4 +-
> tests/qemu-iotests/206.out | 10 ++--
> tests/qemu-iotests/209 | 7 +--
> tests/qemu-iotests/209.out | 2 +
> tests/qemu-iotests/210 | 8 +--
> tests/qemu-iotests/214 | 2 +-
> tests/qemu-iotests/242.out | 10 ++--
> tests/qemu-iotests/255.out | 4 --
> tests/qemu-iotests/274.out | 39 ++-------------
> tests/qemu-iotests/280.out | 1 -
> tests/qemu-iotests/287 | 8 +--
> tests/qemu-iotests/290 | 2 +-
> tests/qemu-iotests/302 | 4 +-
> tests/qemu-iotests/302.out | 7 ++-
> tests/qemu-iotests/303 | 25 ++++++----
> tests/qemu-iotests/303.out | 30 +++++++++++-
> tests/qemu-iotests/common.filter | 8 +++
> tests/qemu-iotests/common.rc | 22 +++++++++
> tests/qemu-iotests/iotests.py | 84 ++++++++++++++++++++------------
> 34 files changed, 310 insertions(+), 188 deletions(-)
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 31+ messages in thread