[PATCH v3 0/3] iotests: Check for zstd support

[PATCH v3 0/3] iotests: Check for zstd support

[Hanna Reitz]

Hi,

v1 cover letter:
https://lists.nongnu.org/archive/html/qemu-devel/2022-02/msg04592.html

v2 cover letter:
https://lists.nongnu.org/archive/html/qemu-block/2022-03/msg00041.html

Thanks a lot for the review and testing on v2; but I missed that I’d
have to send v3 because qemu_img_pipe_and_status() has just been
removed, which was used in patch 1 of v2. :/


v3:
- Patch 1: Use qemu_img(check=False) instead of
           qemu_img_pipe_and_status()
- Patch 2: Rebase-caused import conflict


git backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/3:[0011] [FC] 'iotests.py: Add supports_qcow2_zstd_compression()'
002/3:[0004] [FC] 'iotests/065: Check for zstd support'
003/3:[----] [--] 'iotests/303: Check for zstd support'


Hanna Reitz (3):
  iotests.py: Add supports_qcow2_zstd_compression()
  iotests/065: Check for zstd support
  iotests/303: Check for zstd support

 tests/qemu-iotests/065        | 24 ++++++++++++++++++------
 tests/qemu-iotests/303        |  4 +++-
 tests/qemu-iotests/iotests.py | 20 ++++++++++++++++++++
 3 files changed, 41 insertions(+), 7 deletions(-)

-- 
2.35.1

[PATCH v3 1/3] iotests.py: Add supports_qcow2_zstd_compression()

[Hanna Reitz]

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fcec3e51e5..fe10a6cf05 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -1471,6 +1471,26 @@ def verify_working_luks():
     if not working:
         notrun(reason)
 
+def supports_qcow2_zstd_compression() -> bool:
+    img_file = f'{test_dir}/qcow2-zstd-test.qcow2'
+    res = qemu_img('create', '-f', 'qcow2', '-o', 'compression_type=zstd',
+                   img_file, '0',
+                   check=False)
+    try:
+        os.remove(img_file)
+    except OSError:
+        pass
+
+    if res.returncode == 1 and \
+            "'compression-type' does not accept value 'zstd'" in res.stdout:
+        return False
+    else:
+        return True
+
+def verify_qcow2_zstd_compression():
+    if not supports_qcow2_zstd_compression():
+        notrun('zstd compression not supported')
+
 def qemu_pipe(*args: str) -> str:
     """
     Run qemu with an option to print something and exit (e.g. a help option).
-- 
2.35.1

[PATCH v3 2/3] iotests/065: Check for zstd support

[Hanna Reitz]

Some test cases run in iotest 065 want to run with zstd compression just
for added coverage.  Run them with zlib if there is no zstd support
compiled in.

Reported-by: Thomas Huth <thuth@redhat.com>
Fixes: 12a936171d71f839dc907ff ("iotest 065: explicit compression type")
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/065 | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index ba94e19349..b724c89c7c 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -24,7 +24,7 @@ import os
 import re
 import json
 import iotests
-from iotests import qemu_img, qemu_img_info
+from iotests import qemu_img, qemu_img_info, supports_qcow2_zstd_compression
 import unittest
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -95,11 +95,17 @@ class TestQCow2(TestQemuImgInfo):
 
 class TestQCow3NotLazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
-    img_options = 'compat=1.1,lazy_refcounts=off,compression_type=zstd'
+    if supports_qcow2_zstd_compression():
+        compression_type = 'zstd'
+    else:
+        compression_type = 'zlib'
+
+    img_options = 'compat=1.1,lazy_refcounts=off'
+    img_options += f',compression_type={compression_type}'
     json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
                      'refcount-bits': 16, 'corrupt': False,
-                     'compression-type': 'zstd', 'extended-l2': False }
-    human_compare = [ 'compat: 1.1', 'compression type: zstd',
+                     'compression-type': compression_type, 'extended-l2': False }
+    human_compare = [ 'compat: 1.1', f'compression type: {compression_type}',
                       'lazy refcounts: false', 'refcount bits: 16',
                       'corrupt: false', 'extended l2: false' ]
 
@@ -126,11 +132,17 @@ class TestQCow3NotLazyQMP(TestQMP):
 class TestQCow3LazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening
        with lazy refcounts disabled'''
-    img_options = 'compat=1.1,lazy_refcounts=on,compression_type=zstd'
+    if supports_qcow2_zstd_compression():
+        compression_type = 'zstd'
+    else:
+        compression_type = 'zlib'
+
+    img_options = 'compat=1.1,lazy_refcounts=on'
+    img_options += f',compression_type={compression_type}'
     qemu_options = 'lazy-refcounts=off'
     compare = { 'compat': '1.1', 'lazy-refcounts': True,
                 'refcount-bits': 16, 'corrupt': False,
-                'compression-type': 'zstd', 'extended-l2': False }
+                'compression-type': compression_type, 'extended-l2': False }
 
 TestImageInfoSpecific = None
 TestQemuImgInfo = None
-- 
2.35.1

[PATCH v3 3/3] iotests/303: Check for zstd support

[Hanna Reitz]

303 runs two test cases, one of which requires zstd support.
Unfortunately, given that this is not a unittest-style test, we cannot
easily skip that single case, and instead can only skip the whole test.

(Alternatively, we could split this test into a zlib and a zstd part,
but that seems excessive, given that this test is not in auto and thus
likely only run by developers who have zstd support compiled in.)

Fixes: 677e0bae686e7c670a71d1f ("iotest 303: explicit compression type")
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>
---
 tests/qemu-iotests/303 | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303
index 93aa5ce9b7..40e947f26c 100755
--- a/tests/qemu-iotests/303
+++ b/tests/qemu-iotests/303
@@ -21,10 +21,12 @@
 
 import iotests
 import subprocess
-from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io
+from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io, \
+        verify_qcow2_zstd_compression
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           unsupported_imgopts=['refcount_bits', 'compat'])
+verify_qcow2_zstd_compression()
 
 disk = file_path('disk')
 chunk = 1024 * 1024
-- 
2.35.1

Re: [PATCH v3 1/3] iotests.py: Add supports_qcow2_zstd_compression()

[Vladimir Sementsov-Ogievskiy]

23.03.2022 13:55, Hanna Reitz wrote:
> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Signed-off-by: Hanna Reitz <hreitz@redhat.com>
> ---
>   tests/qemu-iotests/iotests.py | 20 ++++++++++++++++++++
>   1 file changed, 20 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index fcec3e51e5..fe10a6cf05 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -1471,6 +1471,26 @@ def verify_working_luks():
>       if not working:
>           notrun(reason)
>   
> +def supports_qcow2_zstd_compression() -> bool:
> +    img_file = f'{test_dir}/qcow2-zstd-test.qcow2'
> +    res = qemu_img('create', '-f', 'qcow2', '-o', 'compression_type=zstd',
> +                   img_file, '0',
> +                   check=False)

check=False may be squashed into previous line

> +    try:
> +        os.remove(img_file)
> +    except OSError:
> +        pass
> +
> +    if res.returncode == 1 and \
> +            "'compression-type' does not accept value 'zstd'" in res.stdout:
> +        return False
> +    else:
> +        return True

May be:

return not(res.returncode == 1 and
    "'compression-type' does not accept value 'zstd'" in res.stdout)

> +
> +def verify_qcow2_zstd_compression():
> +    if not supports_qcow2_zstd_compression():
> +        notrun('zstd compression not supported')
> +
>   def qemu_pipe(*args: str) -> str:
>       """
>       Run qemu with an option to print something and exit (e.g. a help option).

anyway:
Reviewed-by: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>

-- 
Best regards,
Vladimir

Re: [PATCH v3 2/3] iotests/065: Check for zstd support

[Vladimir Sementsov-Ogievskiy]

23.03.2022 13:55, Hanna Reitz wrote:
> Some test cases run in iotest 065 want to run with zstd compression just
> for added coverage.  Run them with zlib if there is no zstd support
> compiled in.
> 
> Reported-by: Thomas Huth<thuth@redhat.com>
> Fixes: 12a936171d71f839dc907ff ("iotest 065: explicit compression type")
> Signed-off-by: Hanna Reitz<hreitz@redhat.com>

Reviewed-by: Vladimir Sementsov-Ogievskiy <v.sementsov-og@mail.ru>

-- 
Best regards,
Vladimir