* [PULL 00/15] bitmaps patches for 2020-06-09
@ 2020-06-09 20:52 Eric Blake
2020-06-09 20:52 ` [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand Eric Blake
` (15 more replies)
0 siblings, 16 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
The following changes since commit 31d321c2b3574dcc74e9f6411af06bca6b5d10f4:
Merge remote-tracking branch 'remotes/philmd-gitlab/tags/sparc-next-20200609' into staging (2020-06-09 17:29:47 +0100)
are available in the Git repository at:
https://repo.or.cz/qemu/ericb.git tags/pull-bitmaps-2020-06-09
for you to fetch changes up to adf92f4645ba46726368735916fed763d3e5a09b:
iotests: Fix 291 across more file systems (2020-06-09 15:48:00 -0500)
----------------------------------------------------------------
bitmaps patches for 2020-06-09
- documenation fix
- various improvements to qcow2.py program used in iotests
----------------------------------------------------------------
Eric Blake (2):
qemu-img: Fix doc typo for 'bitmap' subcommand
iotests: Fix 291 across more file systems
Vladimir Sementsov-Ogievskiy (13):
qcow2.py: python style fixes
qcow2.py: add licensing blurb
qcow2.py: move qcow2 format classes to separate module
qcow2_format.py: drop new line printing at end of dump()
qcow2_format.py: use tuples instead of lists for fields
qcow2_format.py: use modern string formatting
qcow2_format.py: use strings to specify c-type of struct fields
qcow2_format.py: separate generic functionality of structure classes
qcow2_format.py: add field-formatting class
qcow2_format.py: QcowHeaderExtension: add dump method
qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct
qcow2: QcowHeaderExtension print names for extension magics
qcow2_format.py: dump bitmaps header extension
docs/tools/qemu-img.rst | 2 +-
tests/qemu-iotests/031.out | 22 +--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/061.out | 14 +-
tests/qemu-iotests/291 | 8 +-
tests/qemu-iotests/291.out | 37 ++++-
tests/qemu-iotests/qcow2.py | 218 +++++++---------------------
tests/qemu-iotests/qcow2_format.py | 286 +++++++++++++++++++++++++++++++++++++
8 files changed, 398 insertions(+), 193 deletions(-)
create mode 100644 tests/qemu-iotests/qcow2_format.py
--
2.27.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 02/15] qcow2.py: python style fixes Eric Blake
` (14 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Vladimir Sementsov-Ogievskiy
Prefer a consistent naming for the --merge argument.
Fixes: 3b51ab4bf
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200529144527.1943527-1-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
docs/tools/qemu-img.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index 69cd9a30373a..7f0737488ade 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -300,7 +300,7 @@ Command description:
``--disable`` to change *BITMAP* to stop recording future edits.
- ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*.
+ ``--merge`` to merge the contents of the *SOURCE* bitmap into *BITMAP*.
Additional options include ``-g`` which sets a non-default
*GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 02/15] qcow2.py: python style fixes
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
2020-06-09 20:52 ` [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 03/15] qcow2.py: add licensing blurb Eric Blake
` (13 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Fix flake8 complaints.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-2-vsementsov@virtuozzo.com>
Tested-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: commit message improved]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2.py | 95 ++++++++++++++++++++++---------------
1 file changed, 56 insertions(+), 39 deletions(-)
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index 94a07b2f6fa2..d99f4ee3e848 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -4,6 +4,7 @@ import sys
import struct
import string
+
class QcowHeaderExtension:
def __init__(self, magic, length, data):
@@ -11,14 +12,15 @@ class QcowHeaderExtension:
padding = 8 - (length % 8)
data += b"\0" * padding
- self.magic = magic
+ self.magic = magic
self.length = length
- self.data = data
+ self.data = data
@classmethod
def create(cls, magic, data):
return QcowHeaderExtension(magic, len(data), data)
+
class QcowHeader:
uint32_t = 'I'
@@ -26,27 +28,27 @@ class QcowHeader:
fields = [
# Version 2 header fields
- [ uint32_t, '%#x', 'magic' ],
- [ uint32_t, '%d', 'version' ],
- [ uint64_t, '%#x', 'backing_file_offset' ],
- [ uint32_t, '%#x', 'backing_file_size' ],
- [ uint32_t, '%d', 'cluster_bits' ],
- [ uint64_t, '%d', 'size' ],
- [ uint32_t, '%d', 'crypt_method' ],
- [ uint32_t, '%d', 'l1_size' ],
- [ uint64_t, '%#x', 'l1_table_offset' ],
- [ uint64_t, '%#x', 'refcount_table_offset' ],
- [ uint32_t, '%d', 'refcount_table_clusters' ],
- [ uint32_t, '%d', 'nb_snapshots' ],
- [ uint64_t, '%#x', 'snapshot_offset' ],
+ [uint32_t, '%#x', 'magic'],
+ [uint32_t, '%d', 'version'],
+ [uint64_t, '%#x', 'backing_file_offset'],
+ [uint32_t, '%#x', 'backing_file_size'],
+ [uint32_t, '%d', 'cluster_bits'],
+ [uint64_t, '%d', 'size'],
+ [uint32_t, '%d', 'crypt_method'],
+ [uint32_t, '%d', 'l1_size'],
+ [uint64_t, '%#x', 'l1_table_offset'],
+ [uint64_t, '%#x', 'refcount_table_offset'],
+ [uint32_t, '%d', 'refcount_table_clusters'],
+ [uint32_t, '%d', 'nb_snapshots'],
+ [uint64_t, '%#x', 'snapshot_offset'],
# Version 3 header fields
- [ uint64_t, 'mask', 'incompatible_features' ],
- [ uint64_t, 'mask', 'compatible_features' ],
- [ uint64_t, 'mask', 'autoclear_features' ],
- [ uint32_t, '%d', 'refcount_order' ],
- [ uint32_t, '%d', 'header_length' ],
- ];
+ [uint64_t, 'mask', 'incompatible_features'],
+ [uint64_t, 'mask', 'compatible_features'],
+ [uint64_t, 'mask', 'autoclear_features'],
+ [uint32_t, '%d', 'refcount_order'],
+ [uint32_t, '%d', 'header_length'],
+ ]
fmt = '>' + ''.join(field[0] for field in fields)
@@ -59,7 +61,7 @@ class QcowHeader:
header = struct.unpack(QcowHeader.fmt, buf)
self.__dict__ = dict((field[2], header[i])
- for i, field in enumerate(QcowHeader.fields))
+ for i, field in enumerate(QcowHeader.fields))
self.set_defaults()
self.cluster_size = 1 << self.cluster_bits
@@ -96,7 +98,8 @@ class QcowHeader:
else:
padded = (length + 7) & ~7
data = fd.read(padded)
- self.extensions.append(QcowHeaderExtension(magic, length, data))
+ self.extensions.append(QcowHeaderExtension(magic, length,
+ data))
def update_extensions(self, fd):
@@ -108,14 +111,13 @@ class QcowHeader:
fd.write(buf)
fd.write(ex.data)
- if self.backing_file != None:
+ if self.backing_file is not None:
self.backing_file_offset = fd.tell()
fd.write(self.backing_file)
if fd.tell() > self.cluster_size:
raise Exception("I think I just broke the image...")
-
def update(self, fd):
header_bytes = self.header_length
@@ -163,19 +165,21 @@ def cmd_dump_header(fd):
h.dump()
h.dump_extensions()
+
def cmd_dump_header_exts(fd):
h = QcowHeader(fd)
h.dump_extensions()
+
def cmd_set_header(fd, name, value):
try:
value = int(value, 0)
- except:
+ except ValueError:
print("'%s' is not a valid number" % value)
sys.exit(1)
fields = (field[2] for field in QcowHeader.fields)
- if not name in fields:
+ if name not in fields:
print("'%s' is not a known header field" % name)
sys.exit(1)
@@ -183,25 +187,29 @@ def cmd_set_header(fd, name, value):
h.__dict__[name] = value
h.update(fd)
+
def cmd_add_header_ext(fd, magic, data):
try:
magic = int(magic, 0)
- except:
+ except ValueError:
print("'%s' is not a valid magic number" % magic)
sys.exit(1)
h = QcowHeader(fd)
- h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
+ h.extensions.append(QcowHeaderExtension.create(magic,
+ data.encode('ascii')))
h.update(fd)
+
def cmd_add_header_ext_stdio(fd, magic):
data = sys.stdin.read()
cmd_add_header_ext(fd, magic, data)
+
def cmd_del_header_ext(fd, magic):
try:
magic = int(magic, 0)
- except:
+ except ValueError:
print("'%s' is not a valid magic number" % magic)
sys.exit(1)
@@ -219,12 +227,13 @@ def cmd_del_header_ext(fd, magic):
h.update(fd)
+
def cmd_set_feature_bit(fd, group, bit):
try:
bit = int(bit, 0)
if bit < 0 or bit >= 64:
raise ValueError
- except:
+ except ValueError:
print("'%s' is not a valid bit number in range [0, 64)" % bit)
sys.exit(1)
@@ -236,21 +245,27 @@ def cmd_set_feature_bit(fd, group, bit):
elif group == 'autoclear':
h.autoclear_features |= 1 << bit
else:
- print("'%s' is not a valid group, try 'incompatible', 'compatible', or 'autoclear'" % group)
+ print("'%s' is not a valid group, try "
+ "'incompatible', 'compatible', or 'autoclear'" % group)
sys.exit(1)
h.update(fd)
+
cmds = [
- [ 'dump-header', cmd_dump_header, 0, 'Dump image header and header extensions' ],
- [ 'dump-header-exts', cmd_dump_header_exts, 0, 'Dump image header extensions' ],
- [ 'set-header', cmd_set_header, 2, 'Set a field in the header'],
- [ 'add-header-ext', cmd_add_header_ext, 2, 'Add a header extension' ],
- [ 'add-header-ext-stdio', cmd_add_header_ext_stdio, 1, 'Add a header extension, data from stdin' ],
- [ 'del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension' ],
- [ 'set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'],
+ ['dump-header', cmd_dump_header, 0,
+ 'Dump image header and header extensions'],
+ ['dump-header-exts', cmd_dump_header_exts, 0,
+ 'Dump image header extensions'],
+ ['set-header', cmd_set_header, 2, 'Set a field in the header'],
+ ['add-header-ext', cmd_add_header_ext, 2, 'Add a header extension'],
+ ['add-header-ext-stdio', cmd_add_header_ext_stdio, 1,
+ 'Add a header extension, data from stdin'],
+ ['del-header-ext', cmd_del_header_ext, 1, 'Delete a header extension'],
+ ['set-feature-bit', cmd_set_feature_bit, 2, 'Set a feature bit'],
]
+
def main(filename, cmd, args):
fd = open(filename, "r+b")
try:
@@ -267,6 +282,7 @@ def main(filename, cmd, args):
finally:
fd.close()
+
def usage():
print("Usage: %s <file> <cmd> [<arg>, ...]" % sys.argv[0])
print("")
@@ -274,6 +290,7 @@ def usage():
for name, handler, num_args, desc in cmds:
print(" %-20s - %s" % (name, desc))
+
if __name__ == '__main__':
if len(sys.argv) < 3:
usage()
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 03/15] qcow2.py: add licensing blurb
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
2020-06-09 20:52 ` [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand Eric Blake
2020-06-09 20:52 ` [PULL 02/15] qcow2.py: python style fixes Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 04/15] qcow2.py: move qcow2 format classes to separate module Eric Blake
` (12 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Add classic heading, which is missing here. Keep copyright place empty,
prior authors may add a line later.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-3-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: tweak commit message]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index d99f4ee3e848..2da434a013ef 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -1,4 +1,20 @@
#!/usr/bin/env python3
+#
+# Manipulations with qcow2 image
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
import sys
import struct
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 04/15] qcow2.py: move qcow2 format classes to separate module
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (2 preceding siblings ...)
2020-06-09 20:52 ` [PULL 03/15] qcow2.py: add licensing blurb Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 05/15] qcow2_format.py: drop new line printing at end of dump() Eric Blake
` (11 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
We are going to enhance qcow2 format parsing by adding more structure
classes. Let's split format parsing from utility code.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2.py | 160 +-------------------------
tests/qemu-iotests/qcow2_format.py | 173 +++++++++++++++++++++++++++++
2 files changed, 177 insertions(+), 156 deletions(-)
create mode 100644 tests/qemu-iotests/qcow2_format.py
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index 2da434a013ef..e968869ea6b2 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -17,163 +17,11 @@
#
import sys
-import struct
-import string
-
-class QcowHeaderExtension:
-
- def __init__(self, magic, length, data):
- if length % 8 != 0:
- padding = 8 - (length % 8)
- data += b"\0" * padding
-
- self.magic = magic
- self.length = length
- self.data = data
-
- @classmethod
- def create(cls, magic, data):
- return QcowHeaderExtension(magic, len(data), data)
-
-
-class QcowHeader:
-
- uint32_t = 'I'
- uint64_t = 'Q'
-
- fields = [
- # Version 2 header fields
- [uint32_t, '%#x', 'magic'],
- [uint32_t, '%d', 'version'],
- [uint64_t, '%#x', 'backing_file_offset'],
- [uint32_t, '%#x', 'backing_file_size'],
- [uint32_t, '%d', 'cluster_bits'],
- [uint64_t, '%d', 'size'],
- [uint32_t, '%d', 'crypt_method'],
- [uint32_t, '%d', 'l1_size'],
- [uint64_t, '%#x', 'l1_table_offset'],
- [uint64_t, '%#x', 'refcount_table_offset'],
- [uint32_t, '%d', 'refcount_table_clusters'],
- [uint32_t, '%d', 'nb_snapshots'],
- [uint64_t, '%#x', 'snapshot_offset'],
-
- # Version 3 header fields
- [uint64_t, 'mask', 'incompatible_features'],
- [uint64_t, 'mask', 'compatible_features'],
- [uint64_t, 'mask', 'autoclear_features'],
- [uint32_t, '%d', 'refcount_order'],
- [uint32_t, '%d', 'header_length'],
- ]
-
- fmt = '>' + ''.join(field[0] for field in fields)
-
- def __init__(self, fd):
-
- buf_size = struct.calcsize(QcowHeader.fmt)
-
- fd.seek(0)
- buf = fd.read(buf_size)
-
- header = struct.unpack(QcowHeader.fmt, buf)
- self.__dict__ = dict((field[2], header[i])
- for i, field in enumerate(QcowHeader.fields))
-
- self.set_defaults()
- self.cluster_size = 1 << self.cluster_bits
-
- fd.seek(self.header_length)
- self.load_extensions(fd)
-
- if self.backing_file_offset:
- fd.seek(self.backing_file_offset)
- self.backing_file = fd.read(self.backing_file_size)
- else:
- self.backing_file = None
-
- def set_defaults(self):
- if self.version == 2:
- self.incompatible_features = 0
- self.compatible_features = 0
- self.autoclear_features = 0
- self.refcount_order = 4
- self.header_length = 72
-
- def load_extensions(self, fd):
- self.extensions = []
-
- if self.backing_file_offset != 0:
- end = min(self.cluster_size, self.backing_file_offset)
- else:
- end = self.cluster_size
-
- while fd.tell() < end:
- (magic, length) = struct.unpack('>II', fd.read(8))
- if magic == 0:
- break
- else:
- padded = (length + 7) & ~7
- data = fd.read(padded)
- self.extensions.append(QcowHeaderExtension(magic, length,
- data))
-
- def update_extensions(self, fd):
-
- fd.seek(self.header_length)
- extensions = self.extensions
- extensions.append(QcowHeaderExtension(0, 0, b""))
- for ex in extensions:
- buf = struct.pack('>II', ex.magic, ex.length)
- fd.write(buf)
- fd.write(ex.data)
-
- if self.backing_file is not None:
- self.backing_file_offset = fd.tell()
- fd.write(self.backing_file)
-
- if fd.tell() > self.cluster_size:
- raise Exception("I think I just broke the image...")
-
- def update(self, fd):
- header_bytes = self.header_length
-
- self.update_extensions(fd)
-
- fd.seek(0)
- header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields)
- buf = struct.pack(QcowHeader.fmt, *header)
- buf = buf[0:header_bytes-1]
- fd.write(buf)
-
- def dump(self):
- for f in QcowHeader.fields:
- value = self.__dict__[f[2]]
- if f[1] == 'mask':
- bits = []
- for bit in range(64):
- if value & (1 << bit):
- bits.append(bit)
- value_str = str(bits)
- else:
- value_str = f[1] % value
-
- print("%-25s" % f[2], value_str)
- print("")
-
- def dump_extensions(self):
- for ex in self.extensions:
-
- data = ex.data[:ex.length]
- if all(c in string.printable.encode('ascii') for c in data):
- data = "'%s'" % data.decode('ascii')
- else:
- data = "<binary>"
-
- print("Header extension:")
- print("%-25s %#x" % ("magic", ex.magic))
- print("%-25s %d" % ("length", ex.length))
- print("%-25s %s" % ("data", data))
- print("")
+from qcow2_format import (
+ QcowHeader,
+ QcowHeaderExtension
+)
def cmd_dump_header(fd):
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
new file mode 100644
index 000000000000..0e517c1bbc5a
--- /dev/null
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -0,0 +1,173 @@
+# Library for manipulations with qcow2 image
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import struct
+import string
+
+
+class QcowHeaderExtension:
+
+ def __init__(self, magic, length, data):
+ if length % 8 != 0:
+ padding = 8 - (length % 8)
+ data += b"\0" * padding
+
+ self.magic = magic
+ self.length = length
+ self.data = data
+
+ @classmethod
+ def create(cls, magic, data):
+ return QcowHeaderExtension(magic, len(data), data)
+
+
+class QcowHeader:
+
+ uint32_t = 'I'
+ uint64_t = 'Q'
+
+ fields = [
+ # Version 2 header fields
+ [uint32_t, '%#x', 'magic'],
+ [uint32_t, '%d', 'version'],
+ [uint64_t, '%#x', 'backing_file_offset'],
+ [uint32_t, '%#x', 'backing_file_size'],
+ [uint32_t, '%d', 'cluster_bits'],
+ [uint64_t, '%d', 'size'],
+ [uint32_t, '%d', 'crypt_method'],
+ [uint32_t, '%d', 'l1_size'],
+ [uint64_t, '%#x', 'l1_table_offset'],
+ [uint64_t, '%#x', 'refcount_table_offset'],
+ [uint32_t, '%d', 'refcount_table_clusters'],
+ [uint32_t, '%d', 'nb_snapshots'],
+ [uint64_t, '%#x', 'snapshot_offset'],
+
+ # Version 3 header fields
+ [uint64_t, 'mask', 'incompatible_features'],
+ [uint64_t, 'mask', 'compatible_features'],
+ [uint64_t, 'mask', 'autoclear_features'],
+ [uint32_t, '%d', 'refcount_order'],
+ [uint32_t, '%d', 'header_length'],
+ ]
+
+ fmt = '>' + ''.join(field[0] for field in fields)
+
+ def __init__(self, fd):
+
+ buf_size = struct.calcsize(QcowHeader.fmt)
+
+ fd.seek(0)
+ buf = fd.read(buf_size)
+
+ header = struct.unpack(QcowHeader.fmt, buf)
+ self.__dict__ = dict((field[2], header[i])
+ for i, field in enumerate(QcowHeader.fields))
+
+ self.set_defaults()
+ self.cluster_size = 1 << self.cluster_bits
+
+ fd.seek(self.header_length)
+ self.load_extensions(fd)
+
+ if self.backing_file_offset:
+ fd.seek(self.backing_file_offset)
+ self.backing_file = fd.read(self.backing_file_size)
+ else:
+ self.backing_file = None
+
+ def set_defaults(self):
+ if self.version == 2:
+ self.incompatible_features = 0
+ self.compatible_features = 0
+ self.autoclear_features = 0
+ self.refcount_order = 4
+ self.header_length = 72
+
+ def load_extensions(self, fd):
+ self.extensions = []
+
+ if self.backing_file_offset != 0:
+ end = min(self.cluster_size, self.backing_file_offset)
+ else:
+ end = self.cluster_size
+
+ while fd.tell() < end:
+ (magic, length) = struct.unpack('>II', fd.read(8))
+ if magic == 0:
+ break
+ else:
+ padded = (length + 7) & ~7
+ data = fd.read(padded)
+ self.extensions.append(QcowHeaderExtension(magic, length,
+ data))
+
+ def update_extensions(self, fd):
+
+ fd.seek(self.header_length)
+ extensions = self.extensions
+ extensions.append(QcowHeaderExtension(0, 0, b""))
+ for ex in extensions:
+ buf = struct.pack('>II', ex.magic, ex.length)
+ fd.write(buf)
+ fd.write(ex.data)
+
+ if self.backing_file is not None:
+ self.backing_file_offset = fd.tell()
+ fd.write(self.backing_file)
+
+ if fd.tell() > self.cluster_size:
+ raise Exception("I think I just broke the image...")
+
+ def update(self, fd):
+ header_bytes = self.header_length
+
+ self.update_extensions(fd)
+
+ fd.seek(0)
+ header = tuple(self.__dict__[f] for t, p, f in QcowHeader.fields)
+ buf = struct.pack(QcowHeader.fmt, *header)
+ buf = buf[0:header_bytes-1]
+ fd.write(buf)
+
+ def dump(self):
+ for f in QcowHeader.fields:
+ value = self.__dict__[f[2]]
+ if f[1] == 'mask':
+ bits = []
+ for bit in range(64):
+ if value & (1 << bit):
+ bits.append(bit)
+ value_str = str(bits)
+ else:
+ value_str = f[1] % value
+
+ print("%-25s" % f[2], value_str)
+ print("")
+
+ def dump_extensions(self):
+ for ex in self.extensions:
+
+ data = ex.data[:ex.length]
+ if all(c in string.printable.encode('ascii') for c in data):
+ data = "'%s'" % data.decode('ascii')
+ else:
+ data = "<binary>"
+
+ print("Header extension:")
+ print("%-25s %#x" % ("magic", ex.magic))
+ print("%-25s %d" % ("length", ex.length))
+ print("%-25s %s" % ("data", data))
+ print("")
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 05/15] qcow2_format.py: drop new line printing at end of dump()
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (3 preceding siblings ...)
2020-06-09 20:52 ` [PULL 04/15] qcow2.py: move qcow2 format classes to separate module Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 06/15] qcow2_format.py: use tuples instead of lists for fields Eric Blake
` (10 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
This will simplify further conversion. To compensate, print this empty
line directly in cmd_dump_header().
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2.py | 1 +
tests/qemu-iotests/qcow2_format.py | 1 -
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index e968869ea6b2..8c187e9a7292 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -27,6 +27,7 @@ from qcow2_format import (
def cmd_dump_header(fd):
h = QcowHeader(fd)
h.dump()
+ print()
h.dump_extensions()
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 0e517c1bbc5a..2b6c9612ae01 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -155,7 +155,6 @@ class QcowHeader:
value_str = f[1] % value
print("%-25s" % f[2], value_str)
- print("")
def dump_extensions(self):
for ex in self.extensions:
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 06/15] qcow2_format.py: use tuples instead of lists for fields
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (4 preceding siblings ...)
2020-06-09 20:52 ` [PULL 05/15] qcow2_format.py: drop new line printing at end of dump() Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 07/15] qcow2_format.py: use modern string formatting Eric Blake
` (9 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
No need in lists: it's a constant variable.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-6-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 40 +++++++++++++++---------------
1 file changed, 20 insertions(+), 20 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 2b6c9612ae01..e2f08ed69194 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -39,29 +39,29 @@ class QcowHeader:
uint32_t = 'I'
uint64_t = 'Q'
- fields = [
+ fields = (
# Version 2 header fields
- [uint32_t, '%#x', 'magic'],
- [uint32_t, '%d', 'version'],
- [uint64_t, '%#x', 'backing_file_offset'],
- [uint32_t, '%#x', 'backing_file_size'],
- [uint32_t, '%d', 'cluster_bits'],
- [uint64_t, '%d', 'size'],
- [uint32_t, '%d', 'crypt_method'],
- [uint32_t, '%d', 'l1_size'],
- [uint64_t, '%#x', 'l1_table_offset'],
- [uint64_t, '%#x', 'refcount_table_offset'],
- [uint32_t, '%d', 'refcount_table_clusters'],
- [uint32_t, '%d', 'nb_snapshots'],
- [uint64_t, '%#x', 'snapshot_offset'],
+ (uint32_t, '%#x', 'magic'),
+ (uint32_t, '%d', 'version'),
+ (uint64_t, '%#x', 'backing_file_offset'),
+ (uint32_t, '%#x', 'backing_file_size'),
+ (uint32_t, '%d', 'cluster_bits'),
+ (uint64_t, '%d', 'size'),
+ (uint32_t, '%d', 'crypt_method'),
+ (uint32_t, '%d', 'l1_size'),
+ (uint64_t, '%#x', 'l1_table_offset'),
+ (uint64_t, '%#x', 'refcount_table_offset'),
+ (uint32_t, '%d', 'refcount_table_clusters'),
+ (uint32_t, '%d', 'nb_snapshots'),
+ (uint64_t, '%#x', 'snapshot_offset'),
# Version 3 header fields
- [uint64_t, 'mask', 'incompatible_features'],
- [uint64_t, 'mask', 'compatible_features'],
- [uint64_t, 'mask', 'autoclear_features'],
- [uint32_t, '%d', 'refcount_order'],
- [uint32_t, '%d', 'header_length'],
- ]
+ (uint64_t, 'mask', 'incompatible_features'),
+ (uint64_t, 'mask', 'compatible_features'),
+ (uint64_t, 'mask', 'autoclear_features'),
+ (uint32_t, '%d', 'refcount_order'),
+ (uint32_t, '%d', 'header_length'),
+ )
fmt = '>' + ''.join(field[0] for field in fields)
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 07/15] qcow2_format.py: use modern string formatting
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (5 preceding siblings ...)
2020-06-09 20:52 ` [PULL 06/15] qcow2_format.py: use tuples instead of lists for fields Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 08/15] qcow2_format.py: use strings to specify c-type of struct fields Eric Blake
` (8 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Use .format and f-strings instead of old %style. Also, the file uses
both '' and "" quotes, for consistency let's use '', except for cases
when we need '' inside the string (use "" to avoid extra escaping).
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 54 +++++++++++++++---------------
1 file changed, 27 insertions(+), 27 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index e2f08ed69194..da66df340876 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -23,7 +23,7 @@ class QcowHeaderExtension:
def __init__(self, magic, length, data):
if length % 8 != 0:
padding = 8 - (length % 8)
- data += b"\0" * padding
+ data += b'\0' * padding
self.magic = magic
self.length = length
@@ -41,26 +41,26 @@ class QcowHeader:
fields = (
# Version 2 header fields
- (uint32_t, '%#x', 'magic'),
- (uint32_t, '%d', 'version'),
- (uint64_t, '%#x', 'backing_file_offset'),
- (uint32_t, '%#x', 'backing_file_size'),
- (uint32_t, '%d', 'cluster_bits'),
- (uint64_t, '%d', 'size'),
- (uint32_t, '%d', 'crypt_method'),
- (uint32_t, '%d', 'l1_size'),
- (uint64_t, '%#x', 'l1_table_offset'),
- (uint64_t, '%#x', 'refcount_table_offset'),
- (uint32_t, '%d', 'refcount_table_clusters'),
- (uint32_t, '%d', 'nb_snapshots'),
- (uint64_t, '%#x', 'snapshot_offset'),
+ (uint32_t, '{:#x}', 'magic'),
+ (uint32_t, '{}', 'version'),
+ (uint64_t, '{:#x}', 'backing_file_offset'),
+ (uint32_t, '{:#x}', 'backing_file_size'),
+ (uint32_t, '{}', 'cluster_bits'),
+ (uint64_t, '{}', 'size'),
+ (uint32_t, '{}', 'crypt_method'),
+ (uint32_t, '{}', 'l1_size'),
+ (uint64_t, '{:#x}', 'l1_table_offset'),
+ (uint64_t, '{:#x}', 'refcount_table_offset'),
+ (uint32_t, '{}', 'refcount_table_clusters'),
+ (uint32_t, '{}', 'nb_snapshots'),
+ (uint64_t, '{:#x}', 'snapshot_offset'),
# Version 3 header fields
(uint64_t, 'mask', 'incompatible_features'),
(uint64_t, 'mask', 'compatible_features'),
(uint64_t, 'mask', 'autoclear_features'),
- (uint32_t, '%d', 'refcount_order'),
- (uint32_t, '%d', 'header_length'),
+ (uint32_t, '{}', 'refcount_order'),
+ (uint32_t, '{}', 'header_length'),
)
fmt = '>' + ''.join(field[0] for field in fields)
@@ -118,7 +118,7 @@ class QcowHeader:
fd.seek(self.header_length)
extensions = self.extensions
- extensions.append(QcowHeaderExtension(0, 0, b""))
+ extensions.append(QcowHeaderExtension(0, 0, b''))
for ex in extensions:
buf = struct.pack('>II', ex.magic, ex.length)
fd.write(buf)
@@ -129,7 +129,7 @@ class QcowHeader:
fd.write(self.backing_file)
if fd.tell() > self.cluster_size:
- raise Exception("I think I just broke the image...")
+ raise Exception('I think I just broke the image...')
def update(self, fd):
header_bytes = self.header_length
@@ -152,21 +152,21 @@ class QcowHeader:
bits.append(bit)
value_str = str(bits)
else:
- value_str = f[1] % value
+ value_str = f[1].format(value)
- print("%-25s" % f[2], value_str)
+ print(f'{f[2]:<25} {value_str}')
def dump_extensions(self):
for ex in self.extensions:
data = ex.data[:ex.length]
if all(c in string.printable.encode('ascii') for c in data):
- data = "'%s'" % data.decode('ascii')
+ data = f"'{ data.decode('ascii') }'"
else:
- data = "<binary>"
+ data = '<binary>'
- print("Header extension:")
- print("%-25s %#x" % ("magic", ex.magic))
- print("%-25s %d" % ("length", ex.length))
- print("%-25s %s" % ("data", data))
- print("")
+ print('Header extension:')
+ print(f'{"magic":<25} {ex.magic:#x}')
+ print(f'{"length":<25} {ex.length}')
+ print(f'{"data":<25} {data}')
+ print()
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 08/15] qcow2_format.py: use strings to specify c-type of struct fields
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (6 preceding siblings ...)
2020-06-09 20:52 ` [PULL 07/15] qcow2_format.py: use modern string formatting Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 09/15] qcow2_format.py: separate generic functionality of structure classes Eric Blake
` (7 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
We are going to move field-parsing to super-class, this will be simpler
with simple string specifiers instead of variables.
For some reason, python doesn't allow the definition of ctypes variable
in the class alongside fields: it would not be available then for use
by the 'for' operator. Don't worry: ctypes will be moved to metaclass
soon.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-8-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 50 +++++++++++++++++-------------
1 file changed, 28 insertions(+), 22 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index da66df340876..28f2bfa63bc7 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -34,36 +34,42 @@ class QcowHeaderExtension:
return QcowHeaderExtension(magic, len(data), data)
+# Mapping from c types to python struct format
+ctypes = {
+ 'u8': 'B',
+ 'u16': 'H',
+ 'u32': 'I',
+ 'u64': 'Q'
+}
+
+
class QcowHeader:
- uint32_t = 'I'
- uint64_t = 'Q'
-
fields = (
# Version 2 header fields
- (uint32_t, '{:#x}', 'magic'),
- (uint32_t, '{}', 'version'),
- (uint64_t, '{:#x}', 'backing_file_offset'),
- (uint32_t, '{:#x}', 'backing_file_size'),
- (uint32_t, '{}', 'cluster_bits'),
- (uint64_t, '{}', 'size'),
- (uint32_t, '{}', 'crypt_method'),
- (uint32_t, '{}', 'l1_size'),
- (uint64_t, '{:#x}', 'l1_table_offset'),
- (uint64_t, '{:#x}', 'refcount_table_offset'),
- (uint32_t, '{}', 'refcount_table_clusters'),
- (uint32_t, '{}', 'nb_snapshots'),
- (uint64_t, '{:#x}', 'snapshot_offset'),
+ ('u32', '{:#x}', 'magic'),
+ ('u32', '{}', 'version'),
+ ('u64', '{:#x}', 'backing_file_offset'),
+ ('u32', '{:#x}', 'backing_file_size'),
+ ('u32', '{}', 'cluster_bits'),
+ ('u64', '{}', 'size'),
+ ('u32', '{}', 'crypt_method'),
+ ('u32', '{}', 'l1_size'),
+ ('u64', '{:#x}', 'l1_table_offset'),
+ ('u64', '{:#x}', 'refcount_table_offset'),
+ ('u32', '{}', 'refcount_table_clusters'),
+ ('u32', '{}', 'nb_snapshots'),
+ ('u64', '{:#x}', 'snapshot_offset'),
# Version 3 header fields
- (uint64_t, 'mask', 'incompatible_features'),
- (uint64_t, 'mask', 'compatible_features'),
- (uint64_t, 'mask', 'autoclear_features'),
- (uint32_t, '{}', 'refcount_order'),
- (uint32_t, '{}', 'header_length'),
+ ('u64', 'mask', 'incompatible_features'),
+ ('u64', 'mask', 'compatible_features'),
+ ('u64', 'mask', 'autoclear_features'),
+ ('u32', '{}', 'refcount_order'),
+ ('u32', '{}', 'header_length'),
)
- fmt = '>' + ''.join(field[0] for field in fields)
+ fmt = '>' + ''.join(ctypes[f[0]] for f in fields)
def __init__(self, fd):
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 09/15] qcow2_format.py: separate generic functionality of structure classes
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (7 preceding siblings ...)
2020-06-09 20:52 ` [PULL 08/15] qcow2_format.py: use strings to specify c-type of struct fields Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 10/15] qcow2_format.py: add field-formatting class Eric Blake
` (6 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
We are going to introduce more Qcow2 structure types, defined like
QcowHeader. Move generic functionality into base class to be reused for
further structure classes.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-9-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 101 +++++++++++++++++++----------
1 file changed, 66 insertions(+), 35 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 28f2bfa63bc7..898d388b8adf 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -1,5 +1,7 @@
# Library for manipulations with qcow2 image
#
+# Copyright (c) 2020 Virtuozzo International GmbH.
+#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
@@ -18,6 +20,68 @@ import struct
import string
+class Qcow2StructMeta(type):
+
+ # Mapping from c types to python struct format
+ ctypes = {
+ 'u8': 'B',
+ 'u16': 'H',
+ 'u32': 'I',
+ 'u64': 'Q'
+ }
+
+ def __init__(self, name, bases, attrs):
+ if 'fields' in attrs:
+ self.fmt = '>' + ''.join(self.ctypes[f[0]] for f in self.fields)
+
+
+class Qcow2Struct(metaclass=Qcow2StructMeta):
+
+ """Qcow2Struct: base class for qcow2 data structures
+
+ Successors should define fields class variable, which is: list of tuples,
+ each of three elements:
+ - c-type (one of 'u8', 'u16', 'u32', 'u64')
+ - format (format_spec to use with .format() when dump or 'mask' to dump
+ bitmasks)
+ - field name
+ """
+
+ def __init__(self, fd=None, offset=None, data=None):
+ """
+ Two variants:
+ 1. Specify data. fd and offset must be None.
+ 2. Specify fd and offset, data must be None. offset may be omitted
+ in this case, than current position of fd is used.
+ """
+ if data is None:
+ assert fd is not None
+ buf_size = struct.calcsize(self.fmt)
+ if offset is not None:
+ fd.seek(offset)
+ data = fd.read(buf_size)
+ else:
+ assert fd is None and offset is None
+
+ values = struct.unpack(self.fmt, data)
+ self.__dict__ = dict((field[2], values[i])
+ for i, field in enumerate(self.fields))
+
+ def dump(self):
+ for f in self.fields:
+ value = self.__dict__[f[2]]
+ if f[1] == 'mask':
+ bits = []
+ for bit in range(64):
+ if value & (1 << bit):
+ bits.append(bit)
+ value_str = str(bits)
+ else:
+ value_str = f[1].format(value)
+
+ print('{:<25} {}'.format(f[2], value_str))
+
+
class QcowHeaderExtension:
def __init__(self, magic, length, data):
@@ -34,16 +98,7 @@ class QcowHeaderExtension:
return QcowHeaderExtension(magic, len(data), data)
-# Mapping from c types to python struct format
-ctypes = {
- 'u8': 'B',
- 'u16': 'H',
- 'u32': 'I',
- 'u64': 'Q'
-}
-
-
-class QcowHeader:
+class QcowHeader(Qcow2Struct):
fields = (
# Version 2 header fields
@@ -69,18 +124,8 @@ class QcowHeader:
('u32', '{}', 'header_length'),
)
- fmt = '>' + ''.join(ctypes[f[0]] for f in fields)
-
def __init__(self, fd):
-
- buf_size = struct.calcsize(QcowHeader.fmt)
-
- fd.seek(0)
- buf = fd.read(buf_size)
-
- header = struct.unpack(QcowHeader.fmt, buf)
- self.__dict__ = dict((field[2], header[i])
- for i, field in enumerate(QcowHeader.fields))
+ super().__init__(fd=fd, offset=0)
self.set_defaults()
self.cluster_size = 1 << self.cluster_bits
@@ -148,20 +193,6 @@ class QcowHeader:
buf = buf[0:header_bytes-1]
fd.write(buf)
- def dump(self):
- for f in QcowHeader.fields:
- value = self.__dict__[f[2]]
- if f[1] == 'mask':
- bits = []
- for bit in range(64):
- if value & (1 << bit):
- bits.append(bit)
- value_str = str(bits)
- else:
- value_str = f[1].format(value)
-
- print(f'{f[2]:<25} {value_str}')
-
def dump_extensions(self):
for ex in self.extensions:
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 10/15] qcow2_format.py: add field-formatting class
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (8 preceding siblings ...)
2020-06-09 20:52 ` [PULL 09/15] qcow2_format.py: separate generic functionality of structure classes Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 11/15] qcow2_format.py: QcowHeaderExtension: add dump method Eric Blake
` (5 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Allow formatter class in structure definition instead of hacking with
'mask'. This will simplify further introduction of new formatters.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-10-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 35 +++++++++++++++++++++---------
1 file changed, 25 insertions(+), 10 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 898d388b8adf..74a82f9263d4 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -20,6 +20,25 @@ import struct
import string
+class Qcow2Field:
+
+ def __init__(self, value):
+ self.value = value
+
+ def __str__(self):
+ return str(self.value)
+
+
+class Flags64(Qcow2Field):
+
+ def __str__(self):
+ bits = []
+ for bit in range(64):
+ if self.value & (1 << bit):
+ bits.append(bit)
+ return str(bits)
+
+
class Qcow2StructMeta(type):
# Mapping from c types to python struct format
@@ -70,14 +89,10 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
def dump(self):
for f in self.fields:
value = self.__dict__[f[2]]
- if f[1] == 'mask':
- bits = []
- for bit in range(64):
- if value & (1 << bit):
- bits.append(bit)
- value_str = str(bits)
- else:
+ if isinstance(f[1], str):
value_str = f[1].format(value)
+ else:
+ value_str = str(f[1](value))
print('{:<25} {}'.format(f[2], value_str))
@@ -117,9 +132,9 @@ class QcowHeader(Qcow2Struct):
('u64', '{:#x}', 'snapshot_offset'),
# Version 3 header fields
- ('u64', 'mask', 'incompatible_features'),
- ('u64', 'mask', 'compatible_features'),
- ('u64', 'mask', 'autoclear_features'),
+ ('u64', Flags64, 'incompatible_features'),
+ ('u64', Flags64, 'compatible_features'),
+ ('u64', Flags64, 'autoclear_features'),
('u32', '{}', 'refcount_order'),
('u32', '{}', 'header_length'),
)
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 11/15] qcow2_format.py: QcowHeaderExtension: add dump method
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (9 preceding siblings ...)
2020-06-09 20:52 ` [PULL 10/15] qcow2_format.py: add field-formatting class Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 12/15] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct Eric Blake
` (4 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Obviously, for-loop body in dump_extensions should be the dump method
of extension.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-11-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 74a82f9263d4..d4ad5431b296 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -108,6 +108,17 @@ class QcowHeaderExtension:
self.length = length
self.data = data
+ def dump(self):
+ data = self.data[:self.length]
+ if all(c in string.printable.encode('ascii') for c in data):
+ data = f"'{ data.decode('ascii') }'"
+ else:
+ data = '<binary>'
+
+ print(f'{"magic":<25} {self.magic:#x}')
+ print(f'{"length":<25} {self.length}')
+ print(f'{"data":<25} {data}')
+
@classmethod
def create(cls, magic, data):
return QcowHeaderExtension(magic, len(data), data)
@@ -210,15 +221,6 @@ class QcowHeader(Qcow2Struct):
def dump_extensions(self):
for ex in self.extensions:
-
- data = ex.data[:ex.length]
- if all(c in string.printable.encode('ascii') for c in data):
- data = f"'{ data.decode('ascii') }'"
- else:
- data = '<binary>'
-
print('Header extension:')
- print(f'{"magic":<25} {ex.magic:#x}')
- print(f'{"length":<25} {ex.length}')
- print(f'{"data":<25} {data}')
+ ex.dump()
print()
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 12/15] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (10 preceding siblings ...)
2020-06-09 20:52 ` [PULL 11/15] qcow2_format.py: QcowHeaderExtension: add dump method Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 13/15] qcow2: QcowHeaderExtension print names for extension magics Eric Blake
` (3 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Only two fields we can parse by generic code, but that is better than
nothing. Keep further refactoring of variable-length fields for another
day.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-12-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/qcow2_format.py | 53 +++++++++++++++++++++---------
1 file changed, 37 insertions(+), 16 deletions(-)
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index d4ad5431b296..32371e42da4e 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -97,16 +97,41 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
print('{:<25} {}'.format(f[2], value_str))
-class QcowHeaderExtension:
+class QcowHeaderExtension(Qcow2Struct):
- def __init__(self, magic, length, data):
- if length % 8 != 0:
- padding = 8 - (length % 8)
- data += b'\0' * padding
+ fields = (
+ ('u32', '{:#x}', 'magic'),
+ ('u32', '{}', 'length')
+ # length bytes of data follows
+ # then padding to next multiply of 8
+ )
- self.magic = magic
- self.length = length
- self.data = data
+ def __init__(self, magic=None, length=None, data=None, fd=None):
+ """
+ Support both loading from fd and creation from user data.
+ For fd-based creation current position in a file will be used to read
+ the data.
+
+ This should be somehow refactored and functionality should be moved to
+ superclass (to allow creation of any qcow2 struct), but then, fields
+ of variable length (data here) should be supported in base class
+ somehow. So, it's a TODO. We'll see how to properly refactor this when
+ we have more qcow2 structures.
+ """
+ if fd is None:
+ assert all(v is not None for v in (magic, length, data))
+ self.magic = magic
+ self.length = length
+ if length % 8 != 0:
+ padding = 8 - (length % 8)
+ data += b'\0' * padding
+ self.data = data
+ else:
+ assert all(v is None for v in (magic, length, data))
+ super().__init__(fd=fd)
+ padded = (self.length + 7) & ~7
+ self.data = fd.read(padded)
+ assert self.data is not None
def dump(self):
data = self.data[:self.length]
@@ -115,8 +140,7 @@ class QcowHeaderExtension:
else:
data = '<binary>'
- print(f'{"magic":<25} {self.magic:#x}')
- print(f'{"length":<25} {self.length}')
+ super().dump()
print(f'{"data":<25} {data}')
@classmethod
@@ -182,14 +206,11 @@ class QcowHeader(Qcow2Struct):
end = self.cluster_size
while fd.tell() < end:
- (magic, length) = struct.unpack('>II', fd.read(8))
- if magic == 0:
+ ext = QcowHeaderExtension(fd=fd)
+ if ext.magic == 0:
break
else:
- padded = (length + 7) & ~7
- data = fd.read(padded)
- self.extensions.append(QcowHeaderExtension(magic, length,
- data))
+ self.extensions.append(ext)
def update_extensions(self, fd):
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 13/15] qcow2: QcowHeaderExtension print names for extension magics
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (11 preceding siblings ...)
2020-06-09 20:52 ` [PULL 12/15] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-09 20:52 ` [PULL 14/15] qcow2_format.py: dump bitmaps header extension Eric Blake
` (2 subsequent siblings)
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Suggested-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-13-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/031.out | 22 +++++++++++-----------
tests/qemu-iotests/036.out | 4 ++--
tests/qemu-iotests/061.out | 14 +++++++-------
tests/qemu-iotests/qcow2_format.py | 17 ++++++++++++++++-
4 files changed, 36 insertions(+), 21 deletions(-)
diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
index 5a4beda6a23c..4b21d6a9ba18 100644
--- a/tests/qemu-iotests/031.out
+++ b/tests/qemu-iotests/031.out
@@ -25,7 +25,7 @@ refcount_order 4
header_length 72
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
@@ -53,7 +53,7 @@ refcount_order 4
header_length 72
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
@@ -81,12 +81,12 @@ refcount_order 4
header_length 72
Header extension:
-magic 0xe2792aca
+magic 0xe2792aca (Backing format)
length 11
data 'host_device'
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
@@ -116,12 +116,12 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
@@ -149,12 +149,12 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
@@ -182,17 +182,17 @@ refcount_order 4
header_length 112
Header extension:
-magic 0xe2792aca
+magic 0xe2792aca (Backing format)
length 11
data 'host_device'
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
Header extension:
-magic 0x12345678
+magic 0x12345678 (<unknown>)
length 31
data 'This is a test header extension'
diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
index e409acf60e2b..a9bed828e525 100644
--- a/tests/qemu-iotests/036.out
+++ b/tests/qemu-iotests/036.out
@@ -25,7 +25,7 @@ incompatible_features []
compatible_features []
autoclear_features [63]
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -37,7 +37,7 @@ incompatible_features []
compatible_features []
autoclear_features []
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index a51ad1b5ba52..2f03cf045cce 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -25,7 +25,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -83,7 +83,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -139,7 +139,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -194,7 +194,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -263,7 +263,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -325,7 +325,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
@@ -354,7 +354,7 @@ refcount_order 4
header_length 112
Header extension:
-magic 0x6803f857
+magic 0x6803f857 (Feature table)
length 336
data <binary>
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 32371e42da4e..40b5bf467b24 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -39,6 +39,12 @@ class Flags64(Qcow2Field):
return str(bits)
+class Enum(Qcow2Field):
+
+ def __str__(self):
+ return f'{self.value:#x} ({self.mapping.get(self.value, "<unknown>")})'
+
+
class Qcow2StructMeta(type):
# Mapping from c types to python struct format
@@ -99,8 +105,17 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
class QcowHeaderExtension(Qcow2Struct):
+ class Magic(Enum):
+ mapping = {
+ 0xe2792aca: 'Backing format',
+ 0x6803f857: 'Feature table',
+ 0x0537be77: 'Crypto header',
+ 0x23852875: 'Bitmaps',
+ 0x44415441: 'Data file'
+ }
+
fields = (
- ('u32', '{:#x}', 'magic'),
+ ('u32', Magic, 'magic'),
('u32', '{}', 'length')
# length bytes of data follows
# then padding to next multiply of 8
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 14/15] qcow2_format.py: dump bitmaps header extension
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (12 preceding siblings ...)
2020-06-09 20:52 ` [PULL 13/15] qcow2: QcowHeaderExtension print names for extension magics Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-18 13:13 ` Max Reitz
2020-06-09 20:52 ` [PULL 15/15] iotests: Fix 291 across more file systems Eric Blake
2020-06-11 17:00 ` [PULL 00/15] bitmaps patches for 2020-06-09 Peter Maydell
15 siblings, 1 reply; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core, Max Reitz
From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Add class for bitmap extension and dump its fields. Further work is to
dump bitmap directory.
Test new functionality inside 291 iotest.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-14-vsementsov@virtuozzo.com>
[eblake: fix iotest output]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/291 | 4 +++
tests/qemu-iotests/291.out | 33 +++++++++++++++++++++++
tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++-------
3 files changed, 70 insertions(+), 9 deletions(-)
diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
index 3ca83b9cd1f7..e0cffc7cb119 100755
--- a/tests/qemu-iotests/291
+++ b/tests/qemu-iotests/291
@@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
$QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1
$QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2
$QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
echo
echo "=== Bitmap preservation not possible to non-qcow2 ==="
@@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
$QEMU_IMG bitmap --remove --image-opts \
driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+echo "Check resulting qcow2 header extensions:"
+$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
echo
echo "=== Check bitmap contents ==="
diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out
index 8c62017567e9..ccfcdc5e35ce 100644
--- a/tests/qemu-iotests/291.out
+++ b/tests/qemu-iotests/291.out
@@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 1048576/1048576 bytes at offset 2097152
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Check resulting qcow2 header extensions:
+Header extension:
+magic 0xe2792aca (Backing format)
+length 5
+data 'qcow2'
+
+Header extension:
+magic 0x6803f857 (Feature table)
+length 336
+data <binary>
+
+Header extension:
+magic 0x23852875 (Bitmaps)
+length 24
+nb_bitmaps 2
+reserved32 0
+bitmap_directory_size 0x40
+bitmap_directory_offset 0x510000
+
=== Bitmap preservation not possible to non-qcow2 ===
@@ -65,6 +84,20 @@ Format specific information:
granularity: 65536
refcount bits: 16
corrupt: false
+Check resulting qcow2 header extensions:
+Header extension:
+magic 0x6803f857 (Feature table)
+length 336
+data <binary>
+
+Header extension:
+magic 0x23852875 (Bitmaps)
+length 24
+nb_bitmaps 3
+reserved32 0
+bitmap_directory_size 0x60
+bitmap_directory_offset 0x520000
+
=== Check bitmap contents ===
diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_format.py
index 40b5bf467b24..0f65fd161d5b 100644
--- a/tests/qemu-iotests/qcow2_format.py
+++ b/tests/qemu-iotests/qcow2_format.py
@@ -103,6 +103,19 @@ class Qcow2Struct(metaclass=Qcow2StructMeta):
print('{:<25} {}'.format(f[2], value_str))
+class Qcow2BitmapExt(Qcow2Struct):
+
+ fields = (
+ ('u32', '{}', 'nb_bitmaps'),
+ ('u32', '{}', 'reserved32'),
+ ('u64', '{:#x}', 'bitmap_directory_size'),
+ ('u64', '{:#x}', 'bitmap_directory_offset')
+ )
+
+
+QCOW2_EXT_MAGIC_BITMAPS = 0x23852875
+
+
class QcowHeaderExtension(Qcow2Struct):
class Magic(Enum):
@@ -110,7 +123,7 @@ class QcowHeaderExtension(Qcow2Struct):
0xe2792aca: 'Backing format',
0x6803f857: 'Feature table',
0x0537be77: 'Crypto header',
- 0x23852875: 'Bitmaps',
+ QCOW2_EXT_MAGIC_BITMAPS: 'Bitmaps',
0x44415441: 'Data file'
}
@@ -130,8 +143,11 @@ class QcowHeaderExtension(Qcow2Struct):
This should be somehow refactored and functionality should be moved to
superclass (to allow creation of any qcow2 struct), but then, fields
of variable length (data here) should be supported in base class
- somehow. So, it's a TODO. We'll see how to properly refactor this when
- we have more qcow2 structures.
+ somehow. Note also, that we probably want to parse different
+ extensions. Should they be subclasses of this class, or how to do it
+ better? Should it be something like QAPI union with discriminator field
+ (magic here). So, it's a TODO. We'll see how to properly refactor this
+ when we have more qcow2 structures.
"""
if fd is None:
assert all(v is not None for v in (magic, length, data))
@@ -148,15 +164,23 @@ class QcowHeaderExtension(Qcow2Struct):
self.data = fd.read(padded)
assert self.data is not None
- def dump(self):
- data = self.data[:self.length]
- if all(c in string.printable.encode('ascii') for c in data):
- data = f"'{ data.decode('ascii') }'"
+ if self.magic == QCOW2_EXT_MAGIC_BITMAPS:
+ self.obj = Qcow2BitmapExt(data=self.data)
else:
- data = '<binary>'
+ self.obj = None
+ def dump(self):
super().dump()
- print(f'{"data":<25} {data}')
+
+ if self.obj is None:
+ data = self.data[:self.length]
+ if all(c in string.printable.encode('ascii') for c in data):
+ data = f"'{ data.decode('ascii') }'"
+ else:
+ data = '<binary>'
+ print(f'{"data":<25} {data}')
+ else:
+ self.obj.dump()
@classmethod
def create(cls, magic, data):
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PULL 15/15] iotests: Fix 291 across more file systems
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (13 preceding siblings ...)
2020-06-09 20:52 ` [PULL 14/15] qcow2_format.py: dump bitmaps header extension Eric Blake
@ 2020-06-09 20:52 ` Eric Blake
2020-06-11 17:00 ` [PULL 00/15] bitmaps patches for 2020-06-09 Peter Maydell
15 siblings, 0 replies; 20+ messages in thread
From: Eric Blake @ 2020-06-09 20:52 UTC (permalink / raw)
To: qemu-devel; +Cc: Kevin Wolf, open list:Block layer core, Max Reitz
Depending on the granularity of holes and amount of metadata consumed
by a file, the 'disk size:' number of 'qemu-img info' is not reliable.
Adjust our test to use a different set of filters to avoid spurious
failures.
Reported-by: Kevin Wolf <kwolf@redhat.com>
Fixes: cf2d1203dc
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200608195629.3299649-1-eblake@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
[eblake: fix merge conflict]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
tests/qemu-iotests/291 | 4 ++--
tests/qemu-iotests/291.out | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
index e0cffc7cb119..404f8521f74a 100755
--- a/tests/qemu-iotests/291
+++ b/tests/qemu-iotests/291
@@ -79,7 +79,7 @@ echo
# Only bitmaps from the active layer are copied
$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG"
-$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+_img_info --format-specific
# But we can also merge in bitmaps from other layers. This test is a bit
# contrived to cover more code paths, in reality, you could merge directly
# into b0 without going through tmp
@@ -89,7 +89,7 @@ $QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \
$QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
$QEMU_IMG bitmap --remove --image-opts \
driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
-$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+_img_info --format-specific
echo "Check resulting qcow2 header extensions:"
$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out
index ccfcdc5e35ce..08bfaaaa6b7d 100644
--- a/tests/qemu-iotests/291.out
+++ b/tests/qemu-iotests/291.out
@@ -43,7 +43,7 @@ qemu-img: Format driver 'raw' does not support bitmaps
image: TEST_DIR/t.IMGFMT
file format: IMGFMT
virtual size: 10 MiB (10485760 bytes)
-disk size: 4.39 MiB
+cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
@@ -63,7 +63,7 @@ Format specific information:
image: TEST_DIR/t.IMGFMT
file format: IMGFMT
virtual size: 10 MiB (10485760 bytes)
-disk size: 4.48 MiB
+cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
--
2.27.0
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PULL 00/15] bitmaps patches for 2020-06-09
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
` (14 preceding siblings ...)
2020-06-09 20:52 ` [PULL 15/15] iotests: Fix 291 across more file systems Eric Blake
@ 2020-06-11 17:00 ` Peter Maydell
15 siblings, 0 replies; 20+ messages in thread
From: Peter Maydell @ 2020-06-11 17:00 UTC (permalink / raw)
To: Eric Blake; +Cc: QEMU Developers
On Tue, 9 Jun 2020 at 21:53, Eric Blake <eblake@redhat.com> wrote:
>
> The following changes since commit 31d321c2b3574dcc74e9f6411af06bca6b5d10f4:
>
> Merge remote-tracking branch 'remotes/philmd-gitlab/tags/sparc-next-20200609' into staging (2020-06-09 17:29:47 +0100)
>
> are available in the Git repository at:
>
> https://repo.or.cz/qemu/ericb.git tags/pull-bitmaps-2020-06-09
>
> for you to fetch changes up to adf92f4645ba46726368735916fed763d3e5a09b:
>
> iotests: Fix 291 across more file systems (2020-06-09 15:48:00 -0500)
>
> ----------------------------------------------------------------
> bitmaps patches for 2020-06-09
>
> - documenation fix
> - various improvements to qcow2.py program used in iotests
>
Applied, thanks.
Please update the changelog at https://wiki.qemu.org/ChangeLog/5.1
for any user-visible changes.
-- PMM
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PULL 14/15] qcow2_format.py: dump bitmaps header extension
2020-06-09 20:52 ` [PULL 14/15] qcow2_format.py: dump bitmaps header extension Eric Blake
@ 2020-06-18 13:13 ` Max Reitz
2020-06-18 13:28 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 20+ messages in thread
From: Max Reitz @ 2020-06-18 13:13 UTC (permalink / raw)
To: Eric Blake, qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, Vladimir Sementsov-Ogievskiy,
open list:Block layer core
[-- Attachment #1.1: Type: text/plain, Size: 5425 bytes --]
On 09.06.20 22:52, Eric Blake wrote:
> From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>
> Add class for bitmap extension and dump its fields. Further work is to
> dump bitmap directory.
>
> Test new functionality inside 291 iotest.
Unfortunately, it also breaks 291 with an external data file, which
worked before. (The problems being that an external data file gives you
an additional header extension, and that the bitmap directory offset
changes.)
I think if we want to test testing tools, we have to do that in a
controlled environment where we exactly know what the image is. It
looks to me now as if 291 is not such an environment. Or phrased
differently, we probably shouldn’t test some testing tool in normal
tests that test qemu itself.
If we only test qcow2.py in normal tests, then I don’t think we have to
explicitly test it at all. (Because if you test qcow2.py in a normal
test, and the test breaks, it’s unclear what’s broken. So I think you
might as well forego the qcow2.py test altogether, because if it breaks,
that’ll probably show up in some other test case that uses it.)
In this case here, I can see three things we could do:
First, could just filter out the data file header extension and the
bitmap_directory_offset. But I’m not sure whether that’s the best thing
to do, because it might break with some other obscure IMGOPTS that I
personally never use for the iotests.
I think that if we want a real qcow2.py test somewhere, it should be its
own test. No custom IMGOPTS allowed. So the second idea would be to
move it there, and drop the qcow2.py usage from here.
Or, third, maybe we actually don’t care that much about a real qcow2.py
test, and really want to just *use* (as opposed to “test”) qcow2.py
here. Then we should filter what we really need from its output instead
of dropping what we don’t need.
(We could also disable 291 for external data files, but I don’t really
see why, if the only justification for this addition to it is to test
qcow2.py – which 291 isn’t for.)
Max
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
> Message-Id: <20200606081806.23897-14-vsementsov@virtuozzo.com>
> [eblake: fix iotest output]
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
> tests/qemu-iotests/291 | 4 +++
> tests/qemu-iotests/291.out | 33 +++++++++++++++++++++++
> tests/qemu-iotests/qcow2_format.py | 42 +++++++++++++++++++++++-------
> 3 files changed, 70 insertions(+), 9 deletions(-)
>
> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
> index 3ca83b9cd1f7..e0cffc7cb119 100755
> --- a/tests/qemu-iotests/291
> +++ b/tests/qemu-iotests/291
> @@ -62,6 +62,8 @@ $QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
> $QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1
> $QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2
> $QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
> +echo "Check resulting qcow2 header extensions:"
> +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
>
> echo
> echo "=== Bitmap preservation not possible to non-qcow2 ==="
> @@ -88,6 +90,8 @@ $QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
> $QEMU_IMG bitmap --remove --image-opts \
> driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
> $QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
> +echo "Check resulting qcow2 header extensions:"
> +$PYTHON qcow2.py "$TEST_IMG" dump-header-exts
>
> echo
> echo "=== Check bitmap contents ==="
> diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out
> index 8c62017567e9..ccfcdc5e35ce 100644
> --- a/tests/qemu-iotests/291.out
> +++ b/tests/qemu-iotests/291.out
> @@ -14,6 +14,25 @@ wrote 1048576/1048576 bytes at offset 1048576
> 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> wrote 1048576/1048576 bytes at offset 2097152
> 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +Check resulting qcow2 header extensions:
> +Header extension:
> +magic 0xe2792aca (Backing format)
> +length 5
> +data 'qcow2'
> +
> +Header extension:
> +magic 0x6803f857 (Feature table)
> +length 336
> +data <binary>
> +
> +Header extension:
> +magic 0x23852875 (Bitmaps)
> +length 24
> +nb_bitmaps 2
> +reserved32 0
> +bitmap_directory_size 0x40
> +bitmap_directory_offset 0x510000
> +
>
> === Bitmap preservation not possible to non-qcow2 ===
>
> @@ -65,6 +84,20 @@ Format specific information:
> granularity: 65536
> refcount bits: 16
> corrupt: false
> +Check resulting qcow2 header extensions:
> +Header extension:
> +magic 0x6803f857 (Feature table)
> +length 336
> +data <binary>
> +
> +Header extension:
> +magic 0x23852875 (Bitmaps)
> +length 24
> +nb_bitmaps 3
> +reserved32 0
> +bitmap_directory_size 0x60
> +bitmap_directory_offset 0x520000
> +
>
> === Check bitmap contents ===
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PULL 14/15] qcow2_format.py: dump bitmaps header extension
2020-06-18 13:13 ` Max Reitz
@ 2020-06-18 13:28 ` Vladimir Sementsov-Ogievskiy
2020-06-18 15:09 ` Max Reitz
0 siblings, 1 reply; 20+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-06-18 13:28 UTC (permalink / raw)
To: Max Reitz, Eric Blake, qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, open list:Block layer core
18.06.2020 16:13, Max Reitz wrote:
> On 09.06.20 22:52, Eric Blake wrote:
>> From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>
>> Add class for bitmap extension and dump its fields. Further work is to
>> dump bitmap directory.
>>
>> Test new functionality inside 291 iotest.
>
> Unfortunately, it also breaks 291 with an external data file, which
> worked before. (The problems being that an external data file gives you
> an additional header extension, and that the bitmap directory offset
> changes.)
>
> I think if we want to test testing tools, we have to do that in a
> controlled environment where we exactly know what the image is. It
> looks to me now as if 291 is not such an environment. Or phrased
> differently, we probably shouldn’t test some testing tool in normal
> tests that test qemu itself.
>
> If we only test qcow2.py in normal tests, then I don’t think we have to
> explicitly test it at all. (Because if you test qcow2.py in a normal
> test, and the test breaks, it’s unclear what’s broken. So I think you
> might as well forego the qcow2.py test altogether, because if it breaks,
> that’ll probably show up in some other test case that uses it.)
>
> In this case here, I can see three things we could do:
>
> First, could just filter out the data file header extension and the
> bitmap_directory_offset. But I’m not sure whether that’s the best thing
> to do, because it might break with some other obscure IMGOPTS that I
> personally never use for the iotests.
>
> I think that if we want a real qcow2.py test somewhere, it should be its
> own test. No custom IMGOPTS allowed. So the second idea would be to
> move it there, and drop the qcow2.py usage from here.
>
> Or, third, maybe we actually don’t care that much about a real qcow2.py
> test, and really want to just *use* (as opposed to “test”) qcow2.py
> here. Then we should filter what we really need from its output instead
> of dropping what we don’t need.
>
>
> (We could also disable 291 for external data files, but I don’t really
> see why, if the only justification for this addition to it is to test
> qcow2.py – which 291 isn’t for.)
>
I see your point, agree that the most correct thing is to drop qcow2.py testing from 291, reverting test chunks from this commit. I can send a patch.
I do think, that we'll need some testing for qcow2.py, as we are going to improve it a lot, to be used as separate debugging tool. And we just need a separate iotest for it. Andrey, could you please introduce one, as part of your series?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PULL 14/15] qcow2_format.py: dump bitmaps header extension
2020-06-18 13:28 ` Vladimir Sementsov-Ogievskiy
@ 2020-06-18 15:09 ` Max Reitz
0 siblings, 0 replies; 20+ messages in thread
From: Max Reitz @ 2020-06-18 15:09 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Eric Blake, qemu-devel
Cc: Kevin Wolf, Andrey Shinkevich, open list:Block layer core
[-- Attachment #1.1: Type: text/plain, Size: 2720 bytes --]
On 18.06.20 15:28, Vladimir Sementsov-Ogievskiy wrote:
> 18.06.2020 16:13, Max Reitz wrote:
>> On 09.06.20 22:52, Eric Blake wrote:
>>> From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>
>>> Add class for bitmap extension and dump its fields. Further work is to
>>> dump bitmap directory.
>>>
>>> Test new functionality inside 291 iotest.
>>
>> Unfortunately, it also breaks 291 with an external data file, which
>> worked before. (The problems being that an external data file gives you
>> an additional header extension, and that the bitmap directory offset
>> changes.)
>>
>> I think if we want to test testing tools, we have to do that in a
>> controlled environment where we exactly know what the image is. It
>> looks to me now as if 291 is not such an environment. Or phrased
>> differently, we probably shouldn’t test some testing tool in normal
>> tests that test qemu itself.
>>
>> If we only test qcow2.py in normal tests, then I don’t think we have to
>> explicitly test it at all. (Because if you test qcow2.py in a normal
>> test, and the test breaks, it’s unclear what’s broken. So I think you
>> might as well forego the qcow2.py test altogether, because if it breaks,
>> that’ll probably show up in some other test case that uses it.)
>>
>> In this case here, I can see three things we could do:
>>
>> First, could just filter out the data file header extension and the
>> bitmap_directory_offset. But I’m not sure whether that’s the best thing
>> to do, because it might break with some other obscure IMGOPTS that I
>> personally never use for the iotests.
>>
>> I think that if we want a real qcow2.py test somewhere, it should be its
>> own test. No custom IMGOPTS allowed. So the second idea would be to
>> move it there, and drop the qcow2.py usage from here.
>>
>> Or, third, maybe we actually don’t care that much about a real qcow2.py
>> test, and really want to just *use* (as opposed to “test”) qcow2.py
>> here. Then we should filter what we really need from its output instead
>> of dropping what we don’t need.
>>
>>
>> (We could also disable 291 for external data files, but I don’t really
>> see why, if the only justification for this addition to it is to test
>> qcow2.py – which 291 isn’t for.)
>>
> I see your point, agree that the most correct thing is to drop qcow2.py
> testing from 291, reverting test chunks from this commit. I can send a
> patch.
OK, thanks!
> I do think, that we'll need some testing for qcow2.py, as we are going
> to improve it a lot, to be used as separate debugging tool. And we just
> need a separate iotest for it.
Great.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-06-18 15:11 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-09 20:52 [PULL 00/15] bitmaps patches for 2020-06-09 Eric Blake
2020-06-09 20:52 ` [PULL 01/15] qemu-img: Fix doc typo for 'bitmap' subcommand Eric Blake
2020-06-09 20:52 ` [PULL 02/15] qcow2.py: python style fixes Eric Blake
2020-06-09 20:52 ` [PULL 03/15] qcow2.py: add licensing blurb Eric Blake
2020-06-09 20:52 ` [PULL 04/15] qcow2.py: move qcow2 format classes to separate module Eric Blake
2020-06-09 20:52 ` [PULL 05/15] qcow2_format.py: drop new line printing at end of dump() Eric Blake
2020-06-09 20:52 ` [PULL 06/15] qcow2_format.py: use tuples instead of lists for fields Eric Blake
2020-06-09 20:52 ` [PULL 07/15] qcow2_format.py: use modern string formatting Eric Blake
2020-06-09 20:52 ` [PULL 08/15] qcow2_format.py: use strings to specify c-type of struct fields Eric Blake
2020-06-09 20:52 ` [PULL 09/15] qcow2_format.py: separate generic functionality of structure classes Eric Blake
2020-06-09 20:52 ` [PULL 10/15] qcow2_format.py: add field-formatting class Eric Blake
2020-06-09 20:52 ` [PULL 11/15] qcow2_format.py: QcowHeaderExtension: add dump method Eric Blake
2020-06-09 20:52 ` [PULL 12/15] qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct Eric Blake
2020-06-09 20:52 ` [PULL 13/15] qcow2: QcowHeaderExtension print names for extension magics Eric Blake
2020-06-09 20:52 ` [PULL 14/15] qcow2_format.py: dump bitmaps header extension Eric Blake
2020-06-18 13:13 ` Max Reitz
2020-06-18 13:28 ` Vladimir Sementsov-Ogievskiy
2020-06-18 15:09 ` Max Reitz
2020-06-09 20:52 ` [PULL 15/15] iotests: Fix 291 across more file systems Eric Blake
2020-06-11 17:00 ` [PULL 00/15] bitmaps patches for 2020-06-09 Peter Maydell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.