From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45706) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aImr4-0005Mq-7u for qemu-devel@nongnu.org; Mon, 11 Jan 2016 19:30:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aImr0-00078x-Ub for qemu-devel@nongnu.org; Mon, 11 Jan 2016 19:30:34 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39586) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aImr0-00078k-Lr for qemu-devel@nongnu.org; Mon, 11 Jan 2016 19:30:30 -0500 References: <1452517517-3953-1-git-send-email-vsementsov@virtuozzo.com> From: John Snow Message-ID: <56944923.2060603@redhat.com> Date: Mon, 11 Jan 2016 19:30:27 -0500 MIME-Version: 1.0 In-Reply-To: <1452517517-3953-1-git-send-email-vsementsov@virtuozzo.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vladimir Sementsov-Ogievskiy , qemu-devel@nongnu.org Cc: kwolf@redhat.com, den@openvz.org, famz@redhat.com, stefanha@redhat.com, mreitz@redhat.com On 01/11/2016 08:05 AM, Vladimir Sementsov-Ogievskiy wrote: > The new feature for qcow2: storing bitmaps. >=20 > This patch adds new header extension to qcow2 - Bitmaps Extension. It > provides an ability to store virtual disk related bitmaps in a qcow2 > image. For now there is only one type of such bitmaps: Dirty Tracking > Bitmap, which just tracks virtual disk changes from some moment. >=20 > Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file, > should be stored in this qcow2 file. The size of each bitmap > (considering its granularity) is equal to virtual disk size. >=20 > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- >=20 > v7: >=20 > - Rewordings, grammar. > Max, Eric, John, thank you very much. >=20 > - add last paragraph: remaining bits in bitmap data clusters must be > zero. >=20 > - s/Bitmap Directory/bitmap directory/ and other names like this at > the request of Max. >=20 > v6: >=20 > - reword bitmap_directory_size description > - bitmap type: make 0 reserved > - extra_data_size: resize to 4bytes > Also, I've marked this field as "must be zero". We can always change > it, if we decide allowing managing app to specify any extra data, by > defining some magic value as a top of user extra data.. So, for now > non zeor extra_data_size should be considered as an error. > - swap name and extra_data to give good alignment to extra_data. >=20 >=20 > v5: >=20 > - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of > bitmaps. > - rewordings > - move upper bounds to "Notes about Qemu limits" > - s/should/must somewhere. (but not everywhere) > - move name_size field closer to name itself in bitmap header > - add extra data area to bitmap header > - move bitmap data description to separate section >=20 > docs/specs/qcow2.txt | 172 +++++++++++++++++++++++++++++++++++++++++++= +++++++- > 1 file changed, 171 insertions(+), 1 deletion(-) >=20 > diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt > index 121dfc8..997239d 100644 > --- a/docs/specs/qcow2.txt > +++ b/docs/specs/qcow2.txt > @@ -103,7 +103,18 @@ in the description of a field. > write to an image with unknown auto-clear features= if it > clears the respective bits from this field first. > =20 > - Bits 0-63: Reserved (set to 0) > + Bit 0: Bitmaps extension bit > + This bit indicates consistency for the= bitmaps > + extension data. > + > + It is an error if this bit is set with= out the > + bitmaps extension present. > + > + If the bitmaps extension is present bu= t this > + bit is unset, the bitmaps extension da= ta is > + inconsistent. > + > + Bits 1-63: Reserved (set to 0) > =20 > 96 - 99: refcount_order > Describes the width of a reference count block ent= ry (width > @@ -123,6 +134,7 @@ be stored. Each extension has a structure like the = following: > 0x00000000 - End of the header extension area > 0xE2792ACA - Backing file format name > 0x6803f857 - Feature name table > + 0x23852875 - Bitmaps extension > other - Unknown header extension, can be = safely > ignored > =20 > @@ -166,6 +178,34 @@ the header extension data. Each entry look like th= is: > terminated if it has full length) > =20 > =20 > +=3D=3D Bitmaps extension =3D=3D > + > +The bitmaps extension is an optional header extension. It provides the= ability > +to store bitmaps related to a virtual disk. For now, there is only one= bitmap > +type: the dirty tracking bitmap, which tracks virtual disk changes fro= m some > +point in time. > + > +The data of the extension should be considered consistent only if the > +corresponding auto-clear feature bit is set, see autoclear_features ab= ove. > + > +The fields of the bitmaps extension are: > + > + 0 - 3: nb_bitmaps > + The number of bitmaps contained in the image. Must = be > + greater than or equal to 1. > + > + Note: Qemu currently only supports up to 65535 bitm= aps per > + image. > + > + 4 - 7: bitmap_directory_size > + Size of the bitmap directory in bytes. It is the cu= mulative > + size of all (nb_bitmaps) bitmap headers. > + > + 8 - 15: bitmap_directory_offset > + Offset into the image file at which the bitmap dire= ctory > + starts. Must be aligned to a cluster boundary. > + > + > =3D=3D Host cluster management =3D=3D > =20 > qcow2 manages the allocation of host clusters by maintaining a referen= ce count > @@ -360,3 +400,133 @@ Snapshot table entry: > =20 > variable: Padding to round up the snapshot table entry size = to the > next multiple of 8. > + > + > +=3D=3D Bitmaps =3D=3D > + > +As mentioned above, the bitmaps extension provides the ability to stor= e bitmaps > +related a virtual disk. This section describes how these bitmaps are s= tored. > + > +Note: all bitmaps are related to the virtual disk stored in this image= . > + > +=3D=3D=3D Bitmap directory =3D=3D=3D > + > +Each bitmap saved in the image is described in a bitmap directory entr= y. The > +bitmap directory is a contiguous area in the image file, whose startin= g offset > +and length are given by the header extension fields bitmap_directory_o= ffset and > +bitmap_directory_size. The entries of the bitmap directory have variab= le > +length, depending on the length of the bitmap name and extra data. The= se > +entries are also called bitmap headers. > + > +Structure of a bitmap directory entry: > + > + Byte 0 - 7: bitmap_table_offset > + Offset into the image file at which the bitmap tab= le > + (described below) for the bitmap starts. Must be a= ligned to > + a cluster boundary. > + > + 8 - 11: bitmap_table_size > + Number of entries in the bitmap table of the bitma= p. > + > + 12 - 15: flags > + Bit > + 0: in_use > + The bitmap was not saved correctly and may be > + inconsistent. > + > + 1: auto > + The bitmap must reflect all changes of the vi= rtual > + disk by any application that would write to t= his qcow2 > + file (including writes, snapshot switching, e= tc.). The > + type of this bitmap must be 'dirty tracking b= itmap'. > + > + Bits 2 - 31 are reserved and must be 0. > + > + 16: type > + This field describes the sort of the bitmap. > + Values: > + 1: Dirty tracking bitmap > + > + Values 0, 2 - 255 are reserved. > + > + 17: granularity_bits > + Granularity bits. Valid values: 0 - 63. > + > + Note: Qemu currently doesn't support granularity_b= its > + greater than 31. > + > + Granularity is calculated as > + granularity =3D 1 << granularity_bits > + > + A bitmap's granularity is how many bytes of the im= age > + accounts for one bit of the bitmap. > + > + 18 - 19: name_size > + Size of the bitmap name. Must be non-zero. > + > + Note: Qemu currently doesn't support values greate= r than > + 1023. > + > + 20 - 23: extra_data_size > + Size of type-specific extra data. > + > + For now, as no extra data is defined, extra_data_s= ize is > + reserved and must be zero. > + > + variable: Type-specific extra data for the bitmap. > + > + variable: The name of the bitmap (not null terminated). Must= be > + unique among all bitmap names within the bitmaps e= xtension. > + > + variable: Padding to round up the bitmap directory entry siz= e to the > + next multiple of 8. > + > +=3D=3D=3D Bitmap table =3D=3D=3D > + > +Bitmaps are stored using a one-level structure (as opposed to two-leve= l > +structure like for refcounts and guest clusters mapping) for the mappi= ng of > +bitmap data to host clusters. This structure is called the bitmap tabl= e. > + > +Each bitmap table has a variable size (stored in the bitmap directory = Entry) > +and may use multiple clusters, however, it must be contiguous in the i= mage > +file. > + > +Structure of a bitmap table entry: > + > + Bit 0: Reserved and must be zero if bits 9 - 55 are non-z= ero. > + If bits 9 - 55 are zero: > + 0: Cluster should be read as all zeros. > + 1: Cluster should be read as all ones. > + > + 1 - 8: Reserved and must be zero. > + > + 9 - 55: Bits 9 - 55 of the host cluster offset. Must be al= igned to > + a cluster boundary. If the offset is 0, the cluste= r is > + unallocated; in that case, bit 0 determines how th= is > + cluster should be treated when read from. > + > + 56 - 63: Reserved and must be zero. > + > +=3D=3D=3D Bitmap data =3D=3D=3D > + > +As noted above, bitmap data is stored in separate clusters, described = by the > +bitmap table. Given an offset (in bytes) into the bitmap data, the off= set into > +the image file can be obtained as follows: > + > + image_offset =3D > + bitmap_table[bitmap_data_offset / cluster_size] + > + (bitmap_data_offset % cluster_size) > + > +This offset is not defined if bits 9 - 55 of bitmap table entry are ze= ro (see > +above). > + > +Given an offset byte_nr into the virtual disk and the bitmap's granula= rity, the > +bit offset into the bitmap can be calculated like this: > + > + bit_offset =3D > + image_offset(byte_nr / granularity / 8) * 8 + > + (byte_nr / granularity) % 8 > + > +If the size of the bitmap data is not a multiply of cluster size then = the last > +cluster of the bitmap data contains some unused tail bits. These bits = must be > +zero. >=20 s/multiply/multiple/. Looks good otherwise, pending the results of our discussion about what to do about the extra_data_size / type-specific data fields. Thanks! --=20 =97js