From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42791) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aO7m4-0005T2-Lk for qemu-devel@nongnu.org; Tue, 26 Jan 2016 12:51:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aO7m2-0007pP-Oo for qemu-devel@nongnu.org; Tue, 26 Jan 2016 12:51:28 -0500 References: <1453804705-7205-1-git-send-email-famz@redhat.com> <1453804705-7205-2-git-send-email-famz@redhat.com> From: Eric Blake Message-ID: <56A7B216.6070108@redhat.com> Date: Tue, 26 Jan 2016 10:51:18 -0700 MIME-Version: 1.0 In-Reply-To: <1453804705-7205-2-git-send-email-famz@redhat.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="iGa8wDm2KRwNOKEJkrA3E8Xn5EeBWEX2E" Subject: Re: [Qemu-devel] [RFC PATCH 01/16] doc: Add QBM format specification List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng , qemu-devel@nongnu.org Cc: Kevin Wolf , qemu-block@nongnu.org, Markus Armbruster , mreitz@redhat.com, vsementsov@parallels.com, Stefan Hajnoczi , jsnow@redhat.com This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --iGa8wDm2KRwNOKEJkrA3E8Xn5EeBWEX2E Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 01/26/2016 03:38 AM, Fam Zheng wrote: > Signed-off-by: Fam Zheng > --- > docs/specs/qbm.md | 118 ++++++++++++++++++++++++++++++++++++++++++++++= ++++++++ > 1 file changed, 118 insertions(+) > create mode 100644 docs/specs/qbm.md >=20 > diff --git a/docs/specs/qbm.md b/docs/specs/qbm.md > new file mode 100644 > index 0000000..b91910b > --- /dev/null > +++ b/docs/specs/qbm.md > @@ -0,0 +1,118 @@ > +QEMU Block Bitmap (QBM) > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D No explicit copyright mention means that this document is GPLv2+ by default. I don't know if any 3rd-party implementation trying to use this spec would object to that, or if a looser license is desirable here (I don't personally care, but just raising the point). > + > +QBM is a multi-file disk format to allow storing persistent block bitm= aps along > +with the tracked data image. A QBM image includes one json descriptor= file, > +one data image, one or more bitmap files that describe the block dirty= status s/one or more/and one or more/ Must it have block dirty status bitmaps, or can you have a QBM image with just an allocation bitmap? > +of the data image. > + > +The json file describes the structure of the image. The structure of t= he json > +descriptor file is: Please mention that the file must be valid JSON per RFC 7159. Probably also worth requiring that the file is a text file (ends in a newline). > + > + QBM-JSON-FILE :=3D { "QBM": DESC-JSON } > + > + DESC-JSON :=3D { "version": 1, > + "image": IMAGE, > + "BITMAPS": BITMAPS s/"BITMAPS"/"bitmaps"/ See my thoughts below on whether this is the ideal top-level structure. > + } > + > +Fields in the top level json dictionary are: > + > +@version: An integer which must be 1. > +@image: A dictionary in IMAGE schema, as described later. It provides = the > + information of the data image where user data is stored. Its f= ormat is > + documented in the "IMAGE schema" section. > +@bitmaps: A dictionary that describes one ore more bitmap files. The k= eys into > + the dictionary are the names of bitmap, which must be string= s, and > + each value is a dictionary describing the information of the= bitmap, > + as documented below in the "BITMAP schema" section. Making 'bitmaps' be a dictionary means that we now have keys that might not be an identifier. Although this is valid JSON, it may trip up some tools. Would it be better to make 'bitmaps' be a list of dictionaries, where each dictionary has a 'name':'value' member, so that bitmap names that are not a (C, Python, whatever) identifier are still valid? > + > +=3D=3D=3D IMAGE schema =3D=3D=3D > + > +An IMAGE records the information of an image (such as a data image or = a backing > +file). It has following fields: I liked how you showed DESC-JSON :=3D for the top level; should you do th= e same here for IMAGE? > + > +@file: The file name string of the referenced image. If it's a relativ= e path, > + the file should be found relative to the descriptor file's > + location. Does that mean we'll have to use 'json:...' encoding for representing network resources? Should we instead reuse some of the qapi/block-core.json representation of a block device? > +@format: The format string of the file. Nice that format is mandatory. Do we want to call out a finite list of supported formats, or leave it open-ended in this spec? > + > +=3D=3D=3D BITMAP schema =3D=3D=3D > + > +A BITMAP dictionary records the information of a bitmap (such as a dir= ty bitmap > +or a block allocation status bitmap). It has following mandatory field= s: > + > +@file: The name of the bitmap file. The bitmap file is in little endia= n, both s/in // > + byte-order-wise and bit-order-wise, which means the LSB in the = byte 0 > + corresponds to the first sectors. s/the byte 0/the first byte/ Again, should we be reusing something from qapi/block-core.json, to allow network devices with more structure than just 'json:...' naming? > +@granularity-bytes: How many bytes of data does one bit in the bitmap = track. > + This value must be a power of 2 and no less than 5= 12. > +@type: The type of the bitmap. Currently only "dirty" and "allocation= " are > + supported. > + "dirty" indicates a block dirty bitmap; "allocation" indicates = a > + allocation status bitmap. There must be at most one "allocation= " bitmap. > + > +If the type of the bitmap is "allocation", an extra field "backing" is= also > +accepted: > + > +@backing: a dictionary as specified in the IMAGE schema. It can be use= d to > + adding a backing file to raw image. s/adding/add/ As promised above, would an alternative representation be any better? That is, I'm trying to see if I could write qapi to describe the structure you've presented here, and I fell short when it comes to naming a particular bitmap. Also, since there is at most one 'type':'allocation' member of 'bitmaps', I wonder if separating it out would make it easier to locate. Here's my proposal for an alternative schema, written in qapi: { 'struct': 'Other', 'data': { ...however we describe a network file... }= } { 'alternate': 'File', 'data': { 'file': 'str', 'struct': 'Other' } } { 'struct': 'Bitmap', 'data': { 'name':'str', 'file': 'File', 'granularity-bytes':'int' } } { 'struct': 'AllocationBitmap', 'base': 'Bitmap', 'data': { '*backing': 'Image' } } { 'struct': 'Image', 'data': { 'file': 'File', 'format': 'str' # would an enum be better? } } { 'struct': 'Desc', 'data': { 'version': 'int', 'image': 'Image', '*allocation': 'AllocationBitmap', 'dirty': [ 'AllocationBitmap' ] } } { 'struct': 'QBM', 'data': { 'QBM': 'Desc' } } where the json description file must consist of a single 'QBM' qapi struct, and where the use of a QAPI alternate type 'File' allows us to specify either a file name or a formal structure for describing a network resource. Below, I'll rewrite your example with my schema... > + > + > +=3D=3D=3D Extended fields =3D=3D=3D > + > +Implementations are allowed to extend the format schema by inserting a= dditinoal s/additinoal/additional/ > +members into above dictionaries, with key names that starts with eithe= r > +an "ext-hard-" or an "ext-soft-" prefix. Should we be more like qcow2 and have a third category of auto-clear keys (if you don't recognize the key, remove it upon editing the file, but reading is okay)? Feature negotiation via this approach requires reading every member of 'bitmaps' (well, we have to do that anyway to parse the full JSON structure); would it be any better to have an up-front section in the top level that describes what features are in use, rather than requiring all new features to use the 'ext-' namespace? Should we require hard failure on any key whose name is not recognized (other than the weirdness of your proposal having the keys of 'bitmaps' be user-supplied names)? > + > +Extended fields prefixed with "ext-soft-" are optional and can be igno= red by > +parsers if they do not support it; fields starting with "ext-hard-" ar= e > +mandatory and cannot be ignored, a parser should not proceed parsing t= he image > +if it does not support it. Is it really the entire image invalidated if an extension is tied to a particular bitmap, or only that bitmap? > + > +It is strongly recommended that the application names are also include= d in the > +extention name string, such as "ext-hard-qemu-", if the effect or s/extention/extension/ > +interpretation of the field is local to a specific application. > + > +For example, QEMU can implement a "checksum" feature to make sure no f= iles > +referred to by the json descriptor are modified inconsistently, by add= ing > +"ext-soft-qemu-checksum" fields in "image" and "bitmaps" descriptions,= like in > +the json text found below. If an extension proves to be useful, how do we standardize it later? Will it always have to carry the 'ext-' prefix? You said that soft extensions can be ignored on parse - but if we write to the file, couldn't we possibly be invalidating the contents of the extension field, and not leaving a breadcrumb for the future reader that understands the extension to know that we messed it up? I think an auto-clear feature would be useful (preserve the checksum field, but clear the associated auto-clear so that the newer reader knows that the checksum has to be checked). Should we be thinking about a write lock extension (similar to the current thread on qcow2 write locks), to make it less likely that two writers will be modifying the descriptor, image file, and/or bitmaps at the same time? > + > +=3D=3D=3D QBM descriptor file example =3D=3D=3D > + > +This is the content of a QBM image's json descriptor file, which conta= ins a > +data image (data.img), and three bitmaps, out of which the "allocation= " bitmap > +associates a backing file to this image (base.img). > + > +{ "QBM": { > + "version": 1, > + "image": { > + "file": "data.img", > + "format": "raw" > + "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8", Comma on wrong row. > + }, > + "bitmaps": { > + "0": { > + "file": "bitmap0.bin", > + "granularity-bytes": 512, > + "type": "dirty" > + }, > + "1": { > + "file": "bitmap1.bin", > + "granularity-bytes": 4096, > + "type": "dirty" > + }, > + "2": { > + "file": "bitmap3.bin", > + "granularity-bytes": 4096, > + "type": "allocation" Missing comma > + "backing": { > + "file": "base.img", > + "format": "raw" > + "ext-soft-qemu-checksum": "fcad1f672b2fb19948405e7a1a1= 8c2a7", Comma on wrong row > + }, No trailing commas in JSON Your approach means that the same bitmap cannot be used for both 'dirty' and 'allocation' (because all entries in the 'bitmaps' dictionary must have distinct names). (Although nothing stops two bitmap entries from naming the same 'file'.) > + } > + } > +} } So, rewriting to my schema above, this would be: { "QBM": { "version": 1, "image": { "file": "data.img", "format": "raw", "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8" }, "allocation": { "name": "2", "file": "bitmap3.bin", "granularity-bytes": 4096, "backing": { "file": "base.img", "format": "raw" "ext-soft-qemu-checksum": "fcad1f672b2fb19948405e7a1a18c2a7" } }, "bitmaps": [ { "name": "0", "file": "bitmap0.bin", "granularity-bytes": 512 }, { "name": "1", "file": "bitmap1.bin", "granularity-bytes": 4096 } ] } } > + >=20 Awkward to end in a blank line. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --iGa8wDm2KRwNOKEJkrA3E8Xn5EeBWEX2E Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJWp7IXAAoJEKeha0olJ0NqFYkH/j8HpWjReq5ixtkt0aPsijxO 9tOROYTvZXcZlwzN3Se9ZHGs51Ok0C5BwlaC87msxFmndqfrf8lfJ23RB72/XTql NPNaaURaed9XcediDaosVXeS0veDKQ5hizzBqHCF97c3TX8D/eOWduSQN+4sc4Fu k5U/1Y6kX4eC7P9sVgKE0bE+Zp4sxfXN1Lirg6dyHSfQSUQxk5yzcmMpHza5NkQX bfti9ZUrziLoOB/j2bE7iPBRS5X3x1Qw7bJtgPxbnKJPlwOzJRXgJBCpLokOjEKx Au+F4ORkKd7OtM8uFXl+aj455EXPdaiUYUcGRXubdspTScV4BI7Lcz9eOomIBio= =jAmy -----END PGP SIGNATURE----- --iGa8wDm2KRwNOKEJkrA3E8Xn5EeBWEX2E--