From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50830) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fQHvp-0003oe-G1 for qemu-devel@nongnu.org; Tue, 05 Jun 2018 15:47:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fQHvo-00017b-25 for qemu-devel@nongnu.org; Tue, 05 Jun 2018 15:47:49 -0400 Date: Tue, 5 Jun 2018 22:47:40 +0300 From: "Michael S. Tsirkin" Message-ID: <20180605223934-mutt-send-email-mst@kernel.org> References: <20180524113251.GB4660@redhat.com> <20180528183058.GG2209@redhat.com> <20180528183833.GJ4580@localhost.localdomain> <20180528212054.GH2209@redhat.com> <20180528212510.GC4660@redhat.com> <20180529064415.GA4756@localhost.localdomain> <2b3eef00-f326-c1e6-0e4b-b7602646eec4@redhat.com> <20180605092159.GA2544@work-vm> <20180605190324.GA11372@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180605190324.GA11372@localhost.localdomain> Subject: Re: [Qemu-devel] storing machine data in qcow images? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Eduardo Habkost Cc: "Dr. David Alan Gilbert" , Max Reitz , Kevin Wolf , qemu-block@nongnu.org, qemu-devel@nongnu.org, "Richard W.M. Jones" , stefanha@redhat.com On Tue, Jun 05, 2018 at 04:03:24PM -0300, Eduardo Habkost wrote: > On Tue, Jun 05, 2018 at 10:21:59AM +0100, Dr. David Alan Gilbert wrote: > > > > > > This seems to have fizzled out because of a lack of a concrete proposal; > > so here is one based on a reply to Max's post: > > > > * Max Reitz (mreitz@redhat.com) wrote: > > > > > > > > > The original problem was that you need to supply a machine type to qemu, > > > and that multiple common architectures now have multiple machine types > > > and not necessarily all work with a single image. So far so good, but I > > > have two issues here already: > > > > > > (1) How is qemu supposed to interpret that information? If it's stored > > > in the image file, I don't see a nice way of retrieving it before the > > > machine is initialized, at least not with qemu's current architecture. > > > > > > > > > (2) Again, I personally just really don't like saving such information > > > in a disk image. One actual argument I can bring up for that distaste > > > is this: Suppose, you have multiple images attached to your VM. Now the > > > VM wants to store the machine type. Where does it go? Into all of > > > them? > > > > > > > > > So I think if we decide to store the machine type, that is kind of a > > > slippery slope and then there are good arguments for storing even more > > > configuration options in the file, too. But I really, really don't like > > > that. > > > > > > > > > For another, how do we store the data? key-value seems wrong if we want > > > to store everything. JSON might be fine. But eventually we just want > > > basically a qemu configuration file in there, I would think (which may > > > support JSON at some point?). So basically we would store the data as > > > a binary blob and let the rest of qemu do its thing with it. But then > > > please tell me why I fought so valiantly against storing random bitmaps > > > in qcow2 files. I hate the idea of making qcow2 a random archive > > > format. We have tar for that. > > > > > > > > > tl;dr: I really don't get why it's so hard to supply a config file along > > > with a qcow2 image. Is it so hard for people to realize that a VM does > > > not only consist of a disk? > > > > Yes! Because in many cases that's all it needs, and it's ready to run > > with no unpacking. > > > > I think we should have: > > > > -------------------------------------------------------------- > > Layer 0: > > QCOW provides a way to store a single string of arbitrary (but > > limited?) length. > > QCOW provides a way to replace the string by a new string. > > The original or the new string will be stored after that; > > never some mix. > > Where a file 'b' has a backing file 'a', 'b' inherits the > > string from 'a' unless 'b' has it's own string. > > Snapshots inherit their string from the main unless they have > > their own string. > > > > Layer 1: > > The string shall always be a JSON 'object'; i.e. of the form > > { "something": ... , "more": ... } > > > > The key strings shall be non-null and non-empty and shall > > be unique. > > > > I'd prefer layer 0+1 to: > > 1) Allow multiple entries to be stored (implemented by layer 1 > in this proposal) > 2) Identify each entry with a name (implemented by layer 1 in > this proposal) > 3) Allow arbitrary binary data to be stored on an entry > (not possible with the JSON-based proposal, because JSON > strings are not blobs, but Unicode strings). > 4) Make it easy to replace only one entry while keeping others > intact (not the case here, if all entries are stored in the > same JSON string) > > I think it would be simpler if layer 0 simply provided a list of > names/value pairs, where names are ascii strings, and values are > binary data[1]. It would make layer 1 unnecessary, and allow (3) > and (4) to happen. > > [1] In other words, Rich's proposal of "named blobs": > https://www.mail-archive.com/qemu-block@nongnu.org/msg37856.html I think simple is beautiful, too. But assuming they really are binary how are blobs encoded? Did binary really mean UTF-8 here? > > > Layer 2: > > '.'s in the key string shall indicate hierarchy > > > > Key strings shall be listed in qemu's > > docs/specs/qcow-keys.rst > > > > that shall indicate their meaning and the meaning and > > valid formatting of the value associated with the, > > > > Key strings shall start with either: > > qemu. in which case they must be listed in a file in > > the qemu source tree > > > > a reverse dotted name unique to the submitter, they may > > be listed in the same file in the source tree, e.g. > > com.redhat. > > > > Based on this proposal for layer 2, it looks like you expect the > number of keys used on layer 1 to become large. > > I would prefer a solution that expects a very small set of keys > for layer 0+1, and point to other specifications of how the blob > can be interpreted for each key. This way we can experiment with > different solutions for layers 2-4, instead of deciding on a > specific format like JSON. > > > > Layer 3: > > QEMU shall, for a given qcow2 file be able to dump the > > key values. > > > > Layer 4: > > On creating a VM by importing a qcow2, a management layer > > shall inspect the key/values to influence the configuration > > of the VM created. Where it imports multiple qcow2's it > > shall inspect all the files and flag disagreements. > > > > Management layers shall, on creating a qcow2 shall set the > > keys based on the VM the qcow2 is created for. If the qcow2 > > is created as an additional disk for an exisitng VM it's > > fine to leave the string empty (e.g. for a data disk). > > > > -------------------------------------------------------------- > > > > > > Some reasoning: > > a) I've avoided the problem of when QEMU interprets the value > > by ignoring it and giving it to management layers at the point > > of VM import. > > b) I hate JSON, but there again nailing down a fixed format > > seems easiest and it makes the job of QCOW easy - a single > > string. > > (I would suggest in layer2 that the keys are sorted, but > > that's a pain to do in some json creators) > > c) Forcing the registry of keys might avoid silly duplication. > > We can but hope. > > d) I've not said it's a libvirt XML file since that seems > > a bit prescriptive. > > > > Some initial suggested keys: > > > > "qemu.machine-types": [ "q35", "i440fx" ] > > "qemu.min-ram-MB": 1024 > > > > > > Dave > > > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > > -- > Eduardo