Re: Domain Save Image Format proposal (draft B)

[Ian Campbell <Ian.Campbell@citrix.com>]

On Tue, 2014-02-11 at 11:40 +0000, David Vrabel wrote:
> On 11/02/14 09:30, Ian Campbell wrote:
> > On Mon, 2014-02-10 at 17:20 +0000, David Vrabel wrote:
> >> 
> >> Tools supporting version _V_ of the specification shall always save
> >> images using version _V_.  Tools shall support restoring from version
> >> _V_ and version _V_ - 1.
> > 
> > This isn't quite right since it is in terms of image format version and
> > not Xen version (unless the two are to be linked somehow). The Xen
> > project supports migration from version X-1 to version X (where X is the
> > Xen version). It's not inconceivable that the image format version
> > wouldn't change over Xen releases.
> 
> I was expecting it to be only necessary to bump the format version with
> each Xen x.y release but I think you're right.  It may be needed to bump
> the version of a x.y.z release.

We try very hard to avoid that (i.e. changing the save format in a
stable branch) actually. 

What I meant was:
	Xen = 4.5, V = 1
	Xen = 4.6, V = 1 (nothing changed in the save file spec for 4.5)
	Xen = 4.7, V = 2

The spec with it's current wording about V would seem to say the Xen 4.7
must support migration from Xen 4.5, which is not the case.

For HVM support there are going to be (at least) two additional record
types "opaque blob of Xen state" and "opaque blob of qemu state". It is
very unlikely that anyone will remember to bump V in the tools when
changing those.

I think the solution is to also include the Xen version in the header,
as well as the "file format" version V. You could do this as part of the
header of those opaque blobs, but it might be better in the actual
overall headers (/as well).

I've no idea how to deal with the qemu blob -- but I think qemu upstream
has been working to make cross version migration work better, so maybe
it isn't a problem any more.

> 
> >> --------------------------------------------------------------------
> >> Field       Description
> >> ----------- --------------------------------------------------------
> >> marker      0xFFFFFFFFFFFFFFFF.
> >>
> >> id          0x58454E46 ("XENF" in ASCII).
> >>
> >> version     0x00000001.  The version of this specification.
> >>
> >> options     bit 0: Endianness.  0 = little-endian, 1 = big-endian.
> > 
> > Couldn't we just specify that things are in a specific endianness
> > related to the header's arch field?
> > 
> > I appreciate the desire to make the format endian neutral and explicit
> > about which is in use but (apart from the header) why would you ever
> > want to go non-native endian for a given arch?
> 
> I am anticipating bi-endian architectures which could mean (for example)
> migrating a little-endian guest from a little-endian host to a
> big-endian host.

It's possible that this would end up looking like a totally separate
arch anyway (cf "armbe"), not just at the save format level, but at the
hypercall and PVIO layers too.

> I would prefer to retain this bit, but I think we can specify that
> certain architectures always use a specific endianness so initially we
> wouldn't need to support anything other than the native endianness
> (little-endian on both x86 and ARM).

I think that makes sense initially.

It's always a bit tricky with a spec to separate the notion of what is
possible within the spec from what you actually intend to implement now.

Perhaps a reasonable compromise is to document a requirement that savers
populate this field accurately but only require restorers to support
their nativeness (leaving anything further as optional).

> >> --------------------------------------------------------------------
> >> Field       Description
> >> ----------- --------------------------------------------------------
> >> arch        0x0000: Reserved.
> >>
> >>             0x0001: x86.
> >>
> >>             0x0002: ARM.
> >>
> >> type        0x0000: Reserved.
> >>
> >>             0x0001: x86 PV.
> >>
> >>             0x0002 - 0xFFFF: Reserved.
> > 
> > Is the type field per-arch? i.e. if arch=0x0002 can we use type = 0x0001
> > for ARM domains?
> 
> I think it would be best to avoid reusing types for different
> architectures -- it's not like we're going to be short on types.

Any reason not to combine the arch+type into a single field then?

> >> --------------------------------------------------------------------
> >> Field       Description
> >> ----------- --------------------------------------------------------
> >> count       Number of pages described in this record.
> >>
> >> pfn         An array of count PFNs. Bits 63-60 contain
> >>             the XEN\_DOMCTL\_PFINFO_* value for that PFN.
> > 
> > Now might be a good time to remove this intertwining? I suppose 60-bits
> > is a lot of pfn's, but if the VMs address space is sparse it isn't out
> > of the question.
> 
> I don't think we want to consider systems with > 64 bits of address
> space so 60-bits is more than enough for PFNs.

Is it? What about systems with 61..63 bits of address space?

Ian.