[Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

I was doing a bit of investigation around how different hypervisors
handle the VM Generation ID feature.  QEMU's behaviour seems quite
strange, I wonder if this is a bug or expected?

(1) I booted a Windows 2016 VM with:

  qemu-system-x86_64 -M pc,accel=kvm -m 2G -hda w2k16-mincore.img \
      -device vmgenid,guid=01020304-0506-0708-090a-0b0c0d0e0f00,id=vmgenid0

(2) Inside the guest I used the VMGENID.EXE program from:

  https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3
  https://docs.microsoft.com/en-gb/windows/desktop/HyperV_v2/virtual-machine-generation-identifier

Note this is self-compiled using mingw64-g++ (not using Visual Studio
which I don't have available), but I don't believe that could have
caused the problem.

(3) The program prints:

  VmCounterValue: 708050601020304:f0e0d0c0b0a09

To make it easier to see, this is the same number but zero-extended:

  VmCounterValue: 07 08 05 06 01 02 03 04 : 00 0f 0e 0d 0c 0b 0a 09
                  \________ LOW ________/   \_______ HIGH _______/
                            WORD                     WORD

As you can see it looks like there is no clear relationship between
the order of the bytes in the guid= parameter and the order that they
are seen by Windows.

BTW if you want to try to reproduce this you will need to use
Windows 2012 R2 or above.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Thu, Jul 05, 2018 at 01:39:29PM +0100, Richard W.M. Jones wrote:
> I was doing a bit of investigation around how different hypervisors
> handle the VM Generation ID feature.  QEMU's behaviour seems quite
> strange, I wonder if this is a bug or expected?
> 
> (1) I booted a Windows 2016 VM with:
> 
>   qemu-system-x86_64 -M pc,accel=kvm -m 2G -hda w2k16-mincore.img \
>       -device vmgenid,guid=01020304-0506-0708-090a-0b0c0d0e0f00,id=vmgenid0
> 
> (2) Inside the guest I used the VMGENID.EXE program from:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3
>   https://docs.microsoft.com/en-gb/windows/desktop/HyperV_v2/virtual-machine-generation-identifier
> 
> Note this is self-compiled using mingw64-g++ (not using Visual Studio
> which I don't have available), but I don't believe that could have
> caused the problem.
> 
> (3) The program prints:
> 
>   VmCounterValue: 708050601020304:f0e0d0c0b0a09
> 
> To make it easier to see, this is the same number but zero-extended:
> 
>   VmCounterValue: 07 08 05 06 01 02 03 04 : 00 0f 0e 0d 0c 0b 0a 09
>                   \________ LOW ________/   \_______ HIGH _______/
>                             WORD                     WORD
> 
> As you can see it looks like there is no clear relationship between
> the order of the bytes in the guid= parameter and the order that they
> are seen by Windows.

OK after examining util/uuid.c and the qemu_uuid_bswap function,
I sort of see what's going on here.

FWIW other hypervisors seem to store these as two 64 bit integers.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Michael S. Tsirkin]

On Thu, Jul 05, 2018 at 01:52:31PM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 01:39:29PM +0100, Richard W.M. Jones wrote:
> > I was doing a bit of investigation around how different hypervisors
> > handle the VM Generation ID feature.  QEMU's behaviour seems quite
> > strange, I wonder if this is a bug or expected?
> > 
> > (1) I booted a Windows 2016 VM with:
> > 
> >   qemu-system-x86_64 -M pc,accel=kvm -m 2G -hda w2k16-mincore.img \
> >       -device vmgenid,guid=01020304-0506-0708-090a-0b0c0d0e0f00,id=vmgenid0
> > 
> > (2) Inside the guest I used the VMGENID.EXE program from:
> > 
> >   https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3
> >   https://docs.microsoft.com/en-gb/windows/desktop/HyperV_v2/virtual-machine-generation-identifier
> > 
> > Note this is self-compiled using mingw64-g++ (not using Visual Studio
> > which I don't have available), but I don't believe that could have
> > caused the problem.
> > 
> > (3) The program prints:
> > 
> >   VmCounterValue: 708050601020304:f0e0d0c0b0a09
> > 
> > To make it easier to see, this is the same number but zero-extended:
> > 
> >   VmCounterValue: 07 08 05 06 01 02 03 04 : 00 0f 0e 0d 0c 0b 0a 09
> >                   \________ LOW ________/   \_______ HIGH _______/
> >                             WORD                     WORD
> > 
> > As you can see it looks like there is no clear relationship between
> > the order of the bytes in the guid= parameter and the order that they
> > are seen by Windows.
> 
> OK after examining util/uuid.c and the qemu_uuid_bswap function,
> I sort of see what's going on here.
> 
> FWIW other hypervisors seem to store these as two 64 bit integers.
> 
> Rich.

What do others do, and what is, in your opinion, the correct behaviour here?


> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Thu, Jul 05, 2018 at 04:54:16PM +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 05, 2018 at 01:52:31PM +0100, Richard W.M. Jones wrote:
> > On Thu, Jul 05, 2018 at 01:39:29PM +0100, Richard W.M. Jones wrote:
> > > I was doing a bit of investigation around how different hypervisors
> > > handle the VM Generation ID feature.  QEMU's behaviour seems quite
> > > strange, I wonder if this is a bug or expected?
> > > 
> > > (1) I booted a Windows 2016 VM with:
> > > 
> > >   qemu-system-x86_64 -M pc,accel=kvm -m 2G -hda w2k16-mincore.img \
> > >       -device vmgenid,guid=01020304-0506-0708-090a-0b0c0d0e0f00,id=vmgenid0
> > > 
> > > (2) Inside the guest I used the VMGENID.EXE program from:
> > > 
> > >   https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3
> > >   https://docs.microsoft.com/en-gb/windows/desktop/HyperV_v2/virtual-machine-generation-identifier
> > > 
> > > Note this is self-compiled using mingw64-g++ (not using Visual Studio
> > > which I don't have available), but I don't believe that could have
> > > caused the problem.
> > > 
> > > (3) The program prints:
> > > 
> > >   VmCounterValue: 708050601020304:f0e0d0c0b0a09
> > > 
> > > To make it easier to see, this is the same number but zero-extended:
> > > 
> > >   VmCounterValue: 07 08 05 06 01 02 03 04 : 00 0f 0e 0d 0c 0b 0a 09
> > >                   \________ LOW ________/   \_______ HIGH _______/
> > >                             WORD                     WORD
> > > 
> > > As you can see it looks like there is no clear relationship between
> > > the order of the bytes in the guid= parameter and the order that they
> > > are seen by Windows.
> > 
> > OK after examining util/uuid.c and the qemu_uuid_bswap function,
> > I sort of see what's going on here.
> > 
> > FWIW other hypervisors seem to store these as two 64 bit integers.
> 
> What do others do, and what is, in your opinion, the correct behaviour here?

I don't want to claim any particular behaviour is correct.  However ...

VMware represents these internally as two signed 64 bit integers, eg:

  vm.genid = "-570734802784577186"
  vm.genidx = "-5042519231342505152"

I am still trying to get verification, but I believe the first is the
low 64 bit word and the second is the high 64 bit word.

Hyper-V's implementation is completely opaque, but their documentation
claims that the VM Generation ID is "a 128-bit, cryptographically
random integer value identifier" whatever that means.

As described in the link above, Microsoft provides some example code
for reading the VM Gen ID from a Windows guest.  In this code it is
clearly represented as a pair of unsigned 64 bit integers.

Windows >= 2012 R2 provides a device driver which has an API which
also reads a struct containing two unsigned 64 bit ints.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Laszlo Ersek]

Hi Rich,

On 07/05/18 14:52, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 01:39:29PM +0100, Richard W.M. Jones wrote:
>> I was doing a bit of investigation around how different hypervisors
>> handle the VM Generation ID feature.  QEMU's behaviour seems quite
>> strange, I wonder if this is a bug or expected?
>>
>> (1) I booted a Windows 2016 VM with:
>>
>>   qemu-system-x86_64 -M pc,accel=kvm -m 2G -hda w2k16-mincore.img \
>>       -device vmgenid,guid=01020304-0506-0708-090a-0b0c0d0e0f00,id=vmgenid0
>>
>> (2) Inside the guest I used the VMGENID.EXE program from:
>>
>>   https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3
>>   https://docs.microsoft.com/en-gb/windows/desktop/HyperV_v2/virtual-machine-generation-identifier
>>
>> Note this is self-compiled using mingw64-g++ (not using Visual Studio
>> which I don't have available), but I don't believe that could have
>> caused the problem.
>>
>> (3) The program prints:
>>
>>   VmCounterValue: 708050601020304:f0e0d0c0b0a09
>>
>> To make it easier to see, this is the same number but zero-extended:
>>
>>   VmCounterValue: 07 08 05 06 01 02 03 04 : 00 0f 0e 0d 0c 0b 0a 09
>>                   \________ LOW ________/   \_______ HIGH _______/
>>                             WORD                     WORD
>>
>> As you can see it looks like there is no clear relationship between
>> the order of the bytes in the guid= parameter and the order that they
>> are seen by Windows.
>
> OK after examining util/uuid.c and the qemu_uuid_bswap function,
> I sort of see what's going on here.
>
> FWIW other hypervisors seem to store these as two 64 bit integers.

This is a bit tricky. Please refer to the following:

* https://bugzilla.redhat.com/show_bug.cgi?id=1118834#c21
* http://mid.mail-archive.com/c052d05e-71a5-1a6a-f34f-17d14167c2f6@redhat.com

in particular steps (8) + (9), and (15) + (16).

The QEMU device model takes the UUID (GUID) in textual representation
and stores it in guest memory in little endian binary UUID
representation. Historically, components of the UUID used to be encoded
in big endian, when represented in binary; however, as e.g. the SMBIOS
spec states (in "7.2.1 System -- UUID"), Microsoft uses little endian
encoding, for representing those fields of the UUID, in binary, that are
not simple byte arrays. (This has BTW leaked into other domains, for
example ACPI and UEFI).

In turn, the test program that you used didn't print the binary value in
either UUID format or raw byte dump format -- instead, it interpreted
the first 8 bytes as a little-endian uint64_t, and the second 8 bytes as
another uint64_t.

Applied to your example, we have:

  01020304-0506-0708-090a-0b0c0d0e0f00

  time_low:                  0x01020304 (DWORD)
  time_mid:                  0x0506 (WORD)
  time_hi_and_version:       0x0708 (WORD)
  clock_seq_hi_and_reserved: 0x09 (BYTE)
  clock_seq_low:             0x0a (BYTE)
  Node:                      0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x00 (BYTE[6])

The DWORD and WORD fields are sensitive to endianness. When using
little-endian, the full raw representation, and the uint64_t
interpretations will be:

  time_low                time_mid    time_hi_and_version  clock_seq_hi_and_reserved  clock_seq_low         Node
  ----------------------  ----------  -------------------  -------------------------  --------------------  ----------------------------------
  0x04, 0x03, 0x02, 0x01, 0x06, 0x05, 0x08, 0x07,          0x09,                      0x0a,                 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x00
  -------------------------------------------------------  -----------------------------------------------------------------------------------
  uint64_t: 0x0708050601020304                             uint64_t: 0x000f0e0d0c0b0a09

QEMU does the right thing. If other hypervisors don't do this -- while
still taking and displaying the value in UUID / GUID textual format --,
they are wrong. The VMGENID spec from Microsoft
<http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
"GUID".

Thanks,
Laszlo

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Thu, Jul 05, 2018 at 04:20:33PM +0200, Laszlo Ersek wrote:
> QEMU does the right thing. If other hypervisors don't do this -- while
> still taking and displaying the value in UUID / GUID textual format --,
> they are wrong. The VMGENID spec from Microsoft
> <http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
> "GUID".

The MSFT spec does mention GUID, but it seems to me that it's only
using GUID as an incidental example -- ie. that you might use the VM
Generation ID to generate a GUID.  Outside that example it
consistently refers to the VM Gen ID as a 128-bit integer.  It also
says that it could be used as a "high entropy random data source",
which is not in fact true if it's a UUID.

It has to be said that after reading the spec again [the MSFT spec,
not qemu's spec] and what other hypervisors are doing, I'm not sure
qemu is doing the right thing here.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Michael S. Tsirkin]

On Thu, Jul 05, 2018 at 05:43:43PM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 04:20:33PM +0200, Laszlo Ersek wrote:
> > QEMU does the right thing. If other hypervisors don't do this -- while
> > still taking and displaying the value in UUID / GUID textual format --,
> > they are wrong. The VMGENID spec from Microsoft
> > <http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
> > "GUID".
> 
> The MSFT spec does mention GUID, but it seems to me that it's only
> using GUID as an incidental example -- ie. that you might use the VM
> Generation ID to generate a GUID.  Outside that example it
> consistently refers to the VM Gen ID as a 128-bit integer.  It also
> says that it could be used as a "high entropy random data source",
> which is not in fact true if it's a UUID.
> 
> It has to be said that after reading the spec again [the MSFT spec,
> not qemu's spec] and what other hypervisors are doing, I'm not sure
> qemu is doing the right thing here.
> 
> Rich.

So right now we have a "GUID" property.
We could always add an alternative property for people who want
to treat the ID as an integer. Would that address the issue?

> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Daniel P. Berrangé]

On Thu, Jul 05, 2018 at 05:43:43PM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 04:20:33PM +0200, Laszlo Ersek wrote:
> > QEMU does the right thing. If other hypervisors don't do this -- while
> > still taking and displaying the value in UUID / GUID textual format --,
> > they are wrong. The VMGENID spec from Microsoft
> > <http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
> > "GUID".
> 
> The MSFT spec does mention GUID, but it seems to me that it's only
> using GUID as an incidental example -- ie. that you might use the VM
> Generation ID to generate a GUID.  Outside that example it
> consistently refers to the VM Gen ID as a 128-bit integer.  It also
> says that it could be used as a "high entropy random data source",
> which is not in fact true if it's a UUID.
> 
> It has to be said that after reading the spec again [the MSFT spec,
> not qemu's spec] and what other hypervisors are doing, I'm not sure
> qemu is doing the right thing here.

I'm not sure it particularly matters, as long as there is a way to
encode the 128-bit integer as a UUID without irreversable mangling
or data loss.  We could document it, or if we really wanted to,
provide an alternative syntax that QEMU internals changes into its
UUID form (or the reverse).  Libvirt would probably still need to
expose it as a UUID though, due to compatibility needs in the XML.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Laszlo Ersek]

On 07/05/18 18:43, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 04:20:33PM +0200, Laszlo Ersek wrote:
>> QEMU does the right thing. If other hypervisors don't do this -- while
>> still taking and displaying the value in UUID / GUID textual format --,
>> they are wrong. The VMGENID spec from Microsoft
>> <http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
>> "GUID".
> 
> The MSFT spec does mention GUID, but it seems to me that it's only
> using GUID as an incidental example -- ie. that you might use the VM
> Generation ID to generate a GUID.  Outside that example it
> consistently refers to the VM Gen ID as a 128-bit integer.  It also
> says that it could be used as a "high entropy random data source",
> which is not in fact true if it's a UUID.

I don't have additional points regarding whether Microsoft genuinely did
or didn't think of VMGENID in terms of UUID.

However, the fact that the fields of the *original* UUID definition have
been repurposed for fully random bytes is not an argument against. Just
because we nowadays populate all those bytes from a PRNG (sort-of),
thereby ignoring their original field names, we still call the structure
UUID and we still preserve the internal field boundaries. For example,
in UEFI (the spec), we have

typedef struct {
  UINT32 Data1;
  UINT16 Data2;
  UINT16 Data3;
  UINT8 Data4[8];
} EFI_GUID;

and in the edk2 codebase, whenever a new GUID is needed for any purpose,
we're supposed to get one by running "uuidgen". From the manual:

       There  are two types of UUIDs which uuidgen can generate: time-
       based UUIDs and random-based UUIDs.  By  default  uuidgen  will
       generate  a  random-based  UUID if a high-quality random number
       generator is present.  Otherwise, it will choose  a  time-based
       UUID.   It  is possible to force the generation of one of these
       two UUID types by using the -r or -t options.

> It has to be said that after reading the spec again [the MSFT spec,
> not qemu's spec] and what other hypervisors are doing, I'm not sure
> qemu is doing the right thing here.

That's the thing about specs -- interpretation. I've read the vmgenid
spec several times as well, and I've always understood it as Microsoft
meaning UUID / GUID as a primary representation for VMGENID. I may have
been biased by UUID/GUID usage in UEFI.

When the feature was being developed for QEMU, we struggled with the
interpretation of a good number of other bits as well; hence the
"Requirements" section of QEMU's spec: "this is how we understood it".
Without an open list to discuss things with Microsoft, sometimes we can
only guess what is a "likely faithful" interpretation, and test it in
practice.

I still believe QEMU's current interpretation is the right one; but I
can't prove it.

Laszlo

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Michael S. Tsirkin]

On Thu, Jul 05, 2018 at 07:15:34PM +0200, Laszlo Ersek wrote:
> > It has to be said that after reading the spec again [the MSFT spec,
> > not qemu's spec] and what other hypervisors are doing, I'm not sure
> > qemu is doing the right thing here.
> 
> That's the thing about specs -- interpretation. I've read the vmgenid
> spec several times as well, and I've always understood it as Microsoft
> meaning UUID / GUID as a primary representation for VMGENID. I may have
> been biased by UUID/GUID usage in UEFI.
> 
> When the feature was being developed for QEMU, we struggled with the
> interpretation of a good number of other bits as well; hence the
> "Requirements" section of QEMU's spec: "this is how we understood it".
> Without an open list to discuss things with Microsoft, sometimes we can
> only guess what is a "likely faithful" interpretation, and test it in
> practice.
> 
> I still believe QEMU's current interpretation is the right one; but I
> can't prove it.
> 
> Laszlo

Well it does seem that at least some guest code wants a counter
there. So if someone posts a patch adding a count property there
to be stored in LE format, plus a unit test - I'll merge it, even
though it does not look like libvirt wants that.

-- 
MST

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Thu, Jul 05, 2018 at 08:10:15PM +0300, Michael S. Tsirkin wrote:
> On Thu, Jul 05, 2018 at 05:43:43PM +0100, Richard W.M. Jones wrote:
> > On Thu, Jul 05, 2018 at 04:20:33PM +0200, Laszlo Ersek wrote:
> > > QEMU does the right thing. If other hypervisors don't do this -- while
> > > still taking and displaying the value in UUID / GUID textual format --,
> > > they are wrong. The VMGENID spec from Microsoft
> > > <http://go.microsoft.com/fwlink/?LinkId=260709> specifically mentions
> > > "GUID".
> > 
> > The MSFT spec does mention GUID, but it seems to me that it's only
> > using GUID as an incidental example -- ie. that you might use the VM
> > Generation ID to generate a GUID.  Outside that example it
> > consistently refers to the VM Gen ID as a 128-bit integer.  It also
> > says that it could be used as a "high entropy random data source",
> > which is not in fact true if it's a UUID.
> > 
> > It has to be said that after reading the spec again [the MSFT spec,
> > not qemu's spec] and what other hypervisors are doing, I'm not sure
> > qemu is doing the right thing here.
> > 
> > Rich.
> 
> So right now we have a "GUID" property.
> We could always add an alternative property for people who want
> to treat the ID as an integer. Would that address the issue?

TBH I'm fairly relaxed about this and I don't want to put in any
effort to change anything.  As Dan says later in the thread, no data
is ever lost in the transformation.

I have now written a "128 bit int" -> "GUID" mangler for virt-v2v's
VM Generation ID handling code, so I'm happy enough.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Thu, Jul 05, 2018 at 03:11:32PM +0100, Richard W.M. Jones wrote:
> VMware represents these internally as two signed 64 bit integers, eg:
> 
>   vm.genid = "-570734802784577186"
>   vm.genidx = "-5042519231342505152"
> 
> I am still trying to get verification, but I believe the first is the
> low 64 bit word and the second is the high 64 bit word.

I have now been able to verify how this works using a real VMware
hypervisor (thanks to help from Ming Xie).  For the record, here is
how it maps, since I could not find any documentation about this.

VMX file contains:

  vm.genid = "7344585841658099715"
  vm.genidX = "-8483171368186442967"

Those numbers are signed 64 bit integers written in hex as:

  vm.genid = 65 ED 35 E8 E2 64 F8 03
  vm.genidX = 8A 45 B8 96 1E 7B 8B 29

In the guest the VMGENID.EXE program prints (with my spaces added for
clarity):

  VmCounterValue: 65 ED 35 E8 E2 64 F8 03 : 8A 45 B8 96 1E 7B 8B 29

So this confirms my original guess.  Note that VMware is not doing any
endianness adjustment, but then VMware only works on LE hardware.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Laszlo Ersek]

On 07/09/18 09:36, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 03:11:32PM +0100, Richard W.M. Jones wrote:
>> VMware represents these internally as two signed 64 bit integers, eg:
>>
>>   vm.genid = "-570734802784577186"
>>   vm.genidx = "-5042519231342505152"
>>
>> I am still trying to get verification, but I believe the first is the
>> low 64 bit word and the second is the high 64 bit word.
> 
> I have now been able to verify how this works using a real VMware
> hypervisor (thanks to help from Ming Xie).  For the record, here is
> how it maps, since I could not find any documentation about this.
> 
> VMX file contains:
> 
>   vm.genid = "7344585841658099715"
>   vm.genidX = "-8483171368186442967"
> 
> Those numbers are signed 64 bit integers written in hex as:
> 
>   vm.genid = 65 ED 35 E8 E2 64 F8 03
>   vm.genidX = 8A 45 B8 96 1E 7B 8B 29

If you mean to describe the byte array representations: these are the
big endian ones.

> In the guest the VMGENID.EXE program prints (with my spaces added for
> clarity):

Right, it's important to note that the spaces below were added for
clarity. The decimal constant 7344585841658099715 is equal to the
hexadecimal constant 0x65ED35E8E264F803. The big endian byte array
representation for that is what you quote above, the little endian one
is the reverse.

>   VmCounterValue: 65 ED 35 E8 E2 64 F8 03 : 8A 45 B8 96 1E 7B 8B 29
> 
> So this confirms my original guess.  Note that VMware is not doing any
> endianness adjustment, but then VMware only works on LE hardware.

Thanks for tracking this down -- TIL.

(My above remarks are not meant as disagreement: it looks like VMWare
simply stores the int64_t values from the config file to guest memory,
without any kind of conversion. I only meant to comment on your
paragraph "written in hex as ...", because it wasn't clear to me whether
you meant "as uint64_t with spaces for clarity" or "as uint8_t[8]". The
latter is endianness-dependent, and the rendering you gave was BE, which
seemed to conflict with your final statement.)

Thanks!
Laszlo

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Mon, Jul 09, 2018 at 11:05:34AM +0200, Laszlo Ersek wrote:
> On 07/09/18 09:36, Richard W.M. Jones wrote:
> > On Thu, Jul 05, 2018 at 03:11:32PM +0100, Richard W.M. Jones wrote:
> >> VMware represents these internally as two signed 64 bit integers, eg:
> >>
> >>   vm.genid = "-570734802784577186"
> >>   vm.genidx = "-5042519231342505152"
> >>
> >> I am still trying to get verification, but I believe the first is the
> >> low 64 bit word and the second is the high 64 bit word.
> > 
> > I have now been able to verify how this works using a real VMware
> > hypervisor (thanks to help from Ming Xie).  For the record, here is
> > how it maps, since I could not find any documentation about this.
> > 
> > VMX file contains:
> > 
> >   vm.genid = "7344585841658099715"
> >   vm.genidX = "-8483171368186442967"
> > 
> > Those numbers are signed 64 bit integers written in hex as:
> > 
> >   vm.genid = 65 ED 35 E8 E2 64 F8 03
> >   vm.genidX = 8A 45 B8 96 1E 7B 8B 29
> 
> If you mean to describe the byte array representations: these are the
> big endian ones.

Yes, the spaces were not meant to indicate that this is the in-memory
representation.

> > In the guest the VMGENID.EXE program prints (with my spaces added for
> > clarity):
> 
> Right, it's important to note that the spaces below were added for
> clarity. The decimal constant 7344585841658099715 is equal to the
> hexadecimal constant 0x65ED35E8E264F803. The big endian byte array
> representation for that is what you quote above, the little endian one
> is the reverse.
> 
> >   VmCounterValue: 65 ED 35 E8 E2 64 F8 03 : 8A 45 B8 96 1E 7B 8B 29
> > 
> > So this confirms my original guess.  Note that VMware is not doing any
> > endianness adjustment, but then VMware only works on LE hardware.
> 
> Thanks for tracking this down -- TIL.
> 
> (My above remarks are not meant as disagreement: it looks like VMWare
> simply stores the int64_t values from the config file to guest memory,
> without any kind of conversion. I only meant to comment on your
> paragraph "written in hex as ...", because it wasn't clear to me whether
> you meant "as uint64_t with spaces for clarity" or "as uint8_t[8]". The
> latter is endianness-dependent, and the rendering you gave was BE, which
> seemed to conflict with your final statement.)

Agreed too.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: [Qemu-devel] Byte ordering of VM Generation ID in Windows VMs

[Richard W.M. Jones]

On Mon, Jul 09, 2018 at 08:36:47AM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 05, 2018 at 03:11:32PM +0100, Richard W.M. Jones wrote:
> > VMware represents these internally as two signed 64 bit integers, eg:
> > 
> >   vm.genid = "-570734802784577186"
> >   vm.genidx = "-5042519231342505152"
> > 
> > I am still trying to get verification, but I believe the first is the
> > low 64 bit word and the second is the high 64 bit word.
> 
> I have now been able to verify how this works using a real VMware
> hypervisor (thanks to help from Ming Xie).  For the record, here is
> how it maps, since I could not find any documentation about this.
> 
> VMX file contains:
> 
>   vm.genid = "7344585841658099715"
>   vm.genidX = "-8483171368186442967"

As a follow-on, here is some code which can convert VMware's VMX
numbers into a qemu-style UUID.  Note for both of these I mean the
textual representation, not the in-memory representation.

  https://github.com/libguestfs/libguestfs/blob/b11b870166b7ae27f728dbb58b6498788bf97329/v2v/input_vmx.ml#L430-L449

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v