* Re: [PATCH 0/1] vmx: Fix <genid/> mapping [not found] <cover.1632900578.git.mprivozn@redhat.com> @ 2021-09-29 9:20 ` Richard W.M. Jones 2021-09-29 9:33 ` Daniel P. Berrangé 0 siblings, 1 reply; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 9:20 UTC (permalink / raw) To: Michal Privoznik; +Cc: libvir-list, qemu-devel On Wed, Sep 29, 2021 at 10:01:55AM +0200, Michal Privoznik wrote: > Apparently, parsing vmx.genid is not as easy as I thought. Anyway, it > was brought up in a private thread that libvirt doesn't report correct > UUIDs. For instance for the following input: > > vm.genid = "-8536691797830587195" > vm.genidX = "-1723453263670062919" (The two lines above come from a VMware .vmx file) The only thing that really matters is what the guest sees. I ran VMGENID.EXE (https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3 https://docs.microsoft.com/en-gb/windows/win32/hyperv_v2/virtual-machine-generation-identifier) inside the guest and it showed: 8987940a09512cc5:e81510634ff550b9 Note these numbers are the hex equivalents of the VMware .vmx numbers: >>> print("%x" % (2**64-8536691797830587195)) 8987940a09512cc5 >>> print("%x" % (2**64-1723453263670062919)) e81510634ff550b9 > Libvirt would report: > > <genid>8987940a-0951-2cc5-e815-10634ff550b9</genid> > > while virt-v2v would report: > > <genid>09512cc5-940a-8987-b950-f54f631015e8</genid> After thinking about this a bit more, IMHO the real problem is with qemu. If you pass this to qemu: -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 then inside the guest VMGENID shows 2cc509518987940a:b950f54f631015e8 (wrong) If you pass this to qemu: ...guid=09512cc5-940a-8987-b950-f54f631015e8... then inside the guest it sees 8987940a09512cc5:e81510634ff550b9 (the correct values, matching VMware). This is the reason why virt-v2v mangles the GUID, in order to trick libvirt into passing a mangled GUID which gets mangled again by qemu which is the same as unmangling it. So another way to fix this could be for us to fix qemu. We could just pass the two numbers to qemu instead of using GUIDs, eg: -device vmgenid,low=0x8987940a09512cc5,high=0xe81510634ff550b9,id=vmgenid0 and then we'd fix the implementation in qemu so that the low/high values match what is seen by VMGENID.EXE in the guest. Or we could have libvirt use the current GUID in <genid> but to do the mangling when it gets passed through to qemu (instead of virt-v2v doing the mangling). Adding qemu-devel because I'm interesting to see if the qemu developers would prefer to fix this properly in qemu. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:20 ` [PATCH 0/1] vmx: Fix <genid/> mapping Richard W.M. Jones @ 2021-09-29 9:33 ` Daniel P. Berrangé 2021-09-29 9:46 ` Richard W.M. Jones 2021-09-29 9:57 ` Richard W.M. Jones 0 siblings, 2 replies; 16+ messages in thread From: Daniel P. Berrangé @ 2021-09-29 9:33 UTC (permalink / raw) To: Richard W.M. Jones; +Cc: libvir-list, Michal Privoznik, qemu-devel On Wed, Sep 29, 2021 at 10:20:44AM +0100, Richard W.M. Jones wrote: > On Wed, Sep 29, 2021 at 10:01:55AM +0200, Michal Privoznik wrote: > > Apparently, parsing vmx.genid is not as easy as I thought. Anyway, it > > was brought up in a private thread that libvirt doesn't report correct > > UUIDs. For instance for the following input: > > > > vm.genid = "-8536691797830587195" > > vm.genidX = "-1723453263670062919" > > (The two lines above come from a VMware .vmx file) > > The only thing that really matters is what the guest sees. I ran > VMGENID.EXE (https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3 > https://docs.microsoft.com/en-gb/windows/win32/hyperv_v2/virtual-machine-generation-identifier) > inside the guest and it showed: > > 8987940a09512cc5:e81510634ff550b9 > > Note these numbers are the hex equivalents of the VMware .vmx numbers: > > >>> print("%x" % (2**64-8536691797830587195)) > 8987940a09512cc5 > >>> print("%x" % (2**64-1723453263670062919)) > e81510634ff550b9 > > > Libvirt would report: > > > > <genid>8987940a-0951-2cc5-e815-10634ff550b9</genid> > > > > while virt-v2v would report: > > > > <genid>09512cc5-940a-8987-b950-f54f631015e8</genid> > > After thinking about this a bit more, IMHO the real problem is > with qemu. If you pass this to qemu: > > -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 > > then inside the guest VMGENID shows 2cc509518987940a:b950f54f631015e8 (wrong) > > If you pass this to qemu: > > ...guid=09512cc5-940a-8987-b950-f54f631015e8... > > then inside the guest it sees 8987940a09512cc5:e81510634ff550b9 > (the correct values, matching VMware). > > This is the reason why virt-v2v mangles the GUID, in order to trick > libvirt into passing a mangled GUID which gets mangled again by qemu > which is the same as unmangling it. > > So another way to fix this could be for us to fix qemu. We could just > pass the two numbers to qemu instead of using GUIDs, eg: > > -device vmgenid,low=0x8987940a09512cc5,high=0xe81510634ff550b9,id=vmgenid0 > > and then we'd fix the implementation in qemu so that the low/high > values match what is seen by VMGENID.EXE in the guest. > > Or we could have libvirt use the current GUID in <genid> but to do the > mangling when it gets passed through to qemu (instead of virt-v2v > doing the mangling). On the libvirt side, the #1 most important thing is that a given XML string <genid>8987940a-0951-2cc5-e815-10634ff550b9</genid> results in the same value being exposed to the guest OS, for both the QEMU and VMWare drivers. Whehter the guest sees the bytes in the same order is not essential, but it would be less of a surprise if it did match. Ultimately as long as the mapping from libvirt XML to guest visible string is consistent across drivers, that's sufficient. > Adding qemu-devel because I'm interesting to see if the qemu > developers would prefer to fix this properly in qemu. No matter what order QEMU accepts the data in, it can be said to be functionally correct. If the order is "unusual" though, it ought to at least be documented how the QEMU order corresponds to guest visible order. Incidentally I wonder how much to trust VMGENID.EXE and whether that actally reports what it gets out of guest memory "as is", or whether it has done any byte re-ordering. I'm curious what Linux kernel sees for the genid on Vmware vs KVM hosts, as probably I'd trust that data more ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:33 ` Daniel P. Berrangé @ 2021-09-29 9:46 ` Richard W.M. Jones 2021-09-29 10:07 ` Daniel P. Berrangé 2021-09-29 10:24 ` Richard W.M. Jones 2021-09-29 9:57 ` Richard W.M. Jones 1 sibling, 2 replies; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 9:46 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, Catangiu, Adrian Costin, qemu-devel On Wed, Sep 29, 2021 at 10:33:43AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 29, 2021 at 10:20:44AM +0100, Richard W.M. Jones wrote: > > On Wed, Sep 29, 2021 at 10:01:55AM +0200, Michal Privoznik wrote: > > > Apparently, parsing vmx.genid is not as easy as I thought. Anyway, it > > > was brought up in a private thread that libvirt doesn't report correct > > > UUIDs. For instance for the following input: > > > > > > vm.genid = "-8536691797830587195" > > > vm.genidX = "-1723453263670062919" > > > > (The two lines above come from a VMware .vmx file) > > > > The only thing that really matters is what the guest sees. I ran > > VMGENID.EXE (https://bugzilla.redhat.com/show_bug.cgi?id=1598350#c3 > > https://docs.microsoft.com/en-gb/windows/win32/hyperv_v2/virtual-machine-generation-identifier) > > inside the guest and it showed: > > > > 8987940a09512cc5:e81510634ff550b9 > > > > Note these numbers are the hex equivalents of the VMware .vmx numbers: > > > > >>> print("%x" % (2**64-8536691797830587195)) > > 8987940a09512cc5 > > >>> print("%x" % (2**64-1723453263670062919)) > > e81510634ff550b9 > > > > > Libvirt would report: > > > > > > <genid>8987940a-0951-2cc5-e815-10634ff550b9</genid> > > > > > > while virt-v2v would report: > > > > > > <genid>09512cc5-940a-8987-b950-f54f631015e8</genid> > > > > After thinking about this a bit more, IMHO the real problem is > > with qemu. If you pass this to qemu: > > > > -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 > > > > then inside the guest VMGENID shows 2cc509518987940a:b950f54f631015e8 (wrong) > > > > If you pass this to qemu: > > > > ...guid=09512cc5-940a-8987-b950-f54f631015e8... > > > > then inside the guest it sees 8987940a09512cc5:e81510634ff550b9 > > (the correct values, matching VMware). > > > > This is the reason why virt-v2v mangles the GUID, in order to trick > > libvirt into passing a mangled GUID which gets mangled again by qemu > > which is the same as unmangling it. > > > > So another way to fix this could be for us to fix qemu. We could just > > pass the two numbers to qemu instead of using GUIDs, eg: > > > > -device vmgenid,low=0x8987940a09512cc5,high=0xe81510634ff550b9,id=vmgenid0 > > > > and then we'd fix the implementation in qemu so that the low/high > > values match what is seen by VMGENID.EXE in the guest. > > > > Or we could have libvirt use the current GUID in <genid> but to do the > > mangling when it gets passed through to qemu (instead of virt-v2v > > doing the mangling). > > On the libvirt side, the #1 most important thing is that a given > XML string > > <genid>8987940a-0951-2cc5-e815-10634ff550b9</genid> > > results in the same value being exposed to the guest OS, for both > the QEMU and VMWare drivers. Whehter the guest sees the bytes in > the same order is not essential, but it would be less of a surprise > if it did match. I don't know why we decided to use a GUID for this. The feature itself (https://go.microsoft.com/fwlink/?LinkId=260709) defines it as an 128 bit / 8 byte number. The only connection to GUIDs is the size. > Ultimately as long as the mapping from libvirt XML to guest visible > string is consistent across drivers, that's sufficient. > > > Adding qemu-devel because I'm interesting to see if the qemu > > developers would prefer to fix this properly in qemu. > > No matter what order QEMU accepts the data in, it can be said to be > functionally correct. If the order is "unusual" though, it ought to > at least be documented how the QEMU order corresponds to guest visible > order. > > Incidentally I wonder how much to trust VMGENID.EXE and whether that > actally reports what it gets out of guest memory "as is", or whether > it has done any byte re-ordering. > > I'm curious what Linux kernel sees for the genid on Vmware vs KVM > hosts, as probably I'd trust that data more ? As far as I can tell no driver has successfully made it upstream, although there have been a few attempts: https://lkml.org/lkml/2018/3/1/498 Here's a more recent one from 10 months ago: https://lore.kernel.org/linux-doc/3E05451B-A9CD-4719-99D0-72750A304044@amazon.com/ If I have time I'll patch a Linux kernel with the second patch and see how it relates to the VMware and KVM parameters, but it probably won't be today. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:46 ` Richard W.M. Jones @ 2021-09-29 10:07 ` Daniel P. Berrangé 2021-09-29 10:24 ` Richard W.M. Jones 2021-09-29 10:24 ` Richard W.M. Jones 1 sibling, 1 reply; 16+ messages in thread From: Daniel P. Berrangé @ 2021-09-29 10:07 UTC (permalink / raw) To: Richard W.M. Jones Cc: libvir-list, Michal Privoznik, Catangiu, Adrian Costin, qemu-devel On Wed, Sep 29, 2021 at 10:46:39AM +0100, Richard W.M. Jones wrote: > On Wed, Sep 29, 2021 at 10:33:43AM +0100, Daniel P. Berrangé wrote: > > Ultimately as long as the mapping from libvirt XML to guest visible > > string is consistent across drivers, that's sufficient. > > > > > Adding qemu-devel because I'm interesting to see if the qemu > > > developers would prefer to fix this properly in qemu. > > > > No matter what order QEMU accepts the data in, it can be said to be > > functionally correct. If the order is "unusual" though, it ought to > > at least be documented how the QEMU order corresponds to guest visible > > order. > > > > Incidentally I wonder how much to trust VMGENID.EXE and whether that > > actally reports what it gets out of guest memory "as is", or whether > > it has done any byte re-ordering. > > > > I'm curious what Linux kernel sees for the genid on Vmware vs KVM > > hosts, as probably I'd trust that data more ? > > As far as I can tell no driver has successfully made it upstream, > although there have been a few attempts: > > https://lkml.org/lkml/2018/3/1/498 > > Here's a more recent one from 10 months ago: > > https://lore.kernel.org/linux-doc/3E05451B-A9CD-4719-99D0-72750A304044@amazon.com/ > > If I have time I'll patch a Linux kernel with the second patch and see > how it relates to the VMware and KVM parameters, but it probably won't > be today. I'm not sure if we actually need the full driver or not for testing purposes. The the GenID is just in memory somewhere, and the somewhere is reported via ACPI table entry. For QEMU its easy as the data is exposed via fw_cfg which can be read from sysfs directly without even needing to look at ACPI entries to find it. Not sure how we find it with VMWare/HyperV though. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 10:07 ` Daniel P. Berrangé @ 2021-09-29 10:24 ` Richard W.M. Jones 0 siblings, 0 replies; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 10:24 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, Catangiu, Adrian Costin, qemu-devel On Wed, Sep 29, 2021 at 11:07:30AM +0100, Daniel P. Berrangé wrote: > I'm not sure if we actually need the full driver or not for testing > purposes. The the GenID is just in memory somewhere, and the somewhere > is reported via ACPI table entry. For QEMU its easy as the data is > exposed via fw_cfg which can be read from sysfs directly without > even needing to look at ACPI entries to find it. Not sure how we > find it with VMWare/HyperV though. This still has the problem that qemu is mangling the vmgenid. Nevertheless, on qemu-6.1.0-5.fc36.x86_64 I added this to a Linux guest: <genid>11223344-5566-7788-99aa-bbccddeeff00</genid> which turned into: -device vmgenid,guid=11223344-5566-7788-99aa-bbccddeeff00,id=vmgenid0 Inside the guest: # ls /sys/firmware/qemu_fw_cfg/by_name/etc/vmgenid_guid/ -l total 0 -r--------. 1 root root 4096 Sep 29 11:16 key -r--------. 1 root root 4096 Sep 29 11:16 name -r--------. 1 root root 0 Sep 29 11:16 raw -r--------. 1 root root 4096 Sep 29 11:16 size # hexdump -C /sys/firmware/qemu_fw_cfg/by_name/etc/vmgenid_guid/raw 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000020 00 00 00 00 00 00 00 00 44 33 22 11 66 55 88 77 |........D3".fU.w| 00000030 99 aa bb cc dd ee ff 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 But I think what I really need to do is look at the raw physical address: # hexdump -C /sys/firmware/qemu_fw_cfg/by_name/etc/vmgenid_addr/raw 00000000 28 f0 ff 7f 00 00 00 00 |(.......| 00000008 # dd if=/dev/mem bs=1 skip=$(( 0x7ffff028 )) count=16 | hexdump -C 16+0 records in 16+0 records out 16 bytes copied, 6.0392e-05 s, 265 kB/s 00000000 44 33 22 11 66 55 88 77 99 aa bb cc dd ee ff 00 |D3".fU.w........| 00000010 I think for VMware I'm really going to need the kernel driver, unless there's some way that iasl can be used to extract the information? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:46 ` Richard W.M. Jones 2021-09-29 10:07 ` Daniel P. Berrangé @ 2021-09-29 10:24 ` Richard W.M. Jones 1 sibling, 0 replies; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 10:24 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, Catangiu, Adrian Costin, qemu-devel On Wed, Sep 29, 2021 at 10:46:38AM +0100, Richard W.M. Jones wrote: > I don't know why we decided to use a GUID for this. The feature > itself (https://go.microsoft.com/fwlink/?LinkId=260709) defines it as > an 128 bit / 8 byte number. The only connection to GUIDs is the size. *cough* .. 16 bytes :-) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:33 ` Daniel P. Berrangé 2021-09-29 9:46 ` Richard W.M. Jones @ 2021-09-29 9:57 ` Richard W.M. Jones 2021-09-29 10:10 ` Daniel P. Berrangé 1 sibling, 1 reply; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 9:57 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: libvir-list, Michal Privoznik, qemu-devel Looking at the qemu code the problem IMHO is: https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/docs/specs/vmgenid.txt#L189 https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/hw/acpi/vmgenid.c#L37 This byte swapping makes no sense to me. How do we know that the guest is little endian? What will this code do for BE guests? I think qemu would be better off treating the "GUID" as a list of bytes and writing that exactly into the guest memory. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 9:57 ` Richard W.M. Jones @ 2021-09-29 10:10 ` Daniel P. Berrangé 2021-09-29 10:34 ` Richard W.M. Jones 0 siblings, 1 reply; 16+ messages in thread From: Daniel P. Berrangé @ 2021-09-29 10:10 UTC (permalink / raw) To: Richard W.M. Jones; +Cc: libvir-list, Michal Privoznik, qemu-devel On Wed, Sep 29, 2021 at 10:57:19AM +0100, Richard W.M. Jones wrote: > Looking at the qemu code the problem IMHO is: > > https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/docs/specs/vmgenid.txt#L189 > https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/hw/acpi/vmgenid.c#L37 > > This byte swapping makes no sense to me. How do we know that the > guest is little endian? What will this code do for BE guests? I > think qemu would be better off treating the "GUID" as a list of bytes > and writing that exactly into the guest memory. This is an artifact of the HyperV only caring about x86 and thus leaving endianness unspecified in the spec for GenID. QEMU docs say Endian-ness Considerations: --------------------------- Although not specified in Microsoft's document, it is assumed that the device is expected to use little-endian format. All GUID passed in via command line or monitor are treated as big-endian. GUID values displayed via monitor are shown in big-endian format. So by extension if libvirt is passing the value from its XML straight to QEMU, then libvirt has effectively defined that the XML is storing it big-endian too. This could be where the confusion with VMX config is coming into play, though the byte re-ordering in v2v seems more complex than just an endianess-fiddle ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 10:10 ` Daniel P. Berrangé @ 2021-09-29 10:34 ` Richard W.M. Jones 2021-09-30 7:33 ` Richard W.M. Jones 0 siblings, 1 reply; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-29 10:34 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: libvir-list, Michal Privoznik, qemu-devel On Wed, Sep 29, 2021 at 11:10:35AM +0100, Daniel P. Berrangé wrote: > On Wed, Sep 29, 2021 at 10:57:19AM +0100, Richard W.M. Jones wrote: > > Looking at the qemu code the problem IMHO is: > > > > https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/docs/specs/vmgenid.txt#L189 > > https://gitlab.com/qemu-project/qemu/-/blob/6b54a31bf7b403672a798b6443b1930ae6c74dea/hw/acpi/vmgenid.c#L37 > > > > This byte swapping makes no sense to me. How do we know that the > > guest is little endian? What will this code do for BE guests? I > > think qemu would be better off treating the "GUID" as a list of bytes > > and writing that exactly into the guest memory. > > This is an artifact of the HyperV only caring about x86 and thus leaving > endianness unspecified in the spec for GenID. QEMU docs say > > > Endian-ness Considerations: > --------------------------- > > Although not specified in Microsoft's document, it is assumed that the > device is expected to use little-endian format. > > All GUID passed in via command line or monitor are treated as big-endian. > GUID values displayed via monitor are shown in big-endian format. > > > So by extension if libvirt is passing the value from its XML straight > to QEMU, then libvirt has effectively defined that the XML is storing > it big-endian too. > > This could be where the confusion with VMX config is coming into play, > though the byte re-ordering in v2v seems more complex than just an > endianess-fiddle ? qemu's qemu_uuid_bswap function only swaps some of the fields. I think even more that applying qemu_uuid_bswap to these (not really) "UUIDs" is a nonsense. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-29 10:34 ` Richard W.M. Jones @ 2021-09-30 7:33 ` Richard W.M. Jones 2021-09-30 8:35 ` Laszlo Ersek 2021-09-30 8:47 ` Daniel P. Berrangé 0 siblings, 2 replies; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-30 7:33 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, lersek, qemu-devel, armbru More data: I found a colleague who has a Hyper-V instance with a Windows guest and he helped me to understand how Hyper-V represents generation IDs. Hyper-V has had support for generation IDs since long before Microsoft proposed the feature for standardization. Originally (I think pre-2013) Hyper-V used an XML description which included: <generation_id type="string">4a07df4361fdf4c883f97bc30e524b9d</generation_id> Note this is a pure hex string, not a GUID. In current Hyper-V, the same representation is used but it's embedded in a binary file. My colleague ran two Windows VMs on Hyper-V and examined the generation_id on the hypervisor and compared it to the output of VMGENID.EXE inside the guest. The first guest had this in the binary hypervisor metadata: 000056b0 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e 5f 69 64 |...generation_id| 000056c0 00 40 00 00 00 38 00 30 00 35 00 61 00 32 00 38 |.@...8.0.5.a.2.8| 000056d0 00 37 00 61 00 32 00 35 00 30 00 39 00 38 00 39 |.7.a.2.5.0.9.8.9| 000056e0 00 65 00 34 00 66 00 65 00 36 00 66 00 36 00 39 |.e.4.f.e.6.f.6.9| 000056f0 00 39 00 32 00 62 00 65 00 33 00 33 00 34 00 61 |.9.2.b.e.3.3.4.a| 00005700 00 34 00 33 00 00 00 00 00 00 00 00 00 00 00 00 |.4.3............| and reported the output of VMGENID in this guest as: VmCounterValue: fe6f6992be334a43:805a287a250989e4 The second guest had: 00005360 c5 06 00 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e |......generation| 00005370 5f 69 64 00 40 00 00 00 65 00 62 00 66 00 62 00 |_id.@...e.b.f.b.| 00005380 62 00 37 00 39 00 37 00 33 00 36 00 35 00 37 00 |b.7.9.7.3.6.5.7.| 00005390 32 00 38 00 65 00 37 00 30 00 62 00 33 00 66 00 |2.8.e.7.0.b.3.f.| 000053a0 64 00 33 00 63 00 37 00 31 00 33 00 65 00 63 00 |d.3.c.7.1.3.e.c.| 000053b0 65 00 38 00 34 00 32 00 00 00 00 00 00 00 00 00 |e.8.4.2.........| and VMGENID was: VmCounterValue: b3fd3c713ece842:ebfbb797365728e7 Note: - In both cases, the generation ID is clearly not a GUID. - The two 64 bit words are swapped over somewhere, but individual bytes are not byte-swapped, and there is again no evidence of UUID-aware byte swapping. ---------------------------------------------------------------------- So how do we get from where we are to a more sane vmgenid in qemu? I propose we deprecate the guid parameter in: -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 Instead it will be replaced by bytes= which will simply write the bytes, in the order they appear, into guest memory with no attempt to interpret or byte-swap. Something like: -device vmgenid,bytes=112233445566778899aabbccddeeff00,id=vmgenid0 (guid although deprecated will need to be kept around for a while, along with its weird byte-swapping behaviour). We will then have a plain and simple method to emulate the behaviour of other hypervisors. We will look at exactly what bytes they write to guest memory and copy that behaviour when v2v converting from those hypervisors. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-30 7:33 ` Richard W.M. Jones @ 2021-09-30 8:35 ` Laszlo Ersek 2021-09-30 8:47 ` Daniel P. Berrangé 1 sibling, 0 replies; 16+ messages in thread From: Laszlo Ersek @ 2021-09-30 8:35 UTC (permalink / raw) To: Richard W.M. Jones, Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, qemu-devel, armbru On 09/30/21 09:33, Richard W.M. Jones wrote: > > More data: I found a colleague who has a Hyper-V instance with a > Windows guest and he helped me to understand how Hyper-V represents > generation IDs. Hyper-V has had support for generation IDs since long > before Microsoft proposed the feature for standardization. Originally > (I think pre-2013) Hyper-V used an XML description which included: > > <generation_id type="string">4a07df4361fdf4c883f97bc30e524b9d</generation_id> > > Note this is a pure hex string, not a GUID. > > In current Hyper-V, the same representation is used but it's embedded > in a binary file. > > My colleague ran two Windows VMs on Hyper-V and examined the > generation_id on the hypervisor and compared it to the output of > VMGENID.EXE inside the guest. > > The first guest had this in the binary hypervisor metadata: > > 000056b0 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e 5f 69 64 |...generation_id| > 000056c0 00 40 00 00 00 38 00 30 00 35 00 61 00 32 00 38 |.@...8.0.5.a.2.8| > 000056d0 00 37 00 61 00 32 00 35 00 30 00 39 00 38 00 39 |.7.a.2.5.0.9.8.9| > 000056e0 00 65 00 34 00 66 00 65 00 36 00 66 00 36 00 39 |.e.4.f.e.6.f.6.9| > 000056f0 00 39 00 32 00 62 00 65 00 33 00 33 00 34 00 61 |.9.2.b.e.3.3.4.a| > 00005700 00 34 00 33 00 00 00 00 00 00 00 00 00 00 00 00 |.4.3............| > > and reported the output of VMGENID in this guest as: > > VmCounterValue: fe6f6992be334a43:805a287a250989e4 > > The second guest had: > > 00005360 c5 06 00 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e |......generation| > 00005370 5f 69 64 00 40 00 00 00 65 00 62 00 66 00 62 00 |_id.@...e.b.f.b.| > 00005380 62 00 37 00 39 00 37 00 33 00 36 00 35 00 37 00 |b.7.9.7.3.6.5.7.| > 00005390 32 00 38 00 65 00 37 00 30 00 62 00 33 00 66 00 |2.8.e.7.0.b.3.f.| > 000053a0 64 00 33 00 63 00 37 00 31 00 33 00 65 00 63 00 |d.3.c.7.1.3.e.c.| > 000053b0 65 00 38 00 34 00 32 00 00 00 00 00 00 00 00 00 |e.8.4.2.........| > > and VMGENID was: > > VmCounterValue: b3fd3c713ece842:ebfbb797365728e7 > > Note: > > - In both cases, the generation ID is clearly not a GUID. > > - The two 64 bit words are swapped over somewhere, but individual > bytes are not byte-swapped, and there is again no evidence of > UUID-aware byte swapping. > > ---------------------------------------------------------------------- > > So how do we get from where we are to a more sane vmgenid in qemu? > > I propose we deprecate the guid parameter in: > > -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 > > Instead it will be replaced by bytes= which will simply write > the bytes, in the order they appear, into guest memory with no > attempt to interpret or byte-swap. Something like: > > -device vmgenid,bytes=112233445566778899aabbccddeeff00,id=vmgenid0 > > (guid although deprecated will need to be kept around for a while, > along with its weird byte-swapping behaviour). > > We will then have a plain and simple method to emulate the behaviour > of other hypervisors. We will look at exactly what bytes they write > to guest memory and copy that behaviour when v2v converting from those > hypervisors. I don't have anything against this, I just don't know who's going to implement the QEMU change. Thanks Laszlo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-30 7:33 ` Richard W.M. Jones 2021-09-30 8:35 ` Laszlo Ersek @ 2021-09-30 8:47 ` Daniel P. Berrangé 2021-09-30 9:16 ` Richard W.M. Jones 1 sibling, 1 reply; 16+ messages in thread From: Daniel P. Berrangé @ 2021-09-30 8:47 UTC (permalink / raw) To: Richard W.M. Jones Cc: libvir-list, Michal Privoznik, lersek, qemu-devel, armbru On Thu, Sep 30, 2021 at 08:33:48AM +0100, Richard W.M. Jones wrote: > > More data: I found a colleague who has a Hyper-V instance with a > Windows guest and he helped me to understand how Hyper-V represents > generation IDs. Hyper-V has had support for generation IDs since long > before Microsoft proposed the feature for standardization. Originally > (I think pre-2013) Hyper-V used an XML description which included: > > <generation_id type="string">4a07df4361fdf4c883f97bc30e524b9d</generation_id> > > Note this is a pure hex string, not a GUID. > > In current Hyper-V, the same representation is used but it's embedded > in a binary file. > > My colleague ran two Windows VMs on Hyper-V and examined the > generation_id on the hypervisor and compared it to the output of > VMGENID.EXE inside the guest. > > The first guest had this in the binary hypervisor metadata: > > 000056b0 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e 5f 69 64 |...generation_id| > 000056c0 00 40 00 00 00 38 00 30 00 35 00 61 00 32 00 38 |.@...8.0.5.a.2.8| > 000056d0 00 37 00 61 00 32 00 35 00 30 00 39 00 38 00 39 |.7.a.2.5.0.9.8.9| > 000056e0 00 65 00 34 00 66 00 65 00 36 00 66 00 36 00 39 |.e.4.f.e.6.f.6.9| > 000056f0 00 39 00 32 00 62 00 65 00 33 00 33 00 34 00 61 |.9.2.b.e.3.3.4.a| > 00005700 00 34 00 33 00 00 00 00 00 00 00 00 00 00 00 00 |.4.3............| > > and reported the output of VMGENID in this guest as: > > VmCounterValue: fe6f6992be334a43:805a287a250989e4 > > The second guest had: > > 00005360 c5 06 00 00 00 0e 67 65 6e 65 72 61 74 69 6f 6e |......generation| > 00005370 5f 69 64 00 40 00 00 00 65 00 62 00 66 00 62 00 |_id.@...e.b.f.b.| > 00005380 62 00 37 00 39 00 37 00 33 00 36 00 35 00 37 00 |b.7.9.7.3.6.5.7.| > 00005390 32 00 38 00 65 00 37 00 30 00 62 00 33 00 66 00 |2.8.e.7.0.b.3.f.| > 000053a0 64 00 33 00 63 00 37 00 31 00 33 00 65 00 63 00 |d.3.c.7.1.3.e.c.| > 000053b0 65 00 38 00 34 00 32 00 00 00 00 00 00 00 00 00 |e.8.4.2.........| > > and VMGENID was: > > VmCounterValue: b3fd3c713ece842:ebfbb797365728e7 > > Note: > > - In both cases, the generation ID is clearly not a GUID. > > - The two 64 bit words are swapped over somewhere, but individual > bytes are not byte-swapped, and there is again no evidence of > UUID-aware byte swapping. > > ---------------------------------------------------------------------- > > So how do we get from where we are to a more sane vmgenid in qemu? > > I propose we deprecate the guid parameter in: > > -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 > > Instead it will be replaced by bytes= which will simply write > the bytes, in the order they appear, into guest memory with no > attempt to interpret or byte-swap. Something like: > > -device vmgenid,bytes=112233445566778899aabbccddeeff00,id=vmgenid0 > > (guid although deprecated will need to be kept around for a while, > along with its weird byte-swapping behaviour). > > We will then have a plain and simple method to emulate the behaviour > of other hypervisors. We will look at exactly what bytes they write > to guest memory and copy that behaviour when v2v converting from those > hypervisors. From the libvirt POV, I'm not expecting anything in QEMU to change in this respect. If guid is replaced by a new attribute taking data in a different way, then libvirt will have to remap itself, so that existing usage in libvirt keeps working the same way as it did with guid. Essentially from libvirt's POV, it is simply a documentation issue to specify how the libvirt XML representation translates to the guest visible representation, and ensure that all libvirt drivers implement it the same way. The QEMU genid support arrived first so that set the standard for how libvirt will represent it, that all further libvirt hypervisor drivers need to match. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-30 8:47 ` Daniel P. Berrangé @ 2021-09-30 9:16 ` Richard W.M. Jones 2021-10-04 9:59 ` Richard W.M. Jones 0 siblings, 1 reply; 16+ messages in thread From: Richard W.M. Jones @ 2021-09-30 9:16 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, lersek, qemu-devel, armbru On Thu, Sep 30, 2021 at 09:47:01AM +0100, Daniel P. Berrangé wrote: > On Thu, Sep 30, 2021 at 08:33:48AM +0100, Richard W.M. Jones wrote: > > I propose we deprecate the guid parameter in: > > > > -device vmgenid,guid=8987940a-0951-2cc5-e815-10634ff550b9,id=vmgenid0 > > > > Instead it will be replaced by bytes= which will simply write > > the bytes, in the order they appear, into guest memory with no > > attempt to interpret or byte-swap. Something like: > > > > -device vmgenid,bytes=112233445566778899aabbccddeeff00,id=vmgenid0 > > > > (guid although deprecated will need to be kept around for a while, > > along with its weird byte-swapping behaviour). > > > > We will then have a plain and simple method to emulate the behaviour > > of other hypervisors. We will look at exactly what bytes they write > > to guest memory and copy that behaviour when v2v converting from those > > hypervisors. > > From the libvirt POV, I'm not expecting anything in QEMU to change > in this respect. If guid is replaced by a new attribute taking data > in a different way, then libvirt will have to remap itself, so that > existing usage in libvirt keeps working the same way as it did with > guid. Essentially from libvirt's POV, it is simply a documentation > issue to specify how the libvirt XML representation translates to > the guest visible representation, and ensure that all libvirt drivers > implement it the same way. The QEMU genid support arrived first so > that set the standard for how libvirt will represent it, that all > further libvirt hypervisor drivers need to match. I was going to suggest something like: <genid type="guid">aa-bb-cc..</genid> or <genid type="binary">aabbcc..</genid> with the type defaulting to guid for backwards compatibility. Does libvirt XML have any other fields were you're passing essentially small snippets of binary data? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-09-30 9:16 ` Richard W.M. Jones @ 2021-10-04 9:59 ` Richard W.M. Jones 2021-10-04 14:50 ` Laszlo Ersek 0 siblings, 1 reply; 16+ messages in thread From: Richard W.M. Jones @ 2021-10-04 9:59 UTC (permalink / raw) To: Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, qemu-devel, armbru, acatan, lersek It turns out that changing the qemu implementation is painful, particularly if we wish to maintain backwards compatibility of the command line and live migration. Instead I opted to document comprehensively what all the different hypervisors do: https://github.com/libguestfs/virt-v2v/blob/master/docs/vm-generation-id-across-hypervisors.txt On Thu, Sep 30, 2021 at 10:16:20AM +0100, Richard W.M. Jones wrote: > I was going to suggest something like: > > <genid type="guid">aa-bb-cc..</genid> > or > <genid type="binary">aabbcc..</genid> After thinking about this some more, the real implementation on Windows guest and host is two 64 bit little-endian integers[1]. How about implementing this exactly the same way as Hyper-V (and VMware): <genid type="pair"> <low>0x8877665544332211</low> <high>0x00ffeeddccbbaa99</high> </genid> This would have to be mapped to the following qemu command line: qemu -device vmgenid,guid=44332211-6655-8877-99aa-bbccddeeff00 which would unmangle to the following in guest physical memory: 11 22 33 44 55 66 77 88 99 aa bb cc dd ee ff 00 The equivalent back-compat option would be: <genid type="qemu">44332211-6655-8877-99aa-bbccddeeff00</genid> Rich. [1] No one has ever thought what to do about big-endian guests. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-10-04 9:59 ` Richard W.M. Jones @ 2021-10-04 14:50 ` Laszlo Ersek 2021-10-04 14:59 ` Richard W.M. Jones 0 siblings, 1 reply; 16+ messages in thread From: Laszlo Ersek @ 2021-10-04 14:50 UTC (permalink / raw) To: Richard W.M. Jones, Daniel P. Berrangé Cc: libvir-list, Michal Privoznik, acatan, qemu-devel, armbru On 10/04/21 11:59, Richard W.M. Jones wrote: > It turns out that changing the qemu implementation is painful, > particularly if we wish to maintain backwards compatibility of the > command line and live migration. > > Instead I opted to document comprehensively what all the > different hypervisors do: > > https://github.com/libguestfs/virt-v2v/blob/master/docs/vm-generation-id-across-hypervisors.txt > Unfortunately QEMU made a significant mistake when implementing this > feature. Because the string is 128 bits wrong, they decided it must ^^^^^^^^^^^^^^ Haha, that's a great typo :) > be a UUID (as you can see above there is no evidence that Microsoft > who wrote the original spec thought it was). Following from this > incorrect assumption, they stated that the "UUID" must be supplied to > qemu in big endian format and must be byteswapped when writing it to > guest memory. Their documentation says that they only do this for > little endian guests, but this is not true of their implementation > which byte swaps it for all guests. I don't think this is what section "Endian-ness Considerations" in "docs/specs/vmgenid.txt" says. That text says that the *device* uses little-endian format. That's independent of the endianness of *CPU* of the particular guest architecture. So the byte-swapping in the QEMU code occurs unconditionally because QEMU's UUID type is inherently big endian, and the device model in question is fixed little endian. The guest CPU's endianness is irrelevant as far as the device is concerned. If a BE guest CPU intends to read the generation ID and to interpret it as a set of integers, then the guest CPU is supposed to byte-swap the appropriate fields itself. > References I suggest adding two links in this section, namely to: - docs/specs/vmgenid.txt - hw/acpi/vmgenid.c Thanks, Laszlo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/1] vmx: Fix <genid/> mapping 2021-10-04 14:50 ` Laszlo Ersek @ 2021-10-04 14:59 ` Richard W.M. Jones 0 siblings, 0 replies; 16+ messages in thread From: Richard W.M. Jones @ 2021-10-04 14:59 UTC (permalink / raw) To: Laszlo Ersek Cc: Daniel P. Berrangé, libvir-list, Michal Privoznik, qemu-devel, armbru, acatan On Mon, Oct 04, 2021 at 04:50:51PM +0200, Laszlo Ersek wrote: > On 10/04/21 11:59, Richard W.M. Jones wrote: > > It turns out that changing the qemu implementation is painful, > > particularly if we wish to maintain backwards compatibility of the > > command line and live migration. > > > > Instead I opted to document comprehensively what all the > > different hypervisors do: > > > > https://github.com/libguestfs/virt-v2v/blob/master/docs/vm-generation-id-across-hypervisors.txt > > > Unfortunately QEMU made a significant mistake when implementing this > > feature. Because the string is 128 bits wrong, they decided it must > ^^^^^^^^^^^^^^ > > Haha, that's a great typo :) > > > be a UUID (as you can see above there is no evidence that Microsoft > > who wrote the original spec thought it was). Following from this > > incorrect assumption, they stated that the "UUID" must be supplied to > > qemu in big endian format and must be byteswapped when writing it to > > guest memory. Their documentation says that they only do this for > > little endian guests, but this is not true of their implementation > > which byte swaps it for all guests. > > I don't think this is what section "Endian-ness Considerations" in > "docs/specs/vmgenid.txt" says. That text says that the *device* uses > little-endian format. That's independent of the endianness of *CPU* of > the particular guest architecture. > > So the byte-swapping in the QEMU code occurs unconditionally because > QEMU's UUID type is inherently big endian, and the device model in > question is fixed little endian. The guest CPU's endianness is > irrelevant as far as the device is concerned. > > If a BE guest CPU intends to read the generation ID and to interpret it > as a set of integers, then the guest CPU is supposed to byte-swap the > appropriate fields itself. > > > References > > I suggest adding two links in this section, namely to: > - docs/specs/vmgenid.txt > - hw/acpi/vmgenid.c Fair enough - I've made the changes you suggest (same URL as above). Thanks, Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-10-04 15:05 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <cover.1632900578.git.mprivozn@redhat.com> 2021-09-29 9:20 ` [PATCH 0/1] vmx: Fix <genid/> mapping Richard W.M. Jones 2021-09-29 9:33 ` Daniel P. Berrangé 2021-09-29 9:46 ` Richard W.M. Jones 2021-09-29 10:07 ` Daniel P. Berrangé 2021-09-29 10:24 ` Richard W.M. Jones 2021-09-29 10:24 ` Richard W.M. Jones 2021-09-29 9:57 ` Richard W.M. Jones 2021-09-29 10:10 ` Daniel P. Berrangé 2021-09-29 10:34 ` Richard W.M. Jones 2021-09-30 7:33 ` Richard W.M. Jones 2021-09-30 8:35 ` Laszlo Ersek 2021-09-30 8:47 ` Daniel P. Berrangé 2021-09-30 9:16 ` Richard W.M. Jones 2021-10-04 9:59 ` Richard W.M. Jones 2021-10-04 14:50 ` Laszlo Ersek 2021-10-04 14:59 ` Richard W.M. Jones
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).