* "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work)
[not found] ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com>
@ 2016-07-20 23:20 ` Marek Marczykowski-Górecki
[not found] ` <20160720232009.GA24847@mail-itl>
1 sibling, 0 replies; 6+ messages in thread
From: Marek Marczykowski-Górecki @ 2016-07-20 23:20 UTC (permalink / raw)
To: Massimo Colombi; +Cc: xen-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote:
> I retried (it's not the first time) to regenerate new savefile, but DispVM
> doesn't work.
> I attach the results.
For me it looks like a bug in savefile handling. Moving to xen-devel. Any
idea?
Background info: it's about restoring a domain through libvirt->libxl
(virsh restore equivalent). Xen 4.6.1. Full error:
> 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error
> 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error
> 2016-07-20 14:23:01 CEST libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: restoring domain: Successo
> 2016-07-20 14:23:01 CEST libxl: error: libxl_create.c:1145:domcreate_rebuild_done: cannot (re-)build domain: -3
If relevant, here is fragment of /proc/cpuinfo (just one core):
processor : 0
vendor_id : AuthenticAMD
cpu family : 22
model : 48
model name : AMD A8-6410 APU with AMD Radeon R5 Graphics
stepping : 1
microcode : 0x7030105
cpu MHz : 1996.290
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic mca cmov pat clflush
mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm constant_tsc rep_good nopl nonstop_tsc
extd_apicid eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
rdrand hypervisor lahf_lm
cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch
perfctr_nb bpext perfctr_l2
arat cpb hw_pstate vmmcall bmi1 xsaveopt
bugs : fxsave_leak
bogomips : 3992.58
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [12] [13]
> Best regards,
> Massimo
>
> On 07/20/2016 01:37 PM, Marek Marczykowski-Górecki wrote:
> > Try `qvm-create-default-dvm --default-template` to regenerate new savefile.
> > It looks like your savefile is somehow broken.
>
> [user@dom0 logs]$ qvm-create-default-dvm --default-template
> --> Creating volatile image: /var/lib/qubes/appvms/fedora-23-dvm/volatile.img...
> --> Loading the VM (type = AppVM)...
> --> Starting Qubes DB...
> --> Setting Qubes DB info for the VM...
> --> Updating firewall rules...
> --> Starting the VM...
> --> Starting Qubes GUId...
> Connecting to VM's GUI agent: ..................connected
> Waiting for DVM fedora-23-dvm ...
> /qubes-used-mem
> Disco scollegato correttamente
>
> DVM boot complete, memory used=303380. Saving image...
>
>
>
> Domain fedora-23-dvm saved to /var/lib/qubes/appvms/fedora-23-dvm/dvm-savefile
>
> DVM savefile created successfully.
> [user@dom0 logs]$ echo xterm | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell dom0 DEFAULT red
> time=1469017378.68, qfile-daemon-dvm init
> time=1469017378.92, creating DispVM
> time=1469017380.48, collection loaded
> time=1469017380.49, VM created
> time=1469017380.59, VM starting
> time=1469017380.6, creating config file
> time=1469017380.86, calling restore
> Traceback (most recent call last):
> File "/usr/lib/qubes/qfile-daemon-dvm", line 200, in <module>
> main()
> File "/usr/lib/qubes/qfile-daemon-dvm", line 188, in main
> dispvm = qfile.get_dvm()
> File "/usr/lib/qubes/qfile-daemon-dvm", line 150, in get_dvm
> return self.do_get_dvm()
> File "/usr/lib/qubes/qfile-daemon-dvm", line 103, in do_get_dvm
> dispvm.start()
> File "/usr/lib64/python2.7/site-packages/qubes/modules/01QubesDisposableVm.py", line 193, in start
> domain_config, libvirt.VIR_DOMAIN_SAVE_PAUSED)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4405, in restoreFlags
> if ret == -1: raise libvirtError ('virDomainRestoreFlags() failed', conn=self)
> libvirt.libvirtError: internal error: libxenlight failed to restore domain 'disp1'
>
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJXkActAAoJENuP0xzK19csHZ4IAJF35BwBKbKzNFo0yOcTJxng
2NqVIUmHg3hgVZuNdkoNm09L7f5yIUai+0AqsFvM8BbPmwC9C6EBcu7FICMub/lT
NUbfyf4rW5YRuEVG+OB7+Zge4mz3++kb1cLqobn2vA5Z9ayCDW32Yq5yYFff9zd7
/qMtjuj35oG25fKEs1PIZbDtMkdnq2ef1rg7KDj695SpDSt0g3AadtTjnJVh7f2V
v7K6aUKR6O2xcHWADWhqxLNaKEBA8cd2yHeGYmbQwD9cICqQNVFwH/jIHWsAqGun
d/IKvRetZnLajFm42X7862UIgQWE1qTpqDXV6OCWT/KNh/cL4SLwn+0RW8RbfSQ=
=gdks
-----END PGP SIGNATURE-----
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
[not found] ` <20160720232009.GA24847@mail-itl>
@ 2016-07-21 0:10 ` Andrew Cooper
2016-07-21 8:39 ` Massimo Colombi
0 siblings, 1 reply; 6+ messages in thread
From: Andrew Cooper @ 2016-07-21 0:10 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, Massimo Colombi; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 2461 bytes --]
On 21/07/2016 00:20, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote:
>> I retried (it's not the first time) to regenerate new savefile, but DispVM
>> doesn't work.
>> I attach the results.
> For me it looks like a bug in savefile handling. Moving to xen-devel. Any
> idea?
> Background info: it's about restoring a domain through libvirt->libxl
> (virsh restore equivalent). Xen 4.6.1. Full error:
>
>> 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error
>> 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error
[EDIT] While writing the explanation below, I have spotted the bug.
The restore side found a record in the stream of type X86_PV_VCPU_MSRS
and 8 bytes long.
The X86_PV_VCPU_MSRS record is an 8 byte header, followed by 1 or more
bytes which are treated as an opaque blob, and handed back to Xen. As
such, 8 bytes is insufficient to contain a nonzero sized payload.
Is it possible to do an `xl save` equivalent on the domain, and run
tools/python/scripts/verify-stream-v2 against the resulting file? That
should identify whether it is a malformed X86_PV_VCPU_MSRS record but
otherwise intact stream, or whether the stream becomes corrupted elsewhere?
If not, another line if inquiry would be to instrument
tools/libxc/xc_sr_save_x86_pv.c write_one_vcpu_msrs() to identify what
XEN_DOMCTL_get_vcpu_msrs is returning on the source side (although this
function does deliberately check for a zero-sized payload and omit the
record).
The hypervisor side implementation is in xen/arch/x86/domctl.c at the
XEN_DOMCTL_get_vcpu_msrs case label.
Re-reading the write_one_vcpu_msrs(), there is an error. We first query
Xen for the maximum number of MSRs it might potentially give to us,
which will return 4 on this hardware. We then actually ask for the MSR
content, but Xen only writes non-zero MSRs into payload.
A domain which isn't using Debug Extensions will return 4 for the first
query (maximum number of MSRs), then 0 for the second query (actual MSR
content), and write a X86_PV_VCPU_MSRS record with 0 payload into the
stream, which the receiving side is objecting to.
I will make a patch series tomorrow to address this issue. I think
there are similar issues for other records as well.
~Andrew
[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 834 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
2016-07-21 0:10 ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper
@ 2016-07-21 8:39 ` Massimo Colombi
2016-07-21 8:53 ` Andrew Cooper
0 siblings, 1 reply; 6+ messages in thread
From: Massimo Colombi @ 2016-07-21 8:39 UTC (permalink / raw)
To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel
This is the output of verify-stream-v2:
[user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
xc: info: Saving domain 4, type x86 PV
xc: Frames: 912384/912384 100%
xc: End of stream: 0/0 0%
[user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
Stream Error:
Traceback (most recent call last):
File "./verify-stream-v2", line 82, in read_stream
VerifyLibxl(info, stream_read).verify()
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
line 82, in verify
while self.verify_record() != REC_TYPE_end:
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
line 136, in verify_record
record_verifiers[rtype](self, content[:length])
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
line 155, in verify_record_libxc_context
VerifyLibxc(self.info, self.read).verify()
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
line 132, in verify
while self.verify_record() != REC_TYPE_end:
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
line 227, in verify_record
record_verifiers[rtype](self, content[:length])
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
line 429, in <lambda>
VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
line 323, in verify_record_x86_pv_vcpu_generic
" bytes long" % (name, minsz))
RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
On 07/21/2016 02:10 AM, Andrew Cooper wrote:
> Is it possible to do an `xl save` equivalent on the domain, and run
> tools/python/scripts/verify-stream-v2 against the resulting file? That
> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
> otherwise intact stream, or whether the stream becomes corrupted elsewhere?
Thanks for your explanation of the bug.
Best regards,
Massimo
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
2016-07-21 8:39 ` Massimo Colombi
@ 2016-07-21 8:53 ` Andrew Cooper
2016-07-21 9:21 ` Massimo Colombi
2016-07-22 15:07 ` Massimo Colombi
0 siblings, 2 replies; 6+ messages in thread
From: Andrew Cooper @ 2016-07-21 8:53 UTC (permalink / raw)
To: Massimo Colombi, Marek Marczykowski-Górecki; +Cc: xen-devel
On 21/07/2016 09:39, Massimo Colombi wrote:
> This is the output of verify-stream-v2:
>
> [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
> Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
> xc: info: Saving domain 4, type x86 PV
> xc: Frames: 912384/912384 100%
> xc: End of stream: 0/0 0%
>
> [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
> Stream Error:
> Traceback (most recent call last):
> File "./verify-stream-v2", line 82, in read_stream
> VerifyLibxl(info, stream_read).verify()
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 82, in verify
> while self.verify_record() != REC_TYPE_end:
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 136, in verify_record
> record_verifiers[rtype](self, content[:length])
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 155, in verify_record_libxc_context
> VerifyLibxc(self.info, self.read).verify()
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 132, in verify
> while self.verify_record() != REC_TYPE_end:
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 227, in verify_record
> record_verifiers[rtype](self, content[:length])
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 429, in <lambda>
> VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 323, in verify_record_x86_pv_vcpu_generic
> " bytes long" % (name, minsz))
> RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
>
>
> On 07/21/2016 02:10 AM, Andrew Cooper wrote:
>> Is it possible to do an `xl save` equivalent on the domain, and run
>> tools/python/scripts/verify-stream-v2 against the resulting file? That
>> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
>> otherwise intact stream, or whether the stream becomes corrupted
>> elsewhere?
>
> Thanks for your explanation of the bug.
I should also improve the reported error message.
Do you mind rerunning with an extra -v to dump a list of the records
found in the stream?
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
2016-07-21 8:53 ` Andrew Cooper
@ 2016-07-21 9:21 ` Massimo Colombi
2016-07-22 15:07 ` Massimo Colombi
1 sibling, 0 replies; 6+ messages in thread
From: Massimo Colombi @ 2016-07-21 9:21 UTC (permalink / raw)
To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 171 bytes --]
OK. I attach verbose output.
On 07/21/2016 10:53 AM, Andrew Cooper wrote:
> Do you mind rerunning with an extra -v to dump a list of the records
> found in the stream?
[-- Attachment #2: stream.txt --]
[-- Type: text/plain, Size: 2047 bytes --]
[user@dom0 scripts]$ ./verify-stream-v2 -f xl -v -i /tmp-savefile
Processed xl header
Libxl Header: little endian
Libxl Record: Libxc context, length 0
Libxc Image Header: little endian
Domain Header: x86 PV from Xen 4.6
Libxc Record: x86 PV info, length 8
64bit guest, 4 levels of pagetables
Libxc Record: x86 PV P2M frames, length 14264
Start pfn 0x0, End 0xdebff
Squashed 891 Page Data records together
Libxc Record: TSC info, length 24
Mode 2, 0 kHz, 0 ns, incarnation 1
Libxc Record: Shared info, length 4096
Libxc Record: x86 PV vcpu basic, length 5176
vcpu0 basic context, 5168 bytes
Libxc Record: x86 PV vcpu extended, length 64
vcpu0 extended context, 56 bytes
Libxc Record: x86 PV vcpu xsave, length 856
vcpu0 xsave context, 848 bytes
Libxc Record: x86 PV vcpu msrs, length 8
Stream Error:
Traceback (most recent call last):
File "./verify-stream-v2", line 82, in read_stream
VerifyLibxl(info, stream_read).verify()
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 82, in verify
while self.verify_record() != REC_TYPE_end:
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 136, in verify_record
record_verifiers[rtype](self, content[:length])
File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 155, in verify_record_libxc_context
VerifyLibxc(self.info, self.read).verify()
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 132, in verify
while self.verify_record() != REC_TYPE_end:
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 227, in verify_record
record_verifiers[rtype](self, content[:length])
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 429, in <lambda>
VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 323, in verify_record_x86_pv_vcpu_generic
" bytes long" % (name, minsz))
RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
[-- Attachment #3: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
2016-07-21 8:53 ` Andrew Cooper
2016-07-21 9:21 ` Massimo Colombi
@ 2016-07-22 15:07 ` Massimo Colombi
1 sibling, 0 replies; 6+ messages in thread
From: Massimo Colombi @ 2016-07-22 15:07 UTC (permalink / raw)
To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel
The patches (that are attached to "[PATCH RFC 0/4] Fix issues with
zero-length records in migration v2") work!
I patched locally qubes-builder to import your patches and recreate rpm
files. These patches also work on Xen 4.6.1.
Best regards,
Massimo
On 07/21/2016 10:53 AM, Andrew Cooper wrote:
> On 21/07/2016 09:39, Massimo Colombi wrote:
>> This is the output of verify-stream-v2:
>>
>> [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
>> Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
>> xc: info: Saving domain 4, type x86 PV
>> xc: Frames: 912384/912384 100%
>> xc: End of stream: 0/0 0%
>>
>> [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
>> Stream Error:
>> Traceback (most recent call last):
>> File "./verify-stream-v2", line 82, in read_stream
>> VerifyLibxl(info, stream_read).verify()
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 82, in verify
>> while self.verify_record() != REC_TYPE_end:
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 136, in verify_record
>> record_verifiers[rtype](self, content[:length])
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 155, in verify_record_libxc_context
>> VerifyLibxc(self.info, self.read).verify()
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 132, in verify
>> while self.verify_record() != REC_TYPE_end:
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 227, in verify_record
>> record_verifiers[rtype](self, content[:length])
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 429, in <lambda>
>> VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
>> File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 323, in verify_record_x86_pv_vcpu_generic
>> " bytes long" % (name, minsz))
>> RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
>>
>>
>> On 07/21/2016 02:10 AM, Andrew Cooper wrote:
>>> Is it possible to do an `xl save` equivalent on the domain, and run
>>> tools/python/scripts/verify-stream-v2 against the resulting file? That
>>> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
>>> otherwise intact stream, or whether the stream becomes corrupted
>>> elsewhere?
>> Thanks for your explanation of the bug.
> I should also improve the reported error message.
>
> Do you mind rerunning with an extra -v to dump a list of the records
> found in the stream?
>
> ~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-07-22 15:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <nmncv2$m3$1@ger.gmane.org>
[not found] ` <20160720113752.GT1161@mail-itl>
[not found] ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com>
2016-07-20 23:20 ` "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work) Marek Marczykowski-Górecki
[not found] ` <20160720232009.GA24847@mail-itl>
2016-07-21 0:10 ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper
2016-07-21 8:39 ` Massimo Colombi
2016-07-21 8:53 ` Andrew Cooper
2016-07-21 9:21 ` Massimo Colombi
2016-07-22 15:07 ` Massimo Colombi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.