From mboxrd@z Thu Jan 1 00:00:00 1970 From: Doug Goldstein Subject: Re: HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2 Date: Sun, 15 Nov 2015 14:12:23 -0600 Message-ID: <5648E727.6080204@cardoe.com> References: <5643E68C.8090406@web2web.at> <564499B002000078000B43EE@prv-mh.provo.novell.com> <56448D9B.4090007@citrix.com> <5644A248.1060505@web2web.at> <5644C1CD.3020202@citrix.com> <56451A2B.9090706@web2web.at> <56459E5F02000078000B4944@prv-mh.provo.novell.com> <5645B6BC.6030603@citrix.com> <56467D44.5040205@web2web.at> <56479A6B.6080102@citrix.com> <5647CE57.50209@web2web.at> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8851196929492662756==" Return-path: In-Reply-To: <5647CE57.50209@web2web.at> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --===============8851196929492662756== Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Lflqd0VWgntrXJ5BuhNTqxsg7jau5ERVS" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Lflqd0VWgntrXJ5BuhNTqxsg7jau5ERVS Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 11/14/15 6:14 PM, Atom2 wrote: > Am 14.11.15 um 21:32 schrieb Andrew Cooper: >> On 14/11/2015 00:16, Atom2 wrote: >>> Am 13.11.15 um 11:09 schrieb Andrew Cooper: >>>> On 13/11/15 07:25, Jan Beulich wrote: >>>>>>>> On 13.11.15 at 00:00, wrote: >>>>>> Am 12.11.15 um 17:43 schrieb Andrew Cooper: >>>>>>> On 12/11/15 14:29, Atom2 wrote: >>>>>>>> Hi Andrew, >>>>>>>> thanks for your reply. Answers are inline further down. >>>>>>>> >>>>>>>> Am 12.11.15 um 14:01 schrieb Andrew Cooper: >>>>>>>>> On 12/11/15 12:52, Jan Beulich wrote: >>>>>>>>>>>>> On 12.11.15 at 02:08, wrote: >>>>>>>>>>> After the upgrade HVM domUs appear to no longer work - regard= less >>>>>>>>>>> of the >>>>>>>>>>> dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 ke= rnel); PV >>>>>>>>>>> domUs, however, work just fine as before on both dom0 kernels= =2E >>>>>>>>>>> >>>>>>>>>>> xl dmesg shows the following information after the first cras= hed HVM >>>>>>>>>>> domU which is started as part of the machine booting up: >>>>>>>>>>> [...] >>>>>>>>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by inva= lid guest >>>>>>>>>>> state (0). >>>>>>>>>>> (XEN) ************* VMCS Area ************** >>>>>>>>>>> (XEN) *** Guest State *** >>>>>>>>>>> (XEN) CR0: actual=3D0x0000000000000039, shadow=3D0x0000000000= 000011, >>>>>>>>>>> gh_mask=3Dffffffffffffffff >>>>>>>>>>> (XEN) CR4: actual=3D0x0000000000002050, shadow=3D0x0000000000= 000000, >>>>>>>>>>> gh_mask=3Dffffffffffffffff >>>>>>>>>>> (XEN) CR3: actual=3D0x0000000000800000, target_count=3D0 >>>>>>>>>>> (XEN) target0=3D0000000000000000, target1=3D000000000000= 0000 >>>>>>>>>>> (XEN) target2=3D0000000000000000, target3=3D000000000000= 0000 >>>>>>>>>>> (XEN) RSP =3D 0x0000000000006fdc (0x0000000000006fdc) RIP =3D= >>>>>>>>>>> 0x0000000100000000 (0x0000000100000000) >>>>>>>>>> Other than RIP looking odd for a guest still in non-paged prot= ected >>>>>>>>>> mode I can't seem to spot anything wrong with guest state. >>>>>>>>> odd? That will be the source of the failure. >>>>>>>>> >>>>>>>>> Out of long mode, the upper 32bit of %rip should all be zero, a= nd it >>>>>>>>> should not be possible to set any of them. >>>>>>>>> >>>>>>>>> I suspect that the guest has exited for emulation, and there ha= s been a >>>>>>>>> bad update to %rip. The alternative (which I hope is not the c= ase) is >>>>>>>>> that there is a hardware errata which allows the guest to accid= entally >>>>>>>>> get it self into this condition. >>>>>>>>> >>>>>>>>> Are you able to rerun with a debug build of the hypervisor? > [big snip] >>>>>>>> Now _without_ the debug USE flag, but with debug information in >>>>>>>> the binary (I used splitdebug), all is back to where the= problem >>>>>>>> started off (i.e. the system boots without issues until = such >>>>>>>> time it starts a HVM domU which then crashes; PV domUs a= re >>>>>>>> working). I have attached the latest "xl dmesg" output w= ith the >>>>>>>> timing information included. >>>>>>>> =20 >>> >>> I hope any of this makes sense to you. >>> >>> Again many thanks and best regards >>> >> >> Right - it would appear that the USE flag is definitely not what you >> wanted, and causes bad compilation for Xen. The do_IRQ disassembly >> you sent is a the result of disassembling a whole block of zeroes.=20 >> Sorry for leading you on a goose chase - the double faults will be the= >> product of bad compilation, rather than anything to do with your >> specific problem. > Hi Andrew, > there's absolutely no need to appologize as it is me who asked for help= > and you who generously stepped in and provided it. I really do > appreciate your help and it is for me, as the one seeking help, to > provide all the information you deem necessary and you ask for. >> However, the final log you sent (dmesg) is using a debug Xen, which is= >> what I was attempting to get you to do originally. > Next time I know better how to arrive at a debug XEN. It's all about > learning. >> We still observe that the VM ends up in 32bit non-paged mode but with >> an RIP with bit 32 set, which is an invalid state to be in. However, >> there was nothing particularly interesting in the extra log informatio= n. >> >> Please can you rerun with "hvm_debug=3D0xc3f", which will cause far mo= re >> logging to occur to the console while the HVM guest is running. That >> might show some hints. > I haven't done that yet - but please see my next paragraph. If you are > still interested in this, for whatever reason, I am clearly more than > happy to rerun with your suggested option and provide that information > as well. >> Also, the fact that this occurs just after starting SeaBIOS is >> interesting. As you have switched versions of Xen, you have also >> switched hvmloader, which contains the SeaBIOS binary embedded in it. = >> Would you be able to compile both 4.5.1 and 4.5.2 and switch the >> hvmloader binaries in use. It would be very interesting to see >> whether the failure is caused by the hvmloader binary or the >> hypervisor. (With `xl`, you can use >> firmware_override=3D"/full/path/to/firmware" to override the default >> hvmloader). > Your analysis was absolutely spot on. After re-thinking this for a > moment, I thought going down that route first would make a lot of sense= > as PV guests still do work and one of the differences to HVM domUs is > that the former do _not_ require SeaBIOS. Looking at my log files of > installed packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in > the relevant timeframe which obviously had not made it to the hvmloader= > of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS. >=20 > So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS > 1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed= > to strongly indicate that there indeed might be an issue with SeaBIOS a= s > this probably was the only variable that had changed from the original > install of xen-4.5.1. >=20 > My next step was to downgrade SeaBIOS to 1.7.5 and to re-compile > xen-4.5.1. Voila, the system was again up and running. While still > having SeaBIOS 1.7.5 installed, I also re-compiled xen-4.5.2 and ... yo= u > probably guessed it ... the problem was gone: The system boots up with > no issues and everything is fine again. >=20 > So in a nutshell: There seems to be a problem with SeaBIOS 1.8.2 > preventing HVM doamins from successfully starting up. I don't know what= > this is triggered from, if this is specific to my hardware or whether > something else in my environment is to blame. >=20 > In any case, I am again more than happy to provide data / run a few > tests should you wish to get to the grounds of this. >=20 > I do owe you a beer (or any other drink) should you ever be at my > location (i.e. Vienna, Austria). >=20 > Many thanks again for your analysis and your first class support. Xen > and their people absolutely rock! >=20 > Atom2 I'm a little late to the thread but can you send me (you can do it off-list if you'd like) the USE flags you used for xen, xen-tools and seabios? Also emerge --info. You can kill two birds with one stone by using emerge --info xen. I'm not too familiar with the xen ebuilds but I was pretty sure that xen-tools is what builds hvmloader and it downloads a copy of SeaBIOS and builds it so that it remains consistent. But obviously your experience shows otherwise. I'm looking at some ideas to improve SeaBIOS packaging on Gentoo and your info would be helpful. --=20 Doug Goldstein --Lflqd0VWgntrXJ5BuhNTqxsg7jau5ERVS Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0 iQJ8BAEBCgBmBQJWSOcqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNTM5MEQ2RTNFMTkyNzlCNzVDMzIwOTVB MkJDMDNEQzg3RUQxQkQ0AAoJEKK8A9yH7RvULegP/RLqyoXA3ADVYJgD+6KT8MKz Edb37IqCvM0Q7SE5TFZInVHAoql4rpw+ynl6CEEUB5UQzVCszG8AdXTqhXk2uGgB TsMU5010xQ1IF3gpkEQVw88R0lWoBuPcgMlOP14MbvFhFVmaXrSRWcIyKXyiTE/Q M6mHC4gDtGqQ4bbpix+3NrNhVhFkCvc9okGIhvUtJWvQjs3Fr1TAFVAscmfzikFi y3QGYP8l640DHROm6LT5ydvWY+eHIgTKcJL0v0WpgRJsl2xY1sjdF6arEyMjxTRY FnpzSduoqjhi5C69dxdA4GdslazKQz506CQX9//M+fn9V5PTYhDQ+P58w4PX2ffo AzHuDWg7JAlLgUbK4b2Cwu4C4GZJSJDMEGl9SfXhilxsZJzAR222c9mpAkekAXtb yfv55k1jS2rVw/cIqDvPzJXwtxnrSmCIU3k+PnWt9KO3yh4Fdd/ao2vOExOWzX7G 8IQOLrbRdrkIM2HL447mS5A7ynS4SbTqBqPyO7JPCQzmtKIDHkEU0MOd/9juABGL Wk/v8hVVTaXyUW0BZumRkeFXOldBnyhpNSzXy07eLdS+ADQzC4D6BYWNjBjazoil 4fYwV4dq7Ass3TB+7F8BoJ/0pgsza7Yxd+sZA4e+9FwznhlmXNgLd6WX4XVOep0Z YigXOHr0Vp3XsO7GQaXC =Inbf -----END PGP SIGNATURE----- --Lflqd0VWgntrXJ5BuhNTqxsg7jau5ERVS-- --===============8851196929492662756== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============8851196929492662756==--