4.14.x - KVM guests crashing

4.14.x - KVM guests crashing

[Nikola Ciprich]

Hi,

not sure whether this is the best list to ask for that, but maybe
somebody from -stable interested people has hit similar problem..

if it's not appropriate list to ask/report, please let me know

since we started rolling to 4.14.x kernels, we've experienced multiple
KVM guest panics (with 4.14 on hosts)
We've first hit it with 4.14.12, yesterday I had similar problem with 4.14.16. 

backtraces can be observed here: 

http://nik.lbox.cz/download/guest-panic-host-4.14.12.png
http://nik.lbox.cz/download/guest-panic-host-4.14.16.png

sorry I don't have anything more complete

they are a bit different, but there are some similarities (ie double_fault)
also important to note, both are from different host systems. 

after reverting hosts to 4.4, crashes are gone.

my question is, did anyone hit similar problem? Is this something
know, maybe fixed in 4.14.17? (I'll give 4.14.17 try anyways I suppose)
(or latest git?)

could it be related to page table isolation enabled hosts?

btw hosts are running x86_64  centos 7, guests are either centos 6
or centos 7

If I could provide any further info, let me know

BR

nik


-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: 4.14.x - KVM guests crashing

[Greg KH]

On Fri, Feb 09, 2018 at 10:56:37AM +0100, Nikola Ciprich wrote:
> Hi,
> 
> not sure whether this is the best list to ask for that, but maybe
> somebody from -stable interested people has hit similar problem..
> 
> if it's not appropriate list to ask/report, please let me know
> 
> since we started rolling to 4.14.x kernels, we've experienced multiple
> KVM guest panics (with 4.14 on hosts)
> We've first hit it with 4.14.12, yesterday I had similar problem with 4.14.16. 

The KVM mailing list would probably be the best place
(kvm@vger.kernel.org), as we just backport existing upstream patches to
the stable kernels.

That being said, jumping from 4.4 to 4.14 is a very large leap (2 years
of development), it might be good if you can try something in the
middle?

Or, an earlier 4.14.y?  Try 4.14.3 or so and if that works, then it is
easier to use 'git bisect' to track down the offending patch.

thanks,

greg k-h

Re: 4.14.x - KVM guests crashing

[Nikola Ciprich]

Hi Greg,

> 
> The KVM mailing list would probably be the best place
> (kvm@vger.kernel.org), as we just backport existing upstream patches to
> the stable kernels.

OK, I'll report there, thanks.

> 
> That being said, jumping from 4.4 to 4.14 is a very large leap (2 years
> of development), it might be good if you can try something in the
> middle?
> 
> Or, an earlier 4.14.y?  Try 4.14.3 or so and if that works, then it is
> easier to use 'git bisect' to track down the offending patch.

yeah, I know. We always tend to stick to some stable  kernel branch for some time..

the bad thing is it takes quite long for the crash to occur (>1day at least)
so I'm keeping bisect as the last resort. I'll see if I get something
from the KVM maillist and possibly report needed patches here.

thanks

nik


> 
> thanks,
> 
> greg k-h
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: 4.14.x - KVM guests crashing

[Christoph Biedl]

Greg KH wrote...

> That being said, jumping from 4.4 to 4.14 is a very large leap (2 years
> of development), it might be good if you can try something in the
> middle?

My crashes started after switching from 4.9 to 4.14. Rumors have it they
started somewhere 4.13-ish and are gone 4.15. Will check if time
permits.

    Christoph

Re: 4.14.x - KVM guests crashing

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Hi,
> My crashes started after switching from 4.9 to 4.14. Rumors have it they
> started somewhere 4.13-ish and are gone 4.15. Will check if time
> permits.

it's been introduced by 4.14 and seems to be fixed by
2a266f23550be997d783f27e704b9b40c4010292 (no crash with it applied so 
far for me). Greg already has it queued for next 4.14.x release.

n.


> 
>     Christoph
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: 4.14.x - KVM guests crashing

[Greg KH]

On Sun, Feb 11, 2018 at 08:25:00AM +0100, Nikola Ciprich wrote:
> Hi,
> > My crashes started after switching from 4.9 to 4.14. Rumors have it they
> > started somewhere 4.13-ish and are gone 4.15. Will check if time
> > permits.
> 
> it's been introduced by 4.14 and seems to be fixed by
> 2a266f23550be997d783f27e704b9b40c4010292 (no crash with it applied so 
> far for me). Greg already has it queued for next 4.14.x release.

I do?  I don't see it in my queue anywhere.  Are you sure I've applied
it?

confused,

greg k-h

Re: 4.14.x - KVM guests crashing

[Nikola Ciprich]

> > it's been introduced by 4.14 and seems to be fixed by
> > 2a266f23550be997d783f27e704b9b40c4010292 (no crash with it applied so 
> > far for me). Greg already has it queued for next 4.14.x release.
> 
> I do?  I don't see it in my queue anywhere.  Are you sure I've applied
> it?

hmm, you've confirmed you'll queue it up on 9th of Feb:
https://www.spinics.net/lists/stable/msg214724.html

I've been testing 4.14.18 with that one on top since then (it's rock solid
now) and thought it got into the queue then.. my bad, sorry
about confusion..

nik


> 
> confused,
> 
> greg k-h
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: 4.14.x - KVM guests crashing

[Greg KH]

On Tue, Feb 13, 2018 at 04:59:21PM +0100, Nikola Ciprich wrote:
> > > it's been introduced by 4.14 and seems to be fixed by
> > > 2a266f23550be997d783f27e704b9b40c4010292 (no crash with it applied so 
> > > far for me). Greg already has it queued for next 4.14.x release.
> > 
> > I do?  I don't see it in my queue anywhere.  Are you sure I've applied
> > it?
> 
> hmm, you've confirmed you'll queue it up on 9th of Feb:
> https://www.spinics.net/lists/stable/msg214724.html
> 
> I've been testing 4.14.18 with that one on top since then (it's rock solid
> now) and thought it got into the queue then.. my bad, sorry
> about confusion..

Ugh, nevermind, I already queued this up.

Too many patches floating around right now, the merge window is the
worst for stable patches...

greg k-h

Re: 4.14.x - KVM guests crashing

[Christoph Biedl]

Nikola Ciprich wrote...

> it's been introduced by 4.14 and seems to be fixed by
> 2a266f23550be997d783f27e704b9b40c4010292 (no crash with it applied so 
> far for me).

Thanks for sharing, running 4.14.19 with this additional one made all my
problems go away, although they were a bit different: Using the "Resume" 
button in virt-manager after an earlier "Pause" caused the guest to
freeze and using 100% CPU - especially when the guest is the grml rescue
system, this happend almost always.

    Christoph