fedora 10 x86_64 breakage under KVM, due to KVM_CLOCK

fedora 10 x86_64 breakage under KVM, due to KVM_CLOCK

[Lennert Buytenhek]

(please CC, not on the list)

Hi all,

Fedora 10 randomly hangs for me when run under KVM, whereas (unpatched)
F9 works fine.  When this happens, kvm_stat shows that the VM is seeing
1000 IRQ exits per second, but nothing else seems to happen.  The latest
F9 update kernel seems to be bust as well.  (I noticed that the KVM guest
support status page says that "Alexey E." is seeing the same thing.)

I tried about two dozen different kernel packages from koji, and it turns
out that kernel-2.6.26-0.122.rc9.git4.fc10 is the last known good kernel,
and kernel-2.6.26-0.124.rc9.git5.fc10 is the first kernel that's broken.

The F10 instability appears since this commit (attached below as well):

	http://fedora.gitbits.net/?p=kernel;a=commit;h=a5991f36968f44d4d3c64fc5aaa285a21de1ba54

I built three new kernels from the latest 2.6.27-170.2.56.fc10 update
in Fedora 10:
1. One with PARAVIRT and all associated options disabled entirely.
2. One with only KVM_GUEST disabled.
3. One with only KVM_CLOCK disabled (and paravirt and KVM_GUEST enabled).

Kernel (1) is stable.  Kernel (2) is unstable like the unpatched F10
kernels are, in that it randomly locks up.  Kernel (3) is stable as
well, which suggests that KVM_CLOCK is what's causing the lockups.

I tried booting the original F10 update kernel with no-kvmclock on
the command line as well, and that makes things stable as well.

I tried with the latest F10 updates-testing kernel (2.6.29.1-30.fc10),
and that shows the same thing: with no-kvmclock it works fine, otherwise
it hangs randomly.

Running KVM 84 on x86_64 CentOS 5.3, with the kvm rpms from lfarkas.org.
The host CPU is a Q6600 Core 2 Quad.

Any ideas?


thanks,
Lennert


commit a5991f36968f44d4d3c64fc5aaa285a21de1ba54
Author: davej <davej>
Date:   Wed Jul 9 04:54:02 2008 +0000

    Reenable paravirt on x86-64.

diff --git a/config-x86_64-generic b/config-x86_64-generic
index e0c1e61..a2dd13c 100644
--- a/config-x86_64-generic
+++ b/config-x86_64-generic
@@ -254,7 +254,11 @@ CONFIG_INTEL_IOATDMA=m
 
 CONFIG_SENSORS_I5K_AMB=m
 
-# CONFIG_PARAVIRT_GUEST is not set
+CONFIG_PARAVIRT_GUEST=y
+CONFIG_KVM_CLOCK=y
+CONFIG_KVM_GUEST=y
+CONFIG_PARAVIRT=y
+
 # CONFIG_COMPAT_VDSO is not set
 CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
 # CONFIG_DEBUG_PER_CPU_MAPS is not set
diff --git a/kernel.spec b/kernel.spec
index cd4b749..ff4c12a 100644
--- a/kernel.spec
+++ b/kernel.spec
@@ -21,7 +21,7 @@ Summary: The Linux kernel
 # works out to the offset from the rebase, so it doesn't get too ginormous.
 #
 %define fedora_cvs_origin 623
-%define fedora_build %(R="$Revision: 1.744 $"; R="${R%% \$}"; R="${R##: 1.}"; expr $R - %{fedora_cvs_origin})
+%define fedora_build %(R="$Revision: 1.745 $"; R="${R%% \$}"; R="${R##: 1.}"; expr $R - %{fedora_cvs_origin})
 
 # base_sublevel is the kernel version we're starting with and patching
 # on top of -- for example, 2.6.22-rc7-git1 starts with a 2.6.21 base,
@@ -1794,6 +1794,9 @@ fi
 %kernel_variant_files -a /%{image_install_path}/xen*-%{KVERREL}.xen -e /etc/ld.so.conf.d/kernelcap-%{KVERREL}.xen.conf %{with_xen} xen
 
 %changelog
+* Wed Jul 09 2008 Dave Jones <davej@redhat.com>
+- Reenable paravirt on x86-64.
+
 * Tue Jul  8 2008 Roland McGrath <roland@redhat.com>
 - new bleeding-edge utrace code

Re: fedora 10 x86_64 breakage under KVM, due to KVM_CLOCK

[Avi Kivity]

Lennert Buytenhek wrote:
> (please CC, not on the list)
>
> Hi all,
>
> Fedora 10 randomly hangs for me when run under KVM, whereas (unpatched)
> F9 works fine.  When this happens, kvm_stat shows that the VM is seeing
> 1000 IRQ exits per second, but nothing else seems to happen.  The latest
> F9 update kernel seems to be bust as well.  (I noticed that the KVM guest
> support status page says that "Alexey E." is seeing the same thing.)
>
> I tried about two dozen different kernel packages from koji, and it turns
> out that kernel-2.6.26-0.122.rc9.git4.fc10 is the last known good kernel,
> and kernel-2.6.26-0.124.rc9.git5.fc10 is the first kernel that's broken.
>
> The F10 instability appears since this commit (attached below as well):
>
> 	http://fedora.gitbits.net/?p=kernel;a=commit;h=a5991f36968f44d4d3c64fc5aaa285a21de1ba54
>
> I built three new kernels from the latest 2.6.27-170.2.56.fc10 update
> in Fedora 10:
> 1. One with PARAVIRT and all associated options disabled entirely.
> 2. One with only KVM_GUEST disabled.
> 3. One with only KVM_CLOCK disabled (and paravirt and KVM_GUEST enabled).
>
> Kernel (1) is stable.  Kernel (2) is unstable like the unpatched F10
> kernels are, in that it randomly locks up.  Kernel (3) is stable as
> well, which suggests that KVM_CLOCK is what's causing the lockups.
>
> I tried booting the original F10 update kernel with no-kvmclock on
> the command line as well, and that makes things stable as well.
>
> I tried with the latest F10 updates-testing kernel (2.6.29.1-30.fc10),
> and that shows the same thing: with no-kvmclock it works fine, otherwise
> it hangs randomly.
>
> Running KVM 84 on x86_64 CentOS 5.3, with the kvm rpms from lfarkas.org.
> The host CPU is a Q6600 Core 2 Quad.
>
> Any ideas?
>   

This is a known issue.  While kvm-84 fixed kvmclock, an issue remained 
with cpu frequency scaling on older host kernels (including RHEL 5 / 
CentOS 5).  kvm-85 (to be released shortly) contains a fix (b395d156477a 
in kvm-userspace.git).


-- 
error compiling committee.c: too many arguments to function

Re: fedora 10 x86_64 breakage under KVM, due to KVM_CLOCK

[Lennert Buytenhek]

On Mon, Apr 20, 2009 at 02:56:12PM +0300, Avi Kivity wrote:

> >Fedora 10 randomly hangs for me when run under KVM, whereas (unpatched)
> >F9 works fine.  When this happens, kvm_stat shows that the VM is seeing
> >1000 IRQ exits per second, but nothing else seems to happen.  The latest
> >F9 update kernel seems to be bust as well.  (I noticed that the KVM guest
> >support status page says that "Alexey E." is seeing the same thing.)
> >
> >I tried about two dozen different kernel packages from koji, and it turns
> >out that kernel-2.6.26-0.122.rc9.git4.fc10 is the last known good kernel,
> >and kernel-2.6.26-0.124.rc9.git5.fc10 is the first kernel that's broken.
> >
> >The F10 instability appears since this commit (attached below as well):
> >
> >	http://fedora.gitbits.net/?p=kernel;a=commit;h=a5991f36968f44d4d3c64fc5aaa285a21de1ba54
> >
> >I built three new kernels from the latest 2.6.27-170.2.56.fc10 update
> >in Fedora 10:
> >1. One with PARAVIRT and all associated options disabled entirely.
> >2. One with only KVM_GUEST disabled.
> >3. One with only KVM_CLOCK disabled (and paravirt and KVM_GUEST enabled).
> >
> >Kernel (1) is stable.  Kernel (2) is unstable like the unpatched F10
> >kernels are, in that it randomly locks up.  Kernel (3) is stable as
> >well, which suggests that KVM_CLOCK is what's causing the lockups.
> >
> >I tried booting the original F10 update kernel with no-kvmclock on
> >the command line as well, and that makes things stable as well.
> >
> >I tried with the latest F10 updates-testing kernel (2.6.29.1-30.fc10),
> >and that shows the same thing: with no-kvmclock it works fine, otherwise
> >it hangs randomly.
> >
> >Running KVM 84 on x86_64 CentOS 5.3, with the kvm rpms from lfarkas.org.
> >The host CPU is a Q6600 Core 2 Quad.
> >
> >Any ideas?
> 
> This is a known issue.  While kvm-84 fixed kvmclock, an issue remained 
> with cpu frequency scaling on older host kernels (including RHEL 5 / 
> CentOS 5).  kvm-85 (to be released shortly) contains a fix (b395d156477a 
> in kvm-userspace.git).

Seems to work fine, thanks!