From mboxrd@z Thu Jan 1 00:00:00 1970 From: "David S. Ahern" Subject: Re: strange guest slowness after some time Date: Thu, 19 Mar 2009 00:08:58 -0600 Message-ID: <49C1E17A.1090102@cisco.com> References: <49B29705.6000904@wpkg.org> <49BFF8D2.5080000@wpkg.org> <49C094D0.2070905@redhat.com> <200903191529.15081.rusty@rustcorp.com.au> <49C1D6A2.4070604@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Avi Kivity , Tomasz Chmielewski , Felix Leimbach , kvm@vger.kernel.org, Anthony Liguori To: Rusty Russell Return-path: Received: from sj-iport-4.cisco.com ([171.68.10.86]:7663 "EHLO sj-iport-4.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753117AbZCSGS0 (ORCPT ); Thu, 19 Mar 2009 02:18:26 -0400 In-Reply-To: <49C1D6A2.4070604@cisco.com> Sender: kvm-owner@vger.kernel.org List-ID: David S. Ahern wrote: > > Rusty Russell wrote: >> On Wednesday 18 March 2009 16:59:36 Avi Kivity wrote: >>> Tomasz Chmielewski wrote: >>>> virtio_net virtio0: id 64 is not a head! >> This means that qemu said "I've finished with buffer 64" and the guest didn't >> know anything about buffer 64. >> >> We should not lock up, tho networking is toast: I think that qemu got upset >> and that caused this as well as it to chew 100% cpu. >> >> I'll see if I can reproduce with kvm-84 userspace and 2.6.27 guests, 32-bit >> guests on a 64-bit AMD host. What's your kvm/qemu command line? >> > > I've hit this as well. > > Intel host, running RHEL5.3, x86_64 with KVM-81. > > Guest is RHEL4.7, 32-bit, with the virtio drivers from RHEL4.8 beta. > > Happens pretty darn quickly for me. > > david > Like I said, pretty darn quickly. More information for you. Command line (a few elements blurred) for this run (started about 15 minutes ago): kvm -localtime -no-reboot -m 3584 -smp 4 \ -drive file=/dev/cciss/c0d0,if=scsi,cache=off,boot=on \ -drive file=/dev/cciss/c0d1,if=scsi,cache=off,boot=off \ -net nic,vlan=0,macaddr=00:11:22:33:44:55,model=virtio \ -net tap,vlan=0,ifname=tap0,script=no,downscript=no \ -net nic,vlan=1,macaddr=00:12:34:56:78:1,model=virtio \ -net tap,vlan=1,ifname=tap1,script=no,downscript=no \ -usb -usbdevice tablet -mem-path /hugepages \ -pidfile /tmp/1.pid \ -monitor unix:/tmp/1,server,nowait \ -vnc :1 It does not take much network traffic for the network to lock up. In this case, the host shows 2 kvm threads spinning -- for vcpus 2,3. I have vcpus pinned to pcpus (vcpu0:pcpu0, etc). Backtrace for kvm, though nothing interesting: Thread 5 (Thread 0x43344940 (LWP 3153)): #0 0x00002b8af5088c77 in ioctl () from /lib64/libc.so.6 #1 0x0000000000530ece in kvm_run () #2 0x0000000000506529 in kvm_cpu_exec () #3 0x00000000005067c0 in ap_main_loop () #4 0x00002b8af46aa367 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b8af50900ad in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x43d45940 (LWP 3154)): #0 0x00002b8af5088c77 in ioctl () from /lib64/libc.so.6 #1 0x0000000000530ece in kvm_run () #2 0x0000000000506529 in kvm_cpu_exec () #3 0x00000000005067c0 in ap_main_loop () #4 0x00002b8af46aa367 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b8af50900ad in clone () from /lib64/libc.so.6 In the guest, I see 2 threads of 2 separate processes spinning on cpus 2,3. They appear to be spinning kernel side. Attempts to restart the network froze the guest console, and at this point the host shows 3 threads spinning away, though not at 100% cpu. The qemu monitor was able to push a system_powerdown event to the guest, and it showed signs of receiving it though it did not powerdown on its own. david > >> Thanks, >> Rusty. >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >