[Qemu-devel] [Bug 607204] [NEW] New qemu instances often cannot be started if host system is under load

* [Qemu-devel] [Bug 607204] [NEW] New qemu instances often cannot be started if host system is under load
@ 2010-07-19 12:38 Guido Winkelmann
  2010-07-19 12:38 ` [Qemu-devel] [Bug 607204] " Guido Winkelmann
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Guido Winkelmann @ 2010-07-19 12:38 UTC (permalink / raw)
  To: qemu-devel

Public bug reported:

I've got a problem where I cannot start any new VMs with qemu-kvm if the
host machine is under high CPU load. The problem is not 100%
reproducible (it works sometimes), but under load conditions, it happens
most of the time - roughly 95%.

I'm usually using libvirt to start and stop KVM VMs. When using virsh to
start a new VM under those conditions, the output looks like this:

virsh # start testserver-a
error: Failed to start domain testserver-a
error: monitor socket did not show up.: Connection refused

(There is a very long wait after the command has been sent until the
error message shows up.)

This is (an example of) the command line that libvirtd uses to start up
qemu:

----- snip -----
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.12 -enable-kvm -m 256 -smp 1,sockets=1,cores=1,threads=1 -name testserver-a -uuid 7cbb3665-4d58-86b8-ce8f-20541995a99c -nodefaults -chardev socket,id=monitor,path=/usr/local/var/lib/libvirt/qemu/testserver-a.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -no-acpi -boot c -device lsi,id=scsi0,bus=pci.0,addr=0x7 -drive file=/data/testserver-a-system.img,if=none,id=drive-scsi0-0-1,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=1,drive=drive-scsi0-0-1,id=scsi0-0-1 -drive file=/data/testserver-a-data1.img,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/data/testserver-a-data2.img,if=none,id=drive-virtio-disk2 -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/data/gentoo-install-amd64-minimal-20100408.iso,if=none,media=cdrom,id=drive-ide0-0-0,readonly=on -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/data/testserver-a_configfloppy.img,if=none,id=drive-fdc0-0-0 -global isa-fdc.driveA=drive-fdc0-0-0 -device e1000,vlan=0,id=net0,mac=52:54:00:84:6d:69,bus=pci.0,addr=0x6 -net tap,fd=24,vlan=0,name=hostnet0 -usb -vnc 127.0.0.1:1,password -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
----- snip -----

Copy-pasting this to a commandline on the host to start qemu manually
leads to a non-functional qemu process that "just sits there" with
nothing happening. The monitor socket
/usr/local/var/lib/libvirt/qemu/testserver-a.monitor will, indeed, not
show up.

I've tried starting qemu with the same commandline but without the
parameters for redirecting the monitor to a socket, without the fd
parameter for the network interface and without the vnc parameter. This
resulted in a black window with the title "QEMU (testserver-a)
[Stopped]". I could not access the monitor console in graphical mode
either. When I press Ctrl-Alt-2 in graphical mode to access the monitor
console, qemu will sometimes (but not always) crash with a segfault
about 2 seconds after.

Some experimentation I've done suggests that this problem only happens
if the high cpu load is caused by another qemu process, not if it is
caused by something else running on the machine.

The bug appears much less often if I leave off the -nodefaults
parameter.

The bug will still appear if I start qemu as qemu-system-x86_64 instead
of qemu-kvm and replace the -enable-kvm parameter with -no-kvm.

The host machine I'm running this on has got 16 cores in total. It looks
like it is sufficient for this bug to surface if at least one of these
cores is brought to near 100% use by a qemu process.

The version of qemu I'm using is qemu-kvm 0.12.4, built from source.
Libvirt is version 0.8.1, built from source as well. The host OS is
Fedora 12. The Kernel version is 2.6.32.12-115.fc12.x86_64.

Attached is an strace of attempting to start qemu which I hope will help
someone with a better understanding of qemu's internals see what's
actually going on there.

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
New qemu instances often cannot be started if host system is under load
https://bugs.launchpad.net/bugs/607204
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.

Status in QEMU: New

Bug description:
I've got a problem where I cannot start any new VMs with qemu-kvm if the host machine is under high CPU load. The problem is not 100% reproducible (it works sometimes), but under load conditions, it happens most of the time - roughly 95%.

I'm usually using libvirt to start and stop KVM VMs. When using virsh to start a new VM under those conditions, the output looks like this:

virsh # start testserver-a
error: Failed to start domain testserver-a
error: monitor socket did not show up.: Connection refused

(There is a very long wait after the command has been sent until the error message shows up.)

This is (an example of) the command line that libvirtd uses to start up qemu:

----- snip -----
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc-0.12 -enable-kvm -m 256 -smp 1,sockets=1,cores=1,threads=1 -name testserver-a -uuid 7cbb3665-4d58-86b8-ce8f-20541995a99c -nodefaults -chardev socket,id=monitor,path=/usr/local/var/lib/libvirt/qemu/testserver-a.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -no-acpi -boot c -device lsi,id=scsi0,bus=pci.0,addr=0x7 -drive file=/data/testserver-a-system.img,if=none,id=drive-scsi0-0-1,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=1,drive=drive-scsi0-0-1,id=scsi0-0-1 -drive file=/data/testserver-a-data1.img,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/data/testserver-a-data2.img,if=none,id=drive-virtio-disk2 -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/data/gentoo-install-amd64-minimal-20100408.iso,if=none,media=cdrom,id=drive-ide0-0-0,readonly=on -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/data/testserver-a_configfloppy.img,if=none,id=drive-fdc0-0-0 -global isa-fdc.driveA=drive-fdc0-0-0 -device e1000,vlan=0,id=net0,mac=52:54:00:84:6d:69,bus=pci.0,addr=0x6 -net tap,fd=24,vlan=0,name=hostnet0 -usb -vnc 127.0.0.1:1,password -k de -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
----- snip -----

Copy-pasting this to a commandline on the host to start qemu manually leads to a non-functional qemu process that "just sits there" with nothing happening. The monitor socket /usr/local/var/lib/libvirt/qemu/testserver-a.monitor will, indeed, not show up.

I've tried starting qemu with the same commandline but without the parameters for redirecting the monitor to a socket, without the fd parameter for the network interface and without the vnc parameter. This resulted in a black window with the title "QEMU (testserver-a) [Stopped]". I could not access the monitor console in graphical mode either. When I press Ctrl-Alt-2 in graphical mode to access the monitor console, qemu will sometimes (but not always) crash with a segfault about 2 seconds after.

Some experimentation I've done suggests that this problem only happens if the high cpu load is caused by another qemu process, not if it is caused by something else running on the machine.

The bug appears much less often if I leave off the -nodefaults parameter.

The bug will still appear if I start qemu as qemu-system-x86_64 instead of qemu-kvm and replace the -enable-kvm parameter with -no-kvm.

The host machine I'm running this on has got 16 cores in total. It looks like it is sufficient for this bug to surface if at least one of these cores is brought to near 100% use by a qemu process.

The version of qemu I'm using is qemu-kvm 0.12.4, built from source. Libvirt is version 0.8.1, built from source as well. The host OS is Fedora 12. The Kernel version is 2.6.32.12-115.fc12.x86_64.

Attached is an strace of attempting to start qemu which I hope will help someone with a better understanding of qemu's internals see what's actually going on there.

^ permalink raw reply	[flat|nested] 5+ messages in thread