[Qemu-devel] opensuse 11.2 guest hangs after live migration with clocksource=kvm-clock

[Qemu-devel] opensuse 11.2 guest hangs after live migration with clocksource=kvm-clock

[Peter Lieven]

Hi all,

i would like to debug a problem that I encountered some time ago with 
opensuse 11.2 and also
with Ubuntu (karmic/lucid).

If I run an opensuse guest 64-bit and do not touch the clocksource 
settings the guest almost
everytime hangs after live migration at:

(gdb) thread apply all bt

Thread 2 (Thread 0x7f846782a950 (LWP 27356)):
#0  0x00007f8467d24cd7 in ioctl () from /lib/libc.so.6
#1  0x000000000042b945 in kvm_run (env=0x2468170)
    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:921
#2  0x000000000042cea2 in kvm_cpu_exec (env=0x2468170)
    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1651
#3  0x000000000042d62c in kvm_main_loop_cpu (env=0x2468170)
    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1893
#4  0x000000000042d76d in ap_main_loop (_env=0x2468170)
    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943
#5  0x00007f8468caa3ba in start_thread () from /lib/libpthread.so.0
#6  0x00007f8467d2cfcd in clone () from /lib/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f84692d96f0 (LWP 27353)):
#0  0x00007f8467d25742 in select () from /lib/libc.so.6
#1  0x000000000040c25a in main_loop_wait (timeout=1000)
    at /usr/src/qemu-kvm-0.12.4/vl.c:3994
#2  0x000000000042dcf1 in kvm_main_loop ()
    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:2126
#3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.4/vl.c:4212
#4  0x000000000041054b in main (argc=31, argv=0x7fffa91351c8,
    envp=0x7fffa91352c8) at /usr/src/qemu-kvm-0.12.4/vl.c:6252

If I run the same guest with kernel parameter clocksource=acpi_pm, the 
migration succeeds reliably.

The hosts runs:
 /kernel: /2.6.33.3, /bin: /qemu-kvm-0.12.4, /mod: /2.6.33.3

I invoke qemu-kvm with:
 /usr/bin/qemu-kvm-0.12.4  -net none  -drive 
file=/dev/sdb,if=ide,boot=on,cache=none,aio=native  -m 1024 -cpu 
qemu64,model_id='Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz'  
-monitor tcp:0:4001,server,nowait -vnc :1 -name 'test'  -boot 
order=dc,menu=on  -k de  -pidfile /var/run/qemu/vm-149.pid  -mem-path 
/hugepages -mem-prealloc  -rtc base=utc,clock=vm -usb -usbdevice tablet

The Guest is:
 OpenSuse 11.2 64-bit with Kernel 2.6.31.5-0.1-desktop #1 SMP PREEMPT 
2009-10-26 15:49:03 +0100 x86_64
 The clocksource automatically choosen is kvm-clock.

Feedback appreciated. I have observed the same problem with 0.12.2 and 
also with old binaries provided by Ubuntu Karmic (kvm-88).

BR,
Peter

Re: [Qemu-devel] opensuse 11.2 guest hangs after live migration with clocksource=kvm-clock

[Peter Lieven]

Peter Lieven wrote:
> Hi all,
>
> i would like to debug a problem that I encountered some time ago with 
> opensuse 11.2 and also
> with Ubuntu (karmic/lucid).
>
> If I run an opensuse guest 64-bit and do not touch the clocksource 
> settings the guest almost
> everytime hangs after live migration at:
>
> (gdb) thread apply all bt
>
> Thread 2 (Thread 0x7f846782a950 (LWP 27356)):
> #0  0x00007f8467d24cd7 in ioctl () from /lib/libc.so.6
> #1  0x000000000042b945 in kvm_run (env=0x2468170)
>    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:921
> #2  0x000000000042cea2 in kvm_cpu_exec (env=0x2468170)
>    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1651
> #3  0x000000000042d62c in kvm_main_loop_cpu (env=0x2468170)
>    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1893
> #4  0x000000000042d76d in ap_main_loop (_env=0x2468170)
>    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:1943
> #5  0x00007f8468caa3ba in start_thread () from /lib/libpthread.so.0
> #6  0x00007f8467d2cfcd in clone () from /lib/libc.so.6
> #7  0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 0x7f84692d96f0 (LWP 27353)):
> #0  0x00007f8467d25742 in select () from /lib/libc.so.6
> #1  0x000000000040c25a in main_loop_wait (timeout=1000)
>    at /usr/src/qemu-kvm-0.12.4/vl.c:3994
> #2  0x000000000042dcf1 in kvm_main_loop ()
>    at /usr/src/qemu-kvm-0.12.4/qemu-kvm.c:2126
> #3  0x000000000040c98c in main_loop () at 
> /usr/src/qemu-kvm-0.12.4/vl.c:4212
> #4  0x000000000041054b in main (argc=31, argv=0x7fffa91351c8,
>    envp=0x7fffa91352c8) at /usr/src/qemu-kvm-0.12.4/vl.c:6252
>
> If I run the same guest with kernel parameter clocksource=acpi_pm, the 
> migration succeeds reliably.
>
> The hosts runs:
> /kernel: /2.6.33.3, /bin: /qemu-kvm-0.12.4, /mod: /2.6.33.3
>
> I invoke qemu-kvm with:
> /usr/bin/qemu-kvm-0.12.4  -net none  -drive 
> file=/dev/sdb,if=ide,boot=on,cache=none,aio=native  -m 1024 -cpu 
> qemu64,model_id='Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz'  
> -monitor tcp:0:4001,server,nowait -vnc :1 -name 'test'  -boot 
> order=dc,menu=on  -k de  -pidfile /var/run/qemu/vm-149.pid  -mem-path 
> /hugepages -mem-prealloc  -rtc base=utc,clock=vm -usb -usbdevice tablet
>
> The Guest is:
> OpenSuse 11.2 64-bit with Kernel 2.6.31.5-0.1-desktop #1 SMP PREEMPT 
> 2009-10-26 15:49:03 +0100 x86_64
> The clocksource automatically choosen is kvm-clock.
>
> Feedback appreciated. I have observed the same problem with 0.12.2 and 
> also with old binaries provided by Ubuntu Karmic (kvm-88).
>
> BR,
> Peter
>
>
>
The same problem exists with 32-bit guest OS. Sometimes the guest kernel 
reports a Soft Lockup after some time with a really high value for the 
stuck time (in the order of billion seconds).
Is there maybe an overflow or a clock going backwards?