* [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads
@ 2015-12-08 0:44 Neil McGill
2015-12-10 8:47 ` Stefan Hajnoczi
0 siblings, 1 reply; 3+ messages in thread
From: Neil McGill @ 2015-12-08 0:44 UTC (permalink / raw)
To: qemu-devel; +Cc: Rich Wellum (rwellum)
Has anyone seen anything like this ? all 4 qemu threads are stuck on a
pthread condition
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
I'll try and get the qemu source built on this machine to debug and get
some useful symbols out of gdb, but just wondering if there is a known
issue like this on:
Linux virl 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
tx
neil
(gdb) where
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f7a21036909 in ?? ()
#4 0x00007f7a20fc1fdb in ?? ()
#5 0x00007f7a20d153fe in ?? ()
#6 0x00007f7a17176ec5 in __libc_start_main (main=0x7f7a20d13eb0, argc=94, argv=0x7ffcb0b0b948, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb0b0b938) at libc-start.c:287
#7 0x00007f7a20d1b96c in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f793e3ff700 (LWP 65315))]
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) where
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f7a21036b29 in ?? ()
#2 0x00007f7a20fb10c3 in ?? ()
#3 0x00007f7a20fb14c0 in ?? ()
#4 0x00007f7a17522182 in start_thread (arg=0x7f793e3ff700) at pthread_create.c:312
#5 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f7a0cbf8700 (LWP 65311))]
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) where
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f7a21036909 in ?? ()
#4 0x00007f7a20d57d3c in ?? ()
#5 0x00007f7a20d42db2 in ?? ()
#6 0x00007f7a17522182 in start_thread (arg=0x7f7a0cbf8700) at pthread_create.c:312
#7 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) thread 4
[Switching to thread 4 (Thread 0x7f7a0d3f9700 (LWP 65310))]
#0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:56
56 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
(gdb) where
#0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:56
#1 0x00007f7a20fc2b2b in ?? ()
#2 0x00007f7a20fc3d04 in ?? ()
#3 0x00007f7a20fb5e5f in ?? ()
#4 0x00007f7a20fb78f4 in ?? ()
#5 0x00007f7a20ef1732 in ?? ()
#6 0x00007f7a20d58e81 in ?? ()
#7 0x00007f7a20d5e947 in ?? ()
#8 0x00007f7a20d1f933 in ?? ()
#9 0x00007f7a20d57fa1 in ?? ()
#10 0x00007f7a20d42db2 in ?? ()
#11 0x00007f7a17522182 in start_thread (arg=0x7f7a0d3f9700) at pthread_create.c:312
#12 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads
2015-12-08 0:44 [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads Neil McGill
@ 2015-12-10 8:47 ` Stefan Hajnoczi
2015-12-10 12:43 ` Neil McGill
0 siblings, 1 reply; 3+ messages in thread
From: Stefan Hajnoczi @ 2015-12-10 8:47 UTC (permalink / raw)
To: Neil McGill; +Cc: qemu-devel, Rich Wellum (rwellum)
[-- Attachment #1: Type: text/plain, Size: 4328 bytes --]
On Mon, Dec 07, 2015 at 07:44:24PM -0500, Neil McGill wrote:
> Has anyone seen anything like this ? all 4 qemu threads are stuck on a
> pthread condition
This is incorrect. Thread 4 is blocked in poll(2).
> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
>
> I'll try and get the qemu source built on this machine to debug and get
> some useful symbols out of gdb, but just wondering if there is a known
> issue like this on:
>
> Linux virl 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> tx
>
> neil
>
> (gdb) where
> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
> #3 0x00007f7a21036909 in ?? ()
> #4 0x00007f7a20fc1fdb in ?? ()
> #5 0x00007f7a20d153fe in ?? ()
> #6 0x00007f7a17176ec5 in __libc_start_main (main=0x7f7a20d13eb0, argc=94, argv=0x7ffcb0b0b948, init=<optimized out>,
> fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb0b0b938) at libc-start.c:287
> #7 0x00007f7a20d1b96c in ?? ()
This is the QEMU main loop. It's probably trying to lock the global
mutex.
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f793e3ff700 (LWP 65315))]
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> 185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
> (gdb) where
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1 0x00007f7a21036b29 in ?? ()
> #2 0x00007f7a20fb10c3 in ?? ()
> #3 0x00007f7a20fb14c0 in ?? ()
> #4 0x00007f7a17522182 in start_thread (arg=0x7f793e3ff700) at pthread_create.c:312
> #5 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> (gdb) thread 3
> [Switching to thread 3 (Thread 0x7f7a0cbf8700 (LWP 65311))]
> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> 135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
> (gdb) where
> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
> #2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
> #3 0x00007f7a21036909 in ?? ()
> #4 0x00007f7a20d57d3c in ?? ()
> #5 0x00007f7a20d42db2 in ?? ()
> #6 0x00007f7a17522182 in start_thread (arg=0x7f7a0cbf8700) at pthread_create.c:312
> #7 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This could be a vcpu thread. It's also trying to lock the global mutex.
>
> (gdb) thread 4
> [Switching to thread 4 (Thread 0x7f7a0d3f9700 (LWP 65310))]
> #0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
> at ../sysdeps/unix/sysv/linux/ppoll.c:56
> 56 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
> (gdb) where
> #0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
> at ../sysdeps/unix/sysv/linux/ppoll.c:56
> #1 0x00007f7a20fc2b2b in ?? ()
> #2 0x00007f7a20fc3d04 in ?? ()
> #3 0x00007f7a20fb5e5f in ?? ()
> #4 0x00007f7a20fb78f4 in ?? ()
> #5 0x00007f7a20ef1732 in ?? ()
> #6 0x00007f7a20d58e81 in ?? ()
> #7 0x00007f7a20d5e947 in ?? ()
> #8 0x00007f7a20d1f933 in ?? ()
> #9 0x00007f7a20d57fa1 in ?? ()
> #10 0x00007f7a20d42db2 in ?? ()
> #11 0x00007f7a17522182 in start_thread (arg=0x7f7a0d3f9700) at pthread_create.c:312
> #12 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> (gdb)
This is the interesting thread. It is blocked in poll(2) waiting for
activity on just 1 file descriptor. You could check fds[0] and then
look at ls -l /proc/fd/<n> to see what the file is being monitored.
My guess is thread 4 holds the global mutex and is therefore blocking
all the other threads.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads
2015-12-10 8:47 ` Stefan Hajnoczi
@ 2015-12-10 12:43 ` Neil McGill
0 siblings, 0 replies; 3+ messages in thread
From: Neil McGill @ 2015-12-10 12:43 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel, Rich Wellum (rwellum)
Thanks Stefan, will check and update
neil
On 12/10/15, 3:47 AM, Stefan Hajnoczi wrote:
> On Mon, Dec 07, 2015 at 07:44:24PM -0500, Neil McGill wrote:
>> Has anyone seen anything like this ? all 4 qemu threads are stuck on a
>> pthread condition
>
> This is incorrect. Thread 4 is blocked in poll(2).
>
>> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
>>
>> I'll try and get the qemu source built on this machine to debug and get
>> some useful symbols out of gdb, but just wondering if there is a known
>> issue like this on:
>>
>> Linux virl 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> tx
>>
>> neil
>>
>> (gdb) where
>> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
>> #3 0x00007f7a21036909 in ?? ()
>> #4 0x00007f7a20fc1fdb in ?? ()
>> #5 0x00007f7a20d153fe in ?? ()
>> #6 0x00007f7a17176ec5 in __libc_start_main (main=0x7f7a20d13eb0, argc=94, argv=0x7ffcb0b0b948, init=<optimized out>,
>> fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb0b0b938) at libc-start.c:287
>> #7 0x00007f7a20d1b96c in ?? ()
>
> This is the QEMU main loop. It's probably trying to lock the global
> mutex.
>
>> (gdb) thread 2
>> [Switching to thread 2 (Thread 0x7f793e3ff700 (LWP 65315))]
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> 185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
>> (gdb) where
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1 0x00007f7a21036b29 in ?? ()
>> #2 0x00007f7a20fb10c3 in ?? ()
>> #3 0x00007f7a20fb14c0 in ?? ()
>> #4 0x00007f7a17522182 in start_thread (arg=0x7f793e3ff700) at pthread_create.c:312
>> #5 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>>
>> (gdb) thread 3
>> [Switching to thread 3 (Thread 0x7f7a0cbf8700 (LWP 65311))]
>> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> 135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
>> (gdb) where
>> #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>> #1 0x00007f7a17524657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #2 0x00007f7a17524480 in __GI___pthread_mutex_lock (mutex=0x7f7a2150f9a0) at ../nptl/pthread_mutex_lock.c:79
>> #3 0x00007f7a21036909 in ?? ()
>> #4 0x00007f7a20d57d3c in ?? ()
>> #5 0x00007f7a20d42db2 in ?? ()
>> #6 0x00007f7a17522182 in start_thread (arg=0x7f7a0cbf8700) at pthread_create.c:312
>> #7 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
> This could be a vcpu thread. It's also trying to lock the global mutex.
>
>>
>> (gdb) thread 4
>> [Switching to thread 4 (Thread 0x7f7a0d3f9700 (LWP 65310))]
>> #0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>> at ../sysdeps/unix/sysv/linux/ppoll.c:56
>> 56 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
>> (gdb) where
>> #0 0x00007f7a172421ef in __GI_ppoll (fds=0x7f7a234db150, nfds=1, timeout=<optimized out>, sigmask=0x0)
>> at ../sysdeps/unix/sysv/linux/ppoll.c:56
>> #1 0x00007f7a20fc2b2b in ?? ()
>> #2 0x00007f7a20fc3d04 in ?? ()
>> #3 0x00007f7a20fb5e5f in ?? ()
>> #4 0x00007f7a20fb78f4 in ?? ()
>> #5 0x00007f7a20ef1732 in ?? ()
>> #6 0x00007f7a20d58e81 in ?? ()
>> #7 0x00007f7a20d5e947 in ?? ()
>> #8 0x00007f7a20d1f933 in ?? ()
>> #9 0x00007f7a20d57fa1 in ?? ()
>> #10 0x00007f7a20d42db2 in ?? ()
>> #11 0x00007f7a17522182 in start_thread (arg=0x7f7a0d3f9700) at pthread_create.c:312
>> #12 0x00007f7a1724f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>> (gdb)
>
> This is the interesting thread. It is blocked in poll(2) waiting for
> activity on just 1 file descriptor. You could check fds[0] and then
> look at ls -l /proc/fd/<n> to see what the file is being monitored.
>
> My guess is thread 4 holds the global mutex and is therefore blocking
> all the other threads.
>
> Stefan
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-12-10 12:43 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-08 0:44 [Qemu-devel] qemu 2.2 stuck on condition mutex locks on all threads Neil McGill
2015-12-10 8:47 ` Stefan Hajnoczi
2015-12-10 12:43 ` Neil McGill
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.