* deadlock in fio.. client/server
@ 2014-08-30 14:41 Castor Fu
2014-09-11 20:45 ` Jens Axboe
0 siblings, 1 reply; 4+ messages in thread
From: Castor Fu @ 2014-08-30 14:41 UTC (permalink / raw)
To: fio
I see what looks to me like a deadlock when running with multiple jobs
in client/server mode.
However, I'm not too used to debugging multithreaded code, so I
figured maybe some additional eyes would help.
This is on a Gen8 HP DL380 (dual socket x86_64) running CentOS EL6.
I haven't seen this when just running from the command line.
I start the daemon with:
fio --daemonize=/tmp/fio.pid.4987 --server=localhost,8098
and a client with
fio --client=localhost 8098 --output=test_blk.20140830Z1419 test_bad2.fio
#
[global]
ioengine=libaio
thread=1
bs=4k
filename=/dev/sdb
size=1M
do_verify=1
iodepth=8
stonewall
[writes]
rw=write
# 3 passes
# 4 fails
numjobs=4
and sometimes (50% or higher with num_jobs=4) it hangs.
If I look at it in GDB we see:
(gdb) info thread
6 Thread 0x7f446644a700 (LWP 5017) 0x0000003ffa8acb8d in nanosleep
() from /lib64/libc.so.6
5 Thread 0x7f4465a47700 (LWP 5023) 0x0000003ffac0e054 in
__lll_lock_wait () from /lib64/libpthread.so.0
4 Thread 0x7f4465045700 (LWP 5024) 0x0000003ffac0e054 in
__lll_lock_wait () from /lib64/libpthread.so.0
3 Thread 0x7f4464643700 (LWP 5025) 0x0000003ffac0e054 in
__lll_lock_wait () from /lib64/libpthread.so.0
2 Thread 0x7f4463c41700 (LWP 5027) 0x0000003ffac0e054 in
__lll_lock_wait () from /lib64/libpthread.so.0
* 1 Thread 0x7f446f4fe720 (LWP 5016) 0x0000003ffa8acb8d in nanosleep
() from /lib64/libc.so.6
(gdb) p mp[0].lock
$1 = (struct fio_mutex *) 0x7f446f509000
(gdb) p *mp[0].lock
$2 = {
lock = {
__data = {
__lock = 2,
__count = 0,
__owner = 5023,
__nusers = 1,
__kind = 128,
__spins = 0,
__list = { __prev = 0x0, __next = 0x0 }
},
__size = "\002\000\000\000\000\000\000\000\237\023\000\000\001\000\000\000\200",
'\000' <repeats 22 times>,
__align = 2
},
cond = {
__data = {
__lock = 2, __futex = 0,
__total_seq = 18446744073709551615,
__wakeup_seq = 0, __woken_seq = 0,
__mutex = 0xffffffffffffffff,
__nwaiters = 0, __broadcast_seq = 0
},
__size = "\002\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377",
'\000' <repeats 16 times>"\377,
\377\377\377\377\377\377\377\000\000\000\000\000\000\000",
__align = 2
},
value = 0,
waiters = 1,
magic = 1297437765
}
But if we look at lwp 5023 which is supposed to be the owner it looks
like it's waiting on the lock.
gdb) thread 5
[Switching to thread 5 (Thread 0x7f4465a47700 (LWP 5023))]#0
0x0000003ffac0e054 in __lll_lock_wait ()
from /lib64/libpthread.so.0
(gdb) bt
#0 0x0000003ffac0e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003ffac0b4fc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#2 0x000000000043fe81 in fio_mutex_down (mutex=0x7f446f509000) at
util/fio/mutex.c:155
#3 0x000000000044a7c1 in pool_lock (ptr=0x7f446ecf51f0) at
util/fio/smalloc.c:60
#4 sfree_pool (ptr=0x7f446ecf51f0) at util/fio/smalloc.c:328
#5 sfree (ptr=0x7f446ecf51f0) at util/fio/smalloc.c:356
#6 0x00000000004284f2 in close_and_free_files (td=0x7f4466450000) at
util/fio/filesetup.c:1122
#7 0x000000000040b495 in thread_main (data=0x7f4466450000) at
util/fio/backend.c:1548
#8 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
And all threads....
Thread 6 (Thread 0x7f446644a700 (LWP 5017)):
#0 0x0000003ffa8acb8d in nanosleep () from /lib64/libc.so.6
#1 0x0000003ffa8e1d64 in usleep () from /lib64/libc.so.6
#2 0x000000000040a558 in disk_thread_main (data=Unhandled dwarf
expression opcode 0xf3
) at util/fio/backend.c:1992
#3 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f4465a47700 (LWP 5023)):
#0 0x0000003ffac0e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003ffac0b4fc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#2 0x000000000043fe81 in fio_mutex_down (mutex=0x7f446f509000) at
util/fio/mutex.c:155
#3 0x000000000044a7c1 in pool_lock (ptr=0x7f446ecf51f0) at
util/fio/smalloc.c:60
#4 sfree_pool (ptr=0x7f446ecf51f0) at util/fio/smalloc.c:328
#5 sfree (ptr=0x7f446ecf51f0) at util/fio/smalloc.c:356
#6 0x00000000004284f2 in close_and_free_files (td=0x7f4466450000) at
util/fio/filesetup.c:1122
#7 0x000000000040b495 in thread_main (data=0x7f4466450000) at
util/fio/backend.c:1548
#8 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4465045700 (LWP 5024)):
#0 0x0000003ffac0e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003ffac09388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003ffac09257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043fe58 in fio_mutex_down (mutex=0x7f446f509000) at
util/fio/mutex.c:151
#4 0x000000000044a7c1 in pool_lock (ptr=0x7f446ecf5390) at
util/fio/smalloc.c:60
#5 sfree_pool (ptr=0x7f446ecf5390) at util/fio/smalloc.c:328
#6 sfree (ptr=0x7f446ecf5390) at util/fio/smalloc.c:356
#7 0x00000000004284f2 in close_and_free_files (td=0x7f44664553b8) at
util/fio/filesetup.c:1122
#8 0x000000000040b495 in thread_main (data=0x7f44664553b8) at
util/fio/backend.c:1548
#9 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4464643700 (LWP 5025)):
#0 0x0000003ffac0e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003ffac09388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003ffac09257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043fe58 in fio_mutex_down (mutex=0x7f446f509000) at
util/fio/mutex.c:151
#4 0x000000000044a7c1 in pool_lock (ptr=0x7f446ecf5690) at
util/fio/smalloc.c:60
#5 sfree_pool (ptr=0x7f446ecf5690) at util/fio/smalloc.c:328
#6 sfree (ptr=0x7f446ecf5690) at util/fio/smalloc.c:356
#7 0x00000000004284c7 in close_and_free_files (td=0x7f446645a770) at
util/fio/filesetup.c:1118
#8 0x000000000040b495 in thread_main (data=0x7f446645a770) at
util/fio/backend.c:1548
#9 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4463c41700 (LWP 5027)):
#0 0x0000003ffac0e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003ffac09388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003ffac09257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043feda in fio_mutex_up (mutex=0x7f446f509000) at
util/fio/mutex.c:169
#4 0x00000000004284c7 in close_and_free_files (td=0x7f446645fb28) at
util/fio/filesetup.c:1118
#5 0x000000000040b495 in thread_main (data=0x7f446645fb28) at
util/fio/backend.c:1548
#6 0x0000003ffac07851 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003ffa8e890d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f446f4fe720 (LWP 5016)):
#0 0x0000003ffa8acb8d in nanosleep () from /lib64/libc.so.6
#1 0x0000003ffa8e1d64 in usleep () from /lib64/libc.so.6
#2 0x000000000040e343 in do_usleep () at util/fio/backend.c:1727
#3 run_threads () at util/fio/backend.c:1965
#4 0x000000000040e6ed in fio_backend () at util/fio/backend.c:2068
#5 0x000000000044a1f9 in handle_run_cmd () at util/fio/server.c:567
#6 handle_command () at util/fio/server.c:763
#7 handle_connection () at util/fio/server.c:829
#8 accept_loop () at util/fio/server.c:915
#9 fio_server () at util/fio/server.c:1614
#10 0x000000000044a329 in fio_start_server (pidfile=0x268f000
"/tmp/fio.pid.4987") at util/fio/server.c:1724
#11 0x0000000000430574 in parse_cmd_line (argc=Unhandled dwarf
expression opcode 0xf3
) at util/fio/init.c:2154
#12 0x0000000000430b23 in parse_options (argc=3, argv=0x7fff62b22be8)
at util/fio/init.c:2208
#13 0x000000000042974c in main (argc=3, argv=0x7fff62b22be8,
envp=Unhandled dwarf expression opcode 0xf3
) at util/fio/fio.c:40
(
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: deadlock in fio.. client/server
2014-08-30 14:41 deadlock in fio.. client/server Castor Fu
@ 2014-09-11 20:45 ` Jens Axboe
2014-09-11 23:07 ` Jens Axboe
0 siblings, 1 reply; 4+ messages in thread
From: Jens Axboe @ 2014-09-11 20:45 UTC (permalink / raw)
To: Castor Fu, fio
On 08/30/2014 08:41 AM, Castor Fu wrote:
> I see what looks to me like a deadlock when running with multiple jobs
> in client/server mode.
> However, I'm not too used to debugging multithreaded code, so I
> figured maybe some additional eyes would help.
>
> This is on a Gen8 HP DL380 (dual socket x86_64) running CentOS EL6.
>
> I haven't seen this when just running from the command line.
>
> I start the daemon with:
>
> fio --daemonize=/tmp/fio.pid.4987 --server=localhost,8098
>
> and a client with
> fio --client=localhost 8098 --output=test_blk.20140830Z1419 test_bad2.fio
>
> #
> [global]
>
> ioengine=libaio
> thread=1
> bs=4k
> filename=/dev/sdb
> size=1M
> do_verify=1
> iodepth=8
> stonewall
>
> [writes]
> rw=write
> # 3 passes
> # 4 fails
> numjobs=4
>
> and sometimes (50% or higher with num_jobs=4) it hangs.
Sorry for being late on this, I can reproduce it. Will take a look!
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: deadlock in fio.. client/server
2014-09-11 20:45 ` Jens Axboe
@ 2014-09-11 23:07 ` Jens Axboe
2014-09-12 6:34 ` Castor Fu
0 siblings, 1 reply; 4+ messages in thread
From: Jens Axboe @ 2014-09-11 23:07 UTC (permalink / raw)
To: Castor Fu, fio
On 09/11/2014 02:45 PM, Jens Axboe wrote:
> On 08/30/2014 08:41 AM, Castor Fu wrote:
>> I see what looks to me like a deadlock when running with multiple jobs
>> in client/server mode.
>> However, I'm not too used to debugging multithreaded code, so I
>> figured maybe some additional eyes would help.
>>
>> This is on a Gen8 HP DL380 (dual socket x86_64) running CentOS EL6.
>>
>> I haven't seen this when just running from the command line.
>>
>> I start the daemon with:
>>
>> fio --daemonize=/tmp/fio.pid.4987 --server=localhost,8098
>>
>> and a client with
>> fio --client=localhost 8098 --output=test_blk.20140830Z1419 test_bad2.fio
>>
>> #
>> [global]
>>
>> ioengine=libaio
>> thread=1
>> bs=4k
>> filename=/dev/sdb
>> size=1M
>> do_verify=1
>> iodepth=8
>> stonewall
>>
>> [writes]
>> rw=write
>> # 3 passes
>> # 4 fails
>> numjobs=4
>>
>> and sometimes (50% or higher with num_jobs=4) it hangs.
>
> Sorry for being late on this, I can reproduce it. Will take a look!
Committed a fix for this, can you check current -git? This is the fix:
http://git.kernel.dk/?p=fio.git;a=commit;h=44c8268d6e6d3b3f1efb1eb8a1c2d7bba1e8477b
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: deadlock in fio.. client/server
2014-09-11 23:07 ` Jens Axboe
@ 2014-09-12 6:34 ` Castor Fu
0 siblings, 0 replies; 4+ messages in thread
From: Castor Fu @ 2014-09-12 6:34 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Thanks Jens! That seems to work for me.
-castor
On Thu, Sep 11, 2014 at 4:07 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 09/11/2014 02:45 PM, Jens Axboe wrote:
>> On 08/30/2014 08:41 AM, Castor Fu wrote:
>>> I see what looks to me like a deadlock when running with multiple jobs
>>> in client/server mode.
>>> However, I'm not too used to debugging multithreaded code, so I
>>> figured maybe some additional eyes would help.
>>>
>>> This is on a Gen8 HP DL380 (dual socket x86_64) running CentOS EL6.
>>>
>>> I haven't seen this when just running from the command line.
>>>
>>> I start the daemon with:
>>>
>>> fio --daemonize=/tmp/fio.pid.4987 --server=localhost,8098
>>>
>>> and a client with
>>> fio --client=localhost 8098 --output=test_blk.20140830Z1419 test_bad2.fio
>>>
>>> #
>>> [global]
>>>
>>> ioengine=libaio
>>> thread=1
>>> bs=4k
>>> filename=/dev/sdb
>>> size=1M
>>> do_verify=1
>>> iodepth=8
>>> stonewall
>>>
>>> [writes]
>>> rw=write
>>> # 3 passes
>>> # 4 fails
>>> numjobs=4
>>>
>>> and sometimes (50% or higher with num_jobs=4) it hangs.
>>
>> Sorry for being late on this, I can reproduce it. Will take a look!
>
> Committed a fix for this, can you check current -git? This is the fix:
>
> http://git.kernel.dk/?p=fio.git;a=commit;h=44c8268d6e6d3b3f1efb1eb8a1c2d7bba1e8477b
>
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-09-12 6:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-30 14:41 deadlock in fio.. client/server Castor Fu
2014-09-11 20:45 ` Jens Axboe
2014-09-11 23:07 ` Jens Axboe
2014-09-12 6:34 ` Castor Fu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.