* Deadlock? @ 2012-02-13 11:17 Markus 2012-02-13 17:37 ` Deadlock? Jack Stone 2012-02-14 3:16 ` Deadlock? Paul Hartman 0 siblings, 2 replies; 5+ messages in thread From: Markus @ 2012-02-13 11:17 UTC (permalink / raw) To: lkml Hi! I noted some kind of deadlock, where I was not able to write to the raid6, while writing to each disk would still work. This caused many processes to "wait" in d-state, thus making it impossible to unmount, cleanly reboot, sync, ... So I enabled the detection of hung tasks and this deplock option. After running a 3.2.2 for about 5 days: http://pastebin.com/gy1kaYmS I dont know if its a bug or nothing. Or if it has anything to do with my problem as there was no hungtask detected and the raid still seems to work. Can anybody enlight me?! ;) Markus ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Deadlock? 2012-02-13 11:17 Deadlock? Markus @ 2012-02-13 17:37 ` Jack Stone 2012-02-14 12:47 ` Deadlock? Jan Kara 2012-02-14 3:16 ` Deadlock? Paul Hartman 1 sibling, 1 reply; 5+ messages in thread From: Jack Stone @ 2012-02-13 17:37 UTC (permalink / raw) To: Markus; +Cc: lkml, jack, akpm, adilger.kernel, gregkh Adding CCs On 02/13/2012 11:17 AM, Markus wrote: > Hi! > > I noted some kind of deadlock, where I was not able to write to the raid6, > while writing to each disk would still work. > This caused many processes to "wait" in d-state, thus making it impossible to > unmount, cleanly reboot, sync, ... > > So I enabled the detection of hung tasks and this deplock option. > After running a 3.2.2 for about 5 days: > http://pastebin.com/gy1kaYmS > > I dont know if its a bug or nothing. Or if it has anything to do with my > problem as there was no hungtask detected and the raid still seems to work. > > Can anybody enlight me?! ;) > > Markus [493061.240015] [ INFO: RECLAIM_FS-safe -> RECLAIM_FS-unsafe lock order detected ] [493061.240015] 3.2.2 #1 [493061.240015] ------------------------------------------------------ [493061.240015] linuxdcpp/11649 [HC0[0]:SC0[0]:HE1:SE1] is trying to acquire: [493061.240015] (&tty->atomic_write_lock){+.+.+.}, at: [<ffffffff811bf40a>] tty_write_message+0x2a/0x80 [493061.240015] [493061.240015] and this task is already holding: [493061.240015] (&s->s_dquot.dqptr_sem){++++-.}, at: [<ffffffff8111888d>] __dquot_alloc_space+0x9d/0x230 [493061.240015] which would create a new lock dependency: [493061.240015] (&s->s_dquot.dqptr_sem){++++-.} -> (&tty->atomic_write_lock){+.+.+.} [493061.240015] [493061.240015] but this new dependency connects a RECLAIM_FS-irq-safe lock: [493061.240015] (&s->s_dquot.dqptr_sem){++++-.} [493061.240015] ... which became RECLAIM_FS-irq-safe at: [493061.240015] [<ffffffff81069914>] __lock_acquire+0x6e4/0x1eb0 [493061.240015] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493061.240015] [<ffffffff81375e4c>] down_write+0x2c/0x50 [493061.240015] [<ffffffff81119c97>] __dquot_drop+0x27/0x80 [493061.240015] [<ffffffff81119d15>] dquot_drop+0x25/0x30 [493061.240015] [<ffffffff81138278>] ext3_evict_inode+0x158/0x260 [493061.240015] [<ffffffff810e0167>] evict+0xa7/0x1b0 [493061.240015] [<ffffffff810e0807>] dispose_list+0x47/0x60 [493061.240015] [<ffffffff810e1222>] prune_icache_sb+0x192/0x370 [493061.240015] [<ffffffff810c9e03>] prune_super+0x153/0x1b0 [493061.240015] [<ffffffff8109c355>] shrink_slab+0x135/0x1f0 [493061.240015] [<ffffffff8109e932>] kswapd+0x632/0x8e0 [493061.240015] [<ffffffff81055ce6>] kthread+0x96/0xa0 [493061.240015] [<ffffffff81378eb4>] kernel_thread_helper+0x4/0x10 [493061.240015] [493061.240015] to a RECLAIM_FS-irq-unsafe lock: [493061.240015] (&tty->atomic_write_lock){+.+.+.} [493061.240015] ... which became RECLAIM_FS-irq-unsafe at: [493061.240015] ... [<ffffffff810681f6>] mark_held_locks+0x76/0x150 [493061.240015] [<ffffffff81068ab0>] lockdep_trace_alloc+0xb0/0xe0 [493061.240015] [<ffffffff810c2a78>] __kmalloc+0x78/0x160 [493061.240015] [<ffffffff811bf126>] tty_write+0x136/0x270 [493061.240015] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0 [493061.240015] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493061.240015] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493061.240015] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493061.240015] [493061.240015] other info that might help us debug this: [493061.240015] [493061.240015] Possible interrupt unsafe locking scenario: [493061.240015] [493061.240015] CPU0 CPU1 [493061.240015] ---- ---- [493061.240015] lock(&tty->atomic_write_lock); [493061.240015] local_irq_disable(); [493061.240015] lock(&s->s_dquot.dqptr_sem); [493061.240015] lock(&tty->atomic_write_lock); [493061.240015] <Interrupt> [493061.240015] lock(&s->s_dquot.dqptr_sem); [493061.240015] [493061.240015] *** DEADLOCK *** [493061.240015] [493061.240015] 4 locks held by linuxdcpp/11649: [493063.862237] #0: (&sb->s_type->i_mutex_key#3){+.+.+.}, at: [<ffffffff81090947>] generic_file_aio_write+0x57/0xe0 [493063.862237] #1: (jbd_handle){+.+.-.}, at: [<ffffffff81152098>] start_this_handle+0x358/0x450 [493063.862237] #2: (&ei->truncate_mutex){+.+...}, at: [<ffffffff811367c5>] ext3_get_blocks_handle+0xe5/0xbf0 [493063.862237] #3: (&s->s_dquot.dqptr_sem){++++-.}, at: [<ffffffff8111888d>] __dquot_alloc_space+0x9d/0x230 [493063.862237] [493063.862237] the dependencies between RECLAIM_FS-irq-safe lock and the holding lock: [493063.862237] -> (&s->s_dquot.dqptr_sem){++++-.} ops: 187112656 { [493063.862237] HARDIRQ-ON-W at: [493063.862237] [<ffffffff810699c5>] __lock_acquire+0x795/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50 [493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80 [493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510 [493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80 [493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130 [493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560 [493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] HARDIRQ-ON-R at: [493063.862237] [<ffffffff8106988d>] __lock_acquire+0x65d/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e9f>] down_read+0x2f/0x50 [493063.862237] [<ffffffff8111bac3>] dquot_alloc_inode+0x43/0x140 [493063.862237] [<ffffffff8113354b>] ext3_new_inode+0x89b/0x980 [493063.862237] [<ffffffff8113b3fd>] ext3_create+0x8d/0x110 [493063.862237] [<ffffffff810d26fc>] vfs_create+0xac/0xe0 [493063.862237] [<ffffffff810d4149>] do_last.clone.30+0x469/0x7d0 [493063.862237] [<ffffffff810d5b50>] path_openat+0xd0/0x410 [493063.862237] [<ffffffff810d5fa4>] do_filp_open+0x44/0xa0 [493063.862237] [<ffffffff810c5f9c>] do_sys_open+0xfc/0x1e0 [493063.862237] [<ffffffff810c609b>] sys_open+0x1b/0x20 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] SOFTIRQ-ON-W at: [493063.862237] [<ffffffff810699ff>] __lock_acquire+0x7cf/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50 [493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80 [493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510 [493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80 [493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130 [493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560 [493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] SOFTIRQ-ON-R at: [493063.862237] [<ffffffff810699ff>] __lock_acquire+0x7cf/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e9f>] down_read+0x2f/0x50 [493063.862237] [<ffffffff8111bac3>] dquot_alloc_inode+0x43/0x140 [493063.862237] [<ffffffff8113354b>] ext3_new_inode+0x89b/0x980 [493063.862237] [<ffffffff8113b3fd>] ext3_create+0x8d/0x110 [493063.862237] [<ffffffff810d26fc>] vfs_create+0xac/0xe0 [493063.862237] [<ffffffff810d4149>] do_last.clone.30+0x469/0x7d0 [493063.862237] [<ffffffff810d5b50>] path_openat+0xd0/0x410 [493063.862237] [<ffffffff810d5fa4>] do_filp_open+0x44/0xa0 [493063.862237] [<ffffffff810c5f9c>] do_sys_open+0xfc/0x1e0 [493063.862237] [<ffffffff810c609b>] sys_open+0x1b/0x20 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] IN-RECLAIM_FS-W at: [493063.862237] [<ffffffff81069914>] __lock_acquire+0x6e4/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50 [493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80 [493063.862237] [<ffffffff81119d15>] dquot_drop+0x25/0x30 [493063.862237] [<ffffffff81138278>] ext3_evict_inode+0x158/0x260 [493063.862237] [<ffffffff810e0167>] evict+0xa7/0x1b0 [493063.862237] [<ffffffff810e0807>] dispose_list+0x47/0x60 [493063.862237] [<ffffffff810e1222>] prune_icache_sb+0x192/0x370 [493063.862237] [<ffffffff810c9e03>] prune_super+0x153/0x1b0 [493063.862237] [<ffffffff8109c355>] shrink_slab+0x135/0x1f0 [493063.862237] [<ffffffff8109e932>] kswapd+0x632/0x8e0 [493063.862237] [<ffffffff81055ce6>] kthread+0x96/0xa0 [493063.862237] [<ffffffff81378eb4>] kernel_thread_helper+0x4/0x10 [493063.862237] INITIAL USE at: [493063.862237] [<ffffffff81069619>] __lock_acquire+0x3e9/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50 [493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80 [493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510 [493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80 [493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130 [493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560 [493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] } [493063.862237] ... key at: [<ffffffff81f48938>] __key.28495+0x0/0x8 [493063.862237] ... acquired at: [493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0 [493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300 [493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80 [493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200 [493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230 [493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680 [493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0 [493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120 [493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540 [493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200 [493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280 [493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430 [493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0 [493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] [493063.862237] [493063.862237] the dependencies between the lock to be acquired and RECLAIM_FS-irq-unsafe lock: [493063.862237] -> (&tty->atomic_write_lock){+.+.+.} ops: 270462830 { [493063.862237] HARDIRQ-ON-W at: [493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150 [493063.862237] [<ffffffff8106837d>] trace_hardirqs_on_caller+0xad/0x1e0 [493063.862237] [<ffffffff810684bd>] trace_hardirqs_on+0xd/0x10 [493063.862237] [<ffffffff81374cbd>] mutex_trylock+0xfd/0x170 [493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60 [493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270 [493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] SOFTIRQ-ON-W at: [493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150 [493063.862237] [<ffffffff810683ed>] trace_hardirqs_on_caller+0x11d/0x1e0 [493063.862237] [<ffffffff810684bd>] trace_hardirqs_on+0xd/0x10 [493063.862237] [<ffffffff81374cbd>] mutex_trylock+0xfd/0x170 [493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60 [493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270 [493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] RECLAIM_FS-ON-W at: [493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150 [493063.862237] [<ffffffff81068ab0>] lockdep_trace_alloc+0xb0/0xe0 [493063.862237] [<ffffffff810c2a78>] __kmalloc+0x78/0x160 [493063.862237] [<ffffffff811bf126>] tty_write+0x136/0x270 [493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] INITIAL USE at: [493063.862237] [<ffffffff81069619>] __lock_acquire+0x3e9/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81374c6d>] mutex_trylock+0xad/0x170 [493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60 [493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270 [493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] } [493063.862237] ... key at: [<ffffffff81f549b8>] __key.27415+0x0/0x8 [493063.862237] ... acquired at: [493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0 [493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300 [493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80 [493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200 [493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230 [493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680 [493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0 [493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120 [493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540 [493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200 [493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280 [493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430 [493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0 [493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b [493063.862237] [493063.862237] [493063.862237] stack backtrace: [493063.862237] Pid: 11649, comm: linuxdcpp Not tainted 3.2.2 #1 [493063.862237] Call Trace: [493063.862237] [<ffffffff8106774a>] check_usage+0x4aa/0x4b0 [493063.862237] [<ffffffff810050f4>] ? print_context_stack+0x74/0xd0 [493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0 [493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0 [493063.862237] [<ffffffff8100e6fa>] ? save_stack_trace+0x2a/0x50 [493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0 [493063.862237] [<ffffffff811bf40a>] ? tty_write_message+0x2a/0x80 [493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300 [493063.862237] [<ffffffff811bf40a>] ? tty_write_message+0x2a/0x80 [493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80 [493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200 [493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230 [493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680 [493063.862237] [<ffffffff81374f8f>] ? mutex_lock_nested+0x25f/0x300 [493063.862237] [<ffffffff81374f7b>] ? mutex_lock_nested+0x24b/0x300 [493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0 [493063.862237] [<ffffffff8106973a>] ? __lock_acquire+0x50a/0x1eb0 [493063.862237] [<ffffffff8106973a>] ? __lock_acquire+0x50a/0x1eb0 [493063.862237] [<ffffffff810f2c2b>] ? create_empty_buffers+0x4b/0xd0 [493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120 [493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540 [493063.862237] [<ffffffff811372d0>] ? ext3_get_blocks_handle+0xbf0/0xbf0 [493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200 [493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280 [493063.862237] [<ffffffff8103f221>] ? current_fs_time+0x11/0x50 [493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430 [493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0 [493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120 [493063.862237] [<ffffffff810a850b>] ? might_fault+0x3b/0x90 [493063.862237] [<ffffffff81167287>] ? security_file_permission+0x27/0xb0 [493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180 [493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90 [493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Deadlock? 2012-02-13 17:37 ` Deadlock? Jack Stone @ 2012-02-14 12:47 ` Jan Kara 2012-02-15 11:42 ` Deadlock? Markus 0 siblings, 1 reply; 5+ messages in thread From: Jan Kara @ 2012-02-14 12:47 UTC (permalink / raw) To: Jack Stone; +Cc: Markus, lkml, jack, akpm, adilger.kernel, gregkh [-- Attachment #1: Type: text/plain, Size: 1150 bytes --] On Mon 13-02-12 17:37:34, Jack Stone wrote: > Adding CCs > > On 02/13/2012 11:17 AM, Markus wrote: > > I noted some kind of deadlock, where I was not able to write to the raid6, > > while writing to each disk would still work. > > This caused many processes to "wait" in d-state, thus making it impossible to > > unmount, cleanly reboot, sync, ... Thanks for report. Could you please provide output of 'echo w >/proc/sysrq-trigger' in dmesg? That would help us narrow down the real cause the the current deadlock. > > So I enabled the detection of hung tasks and this deplock option. > > After running a 3.2.2 for about 5 days: > > http://pastebin.com/gy1kaYmS > > > > I dont know if its a bug or nothing. Or if it has anything to do with my > > problem as there was no hungtask detected and the raid still seems to work. It's lock debugging code complaining about a possible problem it found. Actually, the problem looks real (although hard to hit AFAICS) and attached patch should fix it. Does the patch fix things for you? If not, please provide the output I describe above. Honza -- Jan Kara <jack@suse.cz> SUSE Labs, CR [-- Attachment #2: 0001-quota-Make-quota-code-not-call-tty-layer-with-dqptr_.patch --] [-- Type: text/x-patch, Size: 16616 bytes --] >From 4e9707d880d6131e9b123e7cdcf6ef7f4e52e478 Mon Sep 17 00:00:00 2001 From: Jan Kara <jack@suse.cz> Date: Tue, 14 Feb 2012 13:28:01 +0100 Subject: [PATCH] quota: Make quota code not call tty layer with dqptr_sem held dqptr_sem can be called from slab reclaim. tty layer uses GFP_KERNEL mask for allocation so it can end up calling slab reclaim. Given quota code can call into tty layer to print warning this creates possibility for lock inversion between tty->atomic_write_lock and dqptr_sem. Using direct printing of warnings from quota layer is obsolete but since it's easy enough to change quota code to not hold any locks when printing warnings, let's just do it. It seems like a good thing to do even when we use netlink layer to transmit warnings to userspace. Signed-off-by: Jan Kara <jack@suse.cz> --- fs/quota/dquot.c | 189 ++++++++++++++++++++++++++++++++---------------------- 1 files changed, 113 insertions(+), 76 deletions(-) diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c index 4674197..439ab11 100644 --- a/fs/quota/dquot.c +++ b/fs/quota/dquot.c @@ -1109,6 +1109,13 @@ static void dquot_decr_space(struct dquot *dquot, qsize_t number) clear_bit(DQ_BLKS_B, &dquot->dq_flags); } +struct dquot_warn { + struct super_block *w_sb; + qid_t w_dq_id; + short w_dq_type; + short w_type; +}; + static int warning_issued(struct dquot *dquot, const int warntype) { int flag = (warntype == QUOTA_NL_BHARDWARN || @@ -1124,41 +1131,42 @@ static int warning_issued(struct dquot *dquot, const int warntype) #ifdef CONFIG_PRINT_QUOTA_WARNING static int flag_print_warnings = 1; -static int need_print_warning(struct dquot *dquot) +static int need_print_warning(struct dquot_warn *warn) { if (!flag_print_warnings) return 0; - switch (dquot->dq_type) { + switch (warn->w_dq_type) { case USRQUOTA: - return current_fsuid() == dquot->dq_id; + return current_fsuid() == warn->w_dq_id; case GRPQUOTA: - return in_group_p(dquot->dq_id); + return in_group_p(warn->w_dq_id); } return 0; } /* Print warning to user which exceeded quota */ -static void print_warning(struct dquot *dquot, const int warntype) +static void print_warning(struct dquot_warn *warn) { char *msg = NULL; struct tty_struct *tty; + int warntype = warn->w_type; if (warntype == QUOTA_NL_IHARDBELOW || warntype == QUOTA_NL_ISOFTBELOW || warntype == QUOTA_NL_BHARDBELOW || - warntype == QUOTA_NL_BSOFTBELOW || !need_print_warning(dquot)) + warntype == QUOTA_NL_BSOFTBELOW || !need_print_warning(warn)) return; tty = get_current_tty(); if (!tty) return; - tty_write_message(tty, dquot->dq_sb->s_id); + tty_write_message(tty, warn->w_sb->s_id); if (warntype == QUOTA_NL_ISOFTWARN || warntype == QUOTA_NL_BSOFTWARN) tty_write_message(tty, ": warning, "); else tty_write_message(tty, ": write failed, "); - tty_write_message(tty, quotatypes[dquot->dq_type]); + tty_write_message(tty, quotatypes[warn->w_dq_type]); switch (warntype) { case QUOTA_NL_IHARDWARN: msg = " file limit reached.\r\n"; @@ -1184,26 +1192,34 @@ static void print_warning(struct dquot *dquot, const int warntype) } #endif +static void prepare_warning(struct dquot_warn *warn, struct dquot *dquot, + int warntype) +{ + if (warning_issued(dquot, warntype)) + return; + warn->w_type = warntype; + warn->w_sb = dquot->dq_sb; + warn->w_dq_id = dquot->dq_id; + warn->w_dq_type = dquot->dq_type; +} + /* * Write warnings to the console and send warning messages over netlink. * - * Note that this function can sleep. + * Note that this function can call into tty and networking code. */ -static void flush_warnings(struct dquot *const *dquots, char *warntype) +static void flush_warnings(struct dquot_warn *warn) { - struct dquot *dq; int i; for (i = 0; i < MAXQUOTAS; i++) { - dq = dquots[i]; - if (dq && warntype[i] != QUOTA_NL_NOWARN && - !warning_issued(dq, warntype[i])) { + if (warn[i].w_type == QUOTA_NL_NOWARN) + continue; #ifdef CONFIG_PRINT_QUOTA_WARNING - print_warning(dq, warntype[i]); + print_warning(&warn[i]); #endif - quota_send_warning(dq->dq_type, dq->dq_id, - dq->dq_sb->s_dev, warntype[i]); - } + quota_send_warning(warn[i].w_dq_type, warn[i].w_dq_id, + warn[i].w_sb->s_dev, warn[i].w_type); } } @@ -1217,11 +1233,11 @@ static int ignore_hardlimit(struct dquot *dquot) } /* needs dq_data_lock */ -static int check_idq(struct dquot *dquot, qsize_t inodes, char *warntype) +static int check_idq(struct dquot *dquot, qsize_t inodes, + struct dquot_warn *warn) { qsize_t newinodes = dquot->dq_dqb.dqb_curinodes + inodes; - *warntype = QUOTA_NL_NOWARN; if (!sb_has_quota_limits_enabled(dquot->dq_sb, dquot->dq_type) || test_bit(DQ_FAKE_B, &dquot->dq_flags)) return 0; @@ -1229,7 +1245,7 @@ static int check_idq(struct dquot *dquot, qsize_t inodes, char *warntype) if (dquot->dq_dqb.dqb_ihardlimit && newinodes > dquot->dq_dqb.dqb_ihardlimit && !ignore_hardlimit(dquot)) { - *warntype = QUOTA_NL_IHARDWARN; + prepare_warning(warn, dquot, QUOTA_NL_IHARDWARN); return -EDQUOT; } @@ -1238,14 +1254,14 @@ static int check_idq(struct dquot *dquot, qsize_t inodes, char *warntype) dquot->dq_dqb.dqb_itime && get_seconds() >= dquot->dq_dqb.dqb_itime && !ignore_hardlimit(dquot)) { - *warntype = QUOTA_NL_ISOFTLONGWARN; + prepare_warning(warn, dquot, QUOTA_NL_ISOFTLONGWARN); return -EDQUOT; } if (dquot->dq_dqb.dqb_isoftlimit && newinodes > dquot->dq_dqb.dqb_isoftlimit && dquot->dq_dqb.dqb_itime == 0) { - *warntype = QUOTA_NL_ISOFTWARN; + prepare_warning(warn, dquot, QUOTA_NL_ISOFTWARN); dquot->dq_dqb.dqb_itime = get_seconds() + sb_dqopt(dquot->dq_sb)->info[dquot->dq_type].dqi_igrace; } @@ -1254,12 +1270,12 @@ static int check_idq(struct dquot *dquot, qsize_t inodes, char *warntype) } /* needs dq_data_lock */ -static int check_bdq(struct dquot *dquot, qsize_t space, int prealloc, char *warntype) +static int check_bdq(struct dquot *dquot, qsize_t space, int prealloc, + struct dquot_warn *warn) { qsize_t tspace; struct super_block *sb = dquot->dq_sb; - *warntype = QUOTA_NL_NOWARN; if (!sb_has_quota_limits_enabled(sb, dquot->dq_type) || test_bit(DQ_FAKE_B, &dquot->dq_flags)) return 0; @@ -1271,7 +1287,7 @@ static int check_bdq(struct dquot *dquot, qsize_t space, int prealloc, char *war tspace > dquot->dq_dqb.dqb_bhardlimit && !ignore_hardlimit(dquot)) { if (!prealloc) - *warntype = QUOTA_NL_BHARDWARN; + prepare_warning(warn, dquot, QUOTA_NL_BHARDWARN); return -EDQUOT; } @@ -1281,7 +1297,7 @@ static int check_bdq(struct dquot *dquot, qsize_t space, int prealloc, char *war get_seconds() >= dquot->dq_dqb.dqb_btime && !ignore_hardlimit(dquot)) { if (!prealloc) - *warntype = QUOTA_NL_BSOFTLONGWARN; + prepare_warning(warn, dquot, QUOTA_NL_BSOFTLONGWARN); return -EDQUOT; } @@ -1289,7 +1305,7 @@ static int check_bdq(struct dquot *dquot, qsize_t space, int prealloc, char *war tspace > dquot->dq_dqb.dqb_bsoftlimit && dquot->dq_dqb.dqb_btime == 0) { if (!prealloc) { - *warntype = QUOTA_NL_BSOFTWARN; + prepare_warning(warn, dquot, QUOTA_NL_BSOFTWARN); dquot->dq_dqb.dqb_btime = get_seconds() + sb_dqopt(sb)->info[dquot->dq_type].dqi_bgrace; } @@ -1542,10 +1558,9 @@ static void inode_decr_space(struct inode *inode, qsize_t number, int reserve) int __dquot_alloc_space(struct inode *inode, qsize_t number, int flags) { int cnt, ret = 0; - char warntype[MAXQUOTAS]; - int warn = flags & DQUOT_SPACE_WARN; + struct dquot_warn warn[MAXQUOTAS]; + struct dquot **dquots = inode->i_dquot; int reserve = flags & DQUOT_SPACE_RESERVE; - int nofail = flags & DQUOT_SPACE_NOFAIL; /* * First test before acquiring mutex - solves deadlocks when we @@ -1558,36 +1573,36 @@ int __dquot_alloc_space(struct inode *inode, qsize_t number, int flags) down_read(&sb_dqopt(inode->i_sb)->dqptr_sem); for (cnt = 0; cnt < MAXQUOTAS; cnt++) - warntype[cnt] = QUOTA_NL_NOWARN; + warn[cnt].w_type = QUOTA_NL_NOWARN; spin_lock(&dq_data_lock); for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + if (!dquots[cnt]) continue; - ret = check_bdq(inode->i_dquot[cnt], number, !warn, - warntype+cnt); - if (ret && !nofail) { + ret = check_bdq(dquots[cnt], number, + !(flags & DQUOT_SPACE_WARN), &warn[cnt]); + if (ret && !(flags & DQUOT_SPACE_NOFAIL)) { spin_unlock(&dq_data_lock); goto out_flush_warn; } } for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + if (!dquots[cnt]) continue; if (reserve) - dquot_resv_space(inode->i_dquot[cnt], number); + dquot_resv_space(dquots[cnt], number); else - dquot_incr_space(inode->i_dquot[cnt], number); + dquot_incr_space(dquots[cnt], number); } inode_incr_space(inode, number, reserve); spin_unlock(&dq_data_lock); if (reserve) goto out_flush_warn; - mark_all_dquot_dirty(inode->i_dquot); + mark_all_dquot_dirty(dquots); out_flush_warn: - flush_warnings(inode->i_dquot, warntype); up_read(&sb_dqopt(inode->i_sb)->dqptr_sem); + flush_warnings(warn); out: return ret; } @@ -1599,36 +1614,37 @@ EXPORT_SYMBOL(__dquot_alloc_space); int dquot_alloc_inode(const struct inode *inode) { int cnt, ret = 0; - char warntype[MAXQUOTAS]; + struct dquot_warn warn[MAXQUOTAS]; + struct dquot * const *dquots = inode->i_dquot; /* First test before acquiring mutex - solves deadlocks when we * re-enter the quota code and are already holding the mutex */ if (!dquot_active(inode)) return 0; for (cnt = 0; cnt < MAXQUOTAS; cnt++) - warntype[cnt] = QUOTA_NL_NOWARN; + warn[cnt].w_type = QUOTA_NL_NOWARN; down_read(&sb_dqopt(inode->i_sb)->dqptr_sem); spin_lock(&dq_data_lock); for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + if (!dquots[cnt]) continue; - ret = check_idq(inode->i_dquot[cnt], 1, warntype + cnt); + ret = check_idq(dquots[cnt], 1, &warn[cnt]); if (ret) goto warn_put_all; } for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + if (!dquots[cnt]) continue; - dquot_incr_inodes(inode->i_dquot[cnt], 1); + dquot_incr_inodes(dquots[cnt], 1); } warn_put_all: spin_unlock(&dq_data_lock); if (ret == 0) - mark_all_dquot_dirty(inode->i_dquot); - flush_warnings(inode->i_dquot, warntype); + mark_all_dquot_dirty(dquots); up_read(&sb_dqopt(inode->i_sb)->dqptr_sem); + flush_warnings(warn); return ret; } EXPORT_SYMBOL(dquot_alloc_inode); @@ -1668,7 +1684,8 @@ EXPORT_SYMBOL(dquot_claim_space_nodirty); void __dquot_free_space(struct inode *inode, qsize_t number, int flags) { unsigned int cnt; - char warntype[MAXQUOTAS]; + struct dquot_warn warn[MAXQUOTAS]; + struct dquot **dquots = inode->i_dquot; int reserve = flags & DQUOT_SPACE_RESERVE; /* First test before acquiring mutex - solves deadlocks when we @@ -1681,23 +1698,28 @@ void __dquot_free_space(struct inode *inode, qsize_t number, int flags) down_read(&sb_dqopt(inode->i_sb)->dqptr_sem); spin_lock(&dq_data_lock); for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + int wtype; + + warn[cnt].w_type = QUOTA_NL_NOWARN; + if (!dquots[cnt]) continue; - warntype[cnt] = info_bdq_free(inode->i_dquot[cnt], number); + wtype = info_bdq_free(dquots[cnt], number); + if (wtype != QUOTA_NL_NOWARN) + prepare_warning(&warn[cnt], dquots[cnt], wtype); if (reserve) - dquot_free_reserved_space(inode->i_dquot[cnt], number); + dquot_free_reserved_space(dquots[cnt], number); else - dquot_decr_space(inode->i_dquot[cnt], number); + dquot_decr_space(dquots[cnt], number); } inode_decr_space(inode, number, reserve); spin_unlock(&dq_data_lock); if (reserve) goto out_unlock; - mark_all_dquot_dirty(inode->i_dquot); + mark_all_dquot_dirty(dquots); out_unlock: - flush_warnings(inode->i_dquot, warntype); up_read(&sb_dqopt(inode->i_sb)->dqptr_sem); + flush_warnings(warn); } EXPORT_SYMBOL(__dquot_free_space); @@ -1707,7 +1729,8 @@ EXPORT_SYMBOL(__dquot_free_space); void dquot_free_inode(const struct inode *inode) { unsigned int cnt; - char warntype[MAXQUOTAS]; + struct dquot_warn warn[MAXQUOTAS]; + struct dquot * const *dquots = inode->i_dquot; /* First test before acquiring mutex - solves deadlocks when we * re-enter the quota code and are already holding the mutex */ @@ -1717,15 +1740,20 @@ void dquot_free_inode(const struct inode *inode) down_read(&sb_dqopt(inode->i_sb)->dqptr_sem); spin_lock(&dq_data_lock); for (cnt = 0; cnt < MAXQUOTAS; cnt++) { - if (!inode->i_dquot[cnt]) + int wtype; + + warn[cnt].w_type = QUOTA_NL_NOWARN; + if (!dquots[cnt]) continue; - warntype[cnt] = info_idq_free(inode->i_dquot[cnt], 1); - dquot_decr_inodes(inode->i_dquot[cnt], 1); + wtype = info_idq_free(dquots[cnt], 1); + if (wtype != QUOTA_NL_NOWARN) + prepare_warning(&warn[cnt], dquots[cnt], wtype); + dquot_decr_inodes(dquots[cnt], 1); } spin_unlock(&dq_data_lock); - mark_all_dquot_dirty(inode->i_dquot); - flush_warnings(inode->i_dquot, warntype); + mark_all_dquot_dirty(dquots); up_read(&sb_dqopt(inode->i_sb)->dqptr_sem); + flush_warnings(warn); } EXPORT_SYMBOL(dquot_free_inode); @@ -1746,16 +1774,20 @@ int __dquot_transfer(struct inode *inode, struct dquot **transfer_to) struct dquot *transfer_from[MAXQUOTAS] = {}; int cnt, ret = 0; char is_valid[MAXQUOTAS] = {}; - char warntype_to[MAXQUOTAS]; - char warntype_from_inodes[MAXQUOTAS], warntype_from_space[MAXQUOTAS]; + struct dquot_warn warn_to[MAXQUOTAS]; + struct dquot_warn warn_from_inodes[MAXQUOTAS]; + struct dquot_warn warn_from_space[MAXQUOTAS]; /* First test before acquiring mutex - solves deadlocks when we * re-enter the quota code and are already holding the mutex */ if (IS_NOQUOTA(inode)) return 0; /* Initialize the arrays */ - for (cnt = 0; cnt < MAXQUOTAS; cnt++) - warntype_to[cnt] = QUOTA_NL_NOWARN; + for (cnt = 0; cnt < MAXQUOTAS; cnt++) { + warn_to[cnt].w_type = QUOTA_NL_NOWARN; + warn_from_inodes[cnt].w_type = QUOTA_NL_NOWARN; + warn_from_space[cnt].w_type = QUOTA_NL_NOWARN; + } down_write(&sb_dqopt(inode->i_sb)->dqptr_sem); if (IS_NOQUOTA(inode)) { /* File without quota accounting? */ up_write(&sb_dqopt(inode->i_sb)->dqptr_sem); @@ -1777,10 +1809,10 @@ int __dquot_transfer(struct inode *inode, struct dquot **transfer_to) continue; is_valid[cnt] = 1; transfer_from[cnt] = inode->i_dquot[cnt]; - ret = check_idq(transfer_to[cnt], 1, warntype_to + cnt); + ret = check_idq(transfer_to[cnt], 1, &warn_to[cnt]); if (ret) goto over_quota; - ret = check_bdq(transfer_to[cnt], space, 0, warntype_to + cnt); + ret = check_bdq(transfer_to[cnt], space, 0, &warn_to[cnt]); if (ret) goto over_quota; } @@ -1793,10 +1825,15 @@ int __dquot_transfer(struct inode *inode, struct dquot **transfer_to) continue; /* Due to IO error we might not have transfer_from[] structure */ if (transfer_from[cnt]) { - warntype_from_inodes[cnt] = - info_idq_free(transfer_from[cnt], 1); - warntype_from_space[cnt] = - info_bdq_free(transfer_from[cnt], space); + int wtype; + wtype = info_idq_free(transfer_from[cnt], 1); + if (wtype != QUOTA_NL_NOWARN) + prepare_warning(&warn_from_inodes[cnt], + transfer_from[cnt], wtype); + wtype = info_bdq_free(transfer_from[cnt], space); + if (wtype != QUOTA_NL_NOWARN) + prepare_warning(&warn_from_space[cnt], + transfer_from[cnt], wtype); dquot_decr_inodes(transfer_from[cnt], 1); dquot_decr_space(transfer_from[cnt], cur_space); dquot_free_reserved_space(transfer_from[cnt], @@ -1814,9 +1851,9 @@ int __dquot_transfer(struct inode *inode, struct dquot **transfer_to) mark_all_dquot_dirty(transfer_from); mark_all_dquot_dirty(transfer_to); - flush_warnings(transfer_to, warntype_to); - flush_warnings(transfer_from, warntype_from_inodes); - flush_warnings(transfer_from, warntype_from_space); + flush_warnings(warn_to); + flush_warnings(warn_from_inodes); + flush_warnings(warn_from_space); /* Pass back references to put */ for (cnt = 0; cnt < MAXQUOTAS; cnt++) if (is_valid[cnt]) @@ -1825,7 +1862,7 @@ int __dquot_transfer(struct inode *inode, struct dquot **transfer_to) over_quota: spin_unlock(&dq_data_lock); up_write(&sb_dqopt(inode->i_sb)->dqptr_sem); - flush_warnings(transfer_to, warntype_to); + flush_warnings(warn_to); return ret; } EXPORT_SYMBOL(__dquot_transfer); -- 1.7.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: Deadlock? 2012-02-14 12:47 ` Deadlock? Jan Kara @ 2012-02-15 11:42 ` Markus 0 siblings, 0 replies; 5+ messages in thread From: Markus @ 2012-02-15 11:42 UTC (permalink / raw) To: lkml; +Cc: Jan Kara, Jack Stone, akpm, adilger.kernel, gregkh Hi! /proc/sysrq-trigger does not exists, I will apply the patch and enable sysrq. But it may take some days for a result. Thanks, Markus Jan Kara schrieb am 14.02.2012: > On Mon 13-02-12 17:37:34, Jack Stone wrote: > > Adding CCs > > > > On 02/13/2012 11:17 AM, Markus wrote: > > > I noted some kind of deadlock, where I was not able to write to the raid6, > > > while writing to each disk would still work. > > > This caused many processes to "wait" in d-state, thus making it impossible to > > > unmount, cleanly reboot, sync, ... > Thanks for report. Could you please provide output of > 'echo w >/proc/sysrq-trigger' > in dmesg? That would help us narrow down the real cause the the current > deadlock. > > > > So I enabled the detection of hung tasks and this deplock option. > > > After running a 3.2.2 for about 5 days: > > > http://pastebin.com/gy1kaYmS > > > > > > I dont know if its a bug or nothing. Or if it has anything to do with my > > > problem as there was no hungtask detected and the raid still seems to work. > It's lock debugging code complaining about a possible problem it found. > Actually, the problem looks real (although hard to hit AFAICS) and attached > patch should fix it. Does the patch fix things for you? If not, please > provide the output I describe above. > > Honza > -- > Jan Kara <jack@suse.cz> > SUSE Labs, CR ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Deadlock? 2012-02-13 11:17 Deadlock? Markus 2012-02-13 17:37 ` Deadlock? Jack Stone @ 2012-02-14 3:16 ` Paul Hartman 1 sibling, 0 replies; 5+ messages in thread From: Paul Hartman @ 2012-02-14 3:16 UTC (permalink / raw) To: lkml On Mon, Feb 13, 2012 at 5:17 AM, Markus <M4rkusXXL@web.de> wrote: > Hi! > > I noted some kind of deadlock, where I was not able to write to the raid6, > while writing to each disk would still work. > This caused many processes to "wait" in d-state, thus making it impossible to > unmount, cleanly reboot, sync, ... I'm using RAID5 and kernel 3.2.2 and encountered a hang when running e4defrag on some large files today. When I tried to browse to that directory in Midnight Commander, it also became D state hung. After magic sysrq REISUB, raid is resyncing. I don't know if it's related, or useful, but here's my dmesg output: [1555304.508443] INFO: task e4defrag:9929 blocked for more than 120 seconds. [1555304.508446] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555304.508449] e4defrag D 000000015ccdc212 0 9929 21774 0x00000000 [1555304.508454] ffff880309b93d38 0000000000000082 ffff880309b93cc8 ffffffff00000000 [1555304.508458] ffff8801009d0000 0000000000010980 ffff880309b93fd8 ffff880309b92000 [1555304.508463] 0000000000010980 0000000000004000 ffff880309b93fd8 0000000000010980 [1555304.508467] Call Trace: [1555304.508474] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555304.508480] [<ffffffff81494f02>] ? _raw_spin_unlock_irqrestore+0x12/0x40 [1555304.508484] [<ffffffff810319ee>] ? __wake_up+0x4e/0x70 [1555304.508487] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555304.508490] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555304.508493] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555304.508496] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555304.508500] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555304.508504] [<ffffffff810a77d0>] ? __lock_page+0x70/0x70 [1555304.508507] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555304.508510] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555304.508513] [<ffffffff810a77d9>] sleep_on_page+0x9/0x10 [1555304.508516] [<ffffffff814934d7>] __wait_on_bit+0x57/0x80 [1555304.508520] [<ffffffff810a79fe>] wait_on_page_bit+0x6e/0x80 [1555304.508524] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555304.508528] [<ffffffff810b2310>] ? pagevec_lookup_tag+0x20/0x30 [1555304.508531] [<ffffffff810a80ca>] filemap_fdatawait_range+0xfa/0x190 [1555304.508537] [<ffffffff81116e76>] sys_sync_file_range+0x176/0x180 [1555304.508541] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555424.340835] INFO: task e4defrag:9929 blocked for more than 120 seconds. [1555424.340838] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555424.340841] e4defrag D 000000015ccdc212 0 9929 21774 0x00000000 [1555424.340845] ffff880309b93d38 0000000000000082 ffff880309b93cc8 ffffffff00000000 [1555424.340850] ffff8801009d0000 0000000000010980 ffff880309b93fd8 ffff880309b92000 [1555424.340854] 0000000000010980 0000000000004000 ffff880309b93fd8 0000000000010980 [1555424.340859] Call Trace: [1555424.340866] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555424.340872] [<ffffffff81494f02>] ? _raw_spin_unlock_irqrestore+0x12/0x40 [1555424.340875] [<ffffffff810319ee>] ? __wake_up+0x4e/0x70 [1555424.340878] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555424.340881] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555424.340885] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555424.340888] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555424.340891] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555424.340895] [<ffffffff810a77d0>] ? __lock_page+0x70/0x70 [1555424.340899] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555424.340902] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555424.340905] [<ffffffff810a77d9>] sleep_on_page+0x9/0x10 [1555424.340908] [<ffffffff814934d7>] __wait_on_bit+0x57/0x80 [1555424.340911] [<ffffffff810a79fe>] wait_on_page_bit+0x6e/0x80 [1555424.340916] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555424.340919] [<ffffffff810b2310>] ? pagevec_lookup_tag+0x20/0x30 [1555424.340923] [<ffffffff810a80ca>] filemap_fdatawait_range+0xfa/0x190 [1555424.340928] [<ffffffff81116e76>] sys_sync_file_range+0x176/0x180 [1555424.340932] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555544.173120] INFO: task jbd2/md2-8:2842 blocked for more than 120 seconds. [1555544.173123] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555544.173126] jbd2/md2-8 D 000000015cd20110 0 2842 2 0x00000000 [1555544.173131] ffff88030fba1ce0 0000000000000046 0000000000000002 0000000100000000 [1555544.173136] ffff88030f9c1650 0000000000010980 ffff88030fba1fd8 ffff88030fba0000 [1555544.173140] 0000000000010980 0000000000004000 ffff88030fba1fd8 0000000000010980 [1555544.173144] Call Trace: [1555544.173152] [<ffffffff8100a1b8>] ? native_sched_clock+0x28/0x90 [1555544.173158] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555544.173163] [<ffffffff811932ac>] jbd2_journal_commit_transaction+0x17c/0x13a0 [1555544.173166] [<ffffffff8149259e>] ? __schedule+0x46e/0xa90 [1555544.173171] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173175] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173179] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555544.173183] [<ffffffff81197472>] kjournald2+0xb2/0x210 [1555544.173187] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555544.173190] [<ffffffff811973c0>] ? commit_timeout+0x10/0x10 [1555544.173194] [<ffffffff8105f666>] kthread+0x96/0xa0 [1555544.173198] [<ffffffff81496f74>] kernel_thread_helper+0x4/0x10 [1555544.173204] [<ffffffff8105f5d0>] ? kthread_worker_fn+0x190/0x190 [1555544.173207] [<ffffffff81496f70>] ? gs_change+0xb/0xb [1555544.173231] INFO: task e4defrag:9929 blocked for more than 120 seconds. [1555544.173233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555544.173235] e4defrag D 000000015ccdc212 0 9929 21774 0x00000000 [1555544.173238] ffff880309b93d38 0000000000000082 ffff880309b93cc8 ffffffff00000000 [1555544.173242] ffff8801009d0000 0000000000010980 ffff880309b93fd8 ffff880309b92000 [1555544.173245] 0000000000010980 0000000000004000 ffff880309b93fd8 0000000000010980 [1555544.173249] Call Trace: [1555544.173252] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173256] [<ffffffff81494f02>] ? _raw_spin_unlock_irqrestore+0x12/0x40 [1555544.173259] [<ffffffff810319ee>] ? __wake_up+0x4e/0x70 [1555544.173261] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173264] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173267] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173270] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173272] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173276] [<ffffffff810a77d0>] ? __lock_page+0x70/0x70 [1555544.173279] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555544.173281] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555544.173284] [<ffffffff810a77d9>] sleep_on_page+0x9/0x10 [1555544.173287] [<ffffffff814934d7>] __wait_on_bit+0x57/0x80 [1555544.173290] [<ffffffff810a79fe>] wait_on_page_bit+0x6e/0x80 [1555544.173293] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555544.173296] [<ffffffff810b2310>] ? pagevec_lookup_tag+0x20/0x30 [1555544.173299] [<ffffffff810a80ca>] filemap_fdatawait_range+0xfa/0x190 [1555544.173303] [<ffffffff81116e76>] sys_sync_file_range+0x176/0x180 [1555544.173307] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555544.173310] INFO: task mc:22249 blocked for more than 120 seconds. [1555544.173312] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555544.173313] mc D 000000015cd1eecf 0 22249 9847 0x00000000 [1555544.173317] ffff8802f7ea9b48 0000000000000082 ffff8802f7ea9a28 0000000000000000 [1555544.173321] ffff8802fc390000 0000000000010980 ffff8802f7ea9fd8 ffff8802f7ea8000 [1555544.173324] 0000000000010980 0000000000004000 ffff8802f7ea9fd8 0000000000010980 [1555544.173327] Call Trace: [1555544.173331] [<ffffffff810b3242>] ? __lru_cache_add+0x72/0xd0 [1555544.173334] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173337] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173339] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173342] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555544.173346] [<ffffffff811194d0>] ? unmap_underlying_metadata+0x50/0x50 [1555544.173349] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555544.173351] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555544.173354] [<ffffffff811194d9>] sleep_on_buffer+0x9/0x10 [1555544.173357] [<ffffffff81493392>] __wait_on_bit_lock+0x52/0xb0 [1555544.173360] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173363] [<ffffffff811194d0>] ? unmap_underlying_metadata+0x50/0x50 [1555544.173366] [<ffffffff81493463>] out_of_line_wait_on_bit_lock+0x73/0x90 [1555544.173369] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555544.173372] [<ffffffff8111a6be>] __lock_buffer+0x2e/0x30 [1555544.173375] [<ffffffff811924e9>] do_get_write_access+0x5a9/0x6c0 [1555544.173379] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555544.173382] [<ffffffff81198dc6>] ? jbd2_journal_add_journal_head+0xd6/0x1f0 [1555544.173385] [<ffffffff811927dc>] jbd2_journal_get_write_access+0x2c/0x50 [1555544.173389] [<ffffffff8117e3b9>] __ext4_journal_get_write_access+0x39/0x80 [1555544.173392] [<ffffffff81164f08>] ext4_reserve_inode_write+0x88/0xb0 [1555544.173395] [<ffffffff81164f69>] ext4_mark_inode_dirty+0x39/0x1e0 [1555544.173399] [<ffffffff810ff8a0>] ? filldir64+0xd0/0xd0 [1555544.173402] [<ffffffff81167148>] ext4_dirty_inode+0x38/0x60 [1555544.173404] [<ffffffff8111246b>] __mark_inode_dirty+0x3b/0x220 [1555544.173407] [<ffffffff81106445>] touch_atime+0x115/0x160 [1555544.173410] [<ffffffff810ffae6>] vfs_readdir+0xb6/0xc0 [1555544.173413] [<ffffffff810ffbd0>] sys_getdents+0x80/0xe0 [1555544.173416] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555664.005410] INFO: task jbd2/md2-8:2842 blocked for more than 120 seconds. [1555664.005413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555664.005416] jbd2/md2-8 D 000000015cd20110 0 2842 2 0x00000000 [1555664.005421] ffff88030fba1ce0 0000000000000046 0000000000000002 0000000100000000 [1555664.005426] ffff88030f9c1650 0000000000010980 ffff88030fba1fd8 ffff88030fba0000 [1555664.005430] 0000000000010980 0000000000004000 ffff88030fba1fd8 0000000000010980 [1555664.005434] Call Trace: [1555664.005442] [<ffffffff8100a1b8>] ? native_sched_clock+0x28/0x90 [1555664.005448] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555664.005453] [<ffffffff811932ac>] jbd2_journal_commit_transaction+0x17c/0x13a0 [1555664.005456] [<ffffffff8149259e>] ? __schedule+0x46e/0xa90 [1555664.005462] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005465] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005469] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555664.005473] [<ffffffff81197472>] kjournald2+0xb2/0x210 [1555664.005477] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555664.005480] [<ffffffff811973c0>] ? commit_timeout+0x10/0x10 [1555664.005483] [<ffffffff8105f666>] kthread+0x96/0xa0 [1555664.005488] [<ffffffff81496f74>] kernel_thread_helper+0x4/0x10 [1555664.005492] [<ffffffff8105f5d0>] ? kthread_worker_fn+0x190/0x190 [1555664.005495] [<ffffffff81496f70>] ? gs_change+0xb/0xb [1555664.005521] INFO: task e4defrag:9929 blocked for more than 120 seconds. [1555664.005523] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555664.005525] e4defrag D 000000015ccdc212 0 9929 21774 0x00000000 [1555664.005529] ffff880309b93d38 0000000000000082 ffff880309b93cc8 ffffffff00000000 [1555664.005533] ffff8801009d0000 0000000000010980 ffff880309b93fd8 ffff880309b92000 [1555664.005537] 0000000000010980 0000000000004000 ffff880309b93fd8 0000000000010980 [1555664.005542] Call Trace: [1555664.005545] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005549] [<ffffffff81494f02>] ? _raw_spin_unlock_irqrestore+0x12/0x40 [1555664.005553] [<ffffffff810319ee>] ? __wake_up+0x4e/0x70 [1555664.005556] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005559] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005562] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005566] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005569] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005573] [<ffffffff810a77d0>] ? __lock_page+0x70/0x70 [1555664.005576] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555664.005579] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555664.005582] [<ffffffff810a77d9>] sleep_on_page+0x9/0x10 [1555664.005585] [<ffffffff814934d7>] __wait_on_bit+0x57/0x80 [1555664.005589] [<ffffffff810a79fe>] wait_on_page_bit+0x6e/0x80 [1555664.005592] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555664.005596] [<ffffffff810b2310>] ? pagevec_lookup_tag+0x20/0x30 [1555664.005599] [<ffffffff810a80ca>] filemap_fdatawait_range+0xfa/0x190 [1555664.005604] [<ffffffff81116e76>] sys_sync_file_range+0x176/0x180 [1555664.005608] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555664.005612] INFO: task mc:22249 blocked for more than 120 seconds. [1555664.005614] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555664.005616] mc D 000000015cd1eecf 0 22249 9847 0x00000000 [1555664.005620] ffff8802f7ea9b48 0000000000000082 ffff8802f7ea9a28 0000000000000000 [1555664.005624] ffff8802fc390000 0000000000010980 ffff8802f7ea9fd8 ffff8802f7ea8000 [1555664.005628] 0000000000010980 0000000000004000 ffff8802f7ea9fd8 0000000000010980 [1555664.005632] Call Trace: [1555664.005636] [<ffffffff810b3242>] ? __lru_cache_add+0x72/0xd0 [1555664.005639] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005642] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005646] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005649] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555664.005653] [<ffffffff811194d0>] ? unmap_underlying_metadata+0x50/0x50 [1555664.005656] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555664.005659] [<ffffffff81492de7>] io_schedule+0x87/0xd0 [1555664.005663] [<ffffffff811194d9>] sleep_on_buffer+0x9/0x10 [1555664.005666] [<ffffffff81493392>] __wait_on_bit_lock+0x52/0xb0 [1555664.005669] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005673] [<ffffffff811194d0>] ? unmap_underlying_metadata+0x50/0x50 [1555664.005676] [<ffffffff81493463>] out_of_line_wait_on_bit_lock+0x73/0x90 [1555664.005680] [<ffffffff8105fb80>] ? autoremove_wake_function+0x40/0x40 [1555664.005684] [<ffffffff8111a6be>] __lock_buffer+0x2e/0x30 [1555664.005687] [<ffffffff811924e9>] do_get_write_access+0x5a9/0x6c0 [1555664.005691] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555664.005695] [<ffffffff81198dc6>] ? jbd2_journal_add_journal_head+0xd6/0x1f0 [1555664.005698] [<ffffffff811927dc>] jbd2_journal_get_write_access+0x2c/0x50 [1555664.005703] [<ffffffff8117e3b9>] __ext4_journal_get_write_access+0x39/0x80 [1555664.005707] [<ffffffff81164f08>] ext4_reserve_inode_write+0x88/0xb0 [1555664.005710] [<ffffffff81164f69>] ext4_mark_inode_dirty+0x39/0x1e0 [1555664.005714] [<ffffffff810ff8a0>] ? filldir64+0xd0/0xd0 [1555664.005717] [<ffffffff81167148>] ext4_dirty_inode+0x38/0x60 [1555664.005721] [<ffffffff8111246b>] __mark_inode_dirty+0x3b/0x220 [1555664.005724] [<ffffffff81106445>] touch_atime+0x115/0x160 [1555664.005728] [<ffffffff810ffae6>] vfs_readdir+0xb6/0xc0 [1555664.005731] [<ffffffff810ffbd0>] sys_getdents+0x80/0xe0 [1555664.005734] [<ffffffff81495b7b>] system_call_fastpath+0x16/0x1b [1555783.837700] INFO: task jbd2/md2-8:2842 blocked for more than 120 seconds. [1555783.837703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1555783.837706] jbd2/md2-8 D 000000015cd20110 0 2842 2 0x00000000 [1555783.837711] ffff88030fba1ce0 0000000000000046 0000000000000002 0000000100000000 [1555783.837715] ffff88030f9c1650 0000000000010980 ffff88030fba1fd8 ffff88030fba0000 [1555783.837719] 0000000000010980 0000000000004000 ffff88030fba1fd8 0000000000010980 [1555783.837724] Call Trace: [1555783.837731] [<ffffffff8100a1b8>] ? native_sched_clock+0x28/0x90 [1555783.837737] [<ffffffff81492d4a>] schedule+0x3a/0x50 [1555783.837742] [<ffffffff811932ac>] jbd2_journal_commit_transaction+0x17c/0x13a0 [1555783.837746] [<ffffffff8149259e>] ? __schedule+0x46e/0xa90 [1555783.837751] [<ffffffff8103a671>] ? get_parent_ip+0x11/0x50 [1555783.837754] [<ffffffff8103a74d>] ? sub_preempt_count+0x9d/0xd0 [1555783.837759] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555783.837763] [<ffffffff81197472>] kjournald2+0xb2/0x210 [1555783.837766] [<ffffffff8105fb40>] ? wake_up_bit+0x40/0x40 [1555783.837770] [<ffffffff811973c0>] ? commit_timeout+0x10/0x10 [1555783.837773] [<ffffffff8105f666>] kthread+0x96/0xa0 [1555783.837777] [<ffffffff81496f74>] kernel_thread_helper+0x4/0x10 [1555783.837781] [<ffffffff8105f5d0>] ? kthread_worker_fn+0x190/0x190 [1555783.837784] [<ffffffff81496f70>] ? gs_change+0xb/0xb ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-02-15 22:19 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-02-13 11:17 Deadlock? Markus 2012-02-13 17:37 ` Deadlock? Jack Stone 2012-02-14 12:47 ` Deadlock? Jan Kara 2012-02-15 11:42 ` Deadlock? Markus 2012-02-14 3:16 ` Deadlock? Paul Hartman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).