* XFS deadlock in 2.6.37
@ 2011-01-20 17:08 Malcolm Scott
2011-01-20 17:27 ` William Pitcock
2011-01-21 5:28 ` [PATCH] " Dave Chinner
0 siblings, 2 replies; 4+ messages in thread
From: Malcolm Scott @ 2011-01-20 17:08 UTC (permalink / raw)
To: linux-kernel
Hi all,
I've had the following deadlock happen twice on a 2.6.37 system with several
XFS filesystems (including root) and no swap (may be relevant, considering
that kswapd is one task involved here). Some minor filesystem corruption
resulted (but maybe only because the root fs couldn't be synced/umounted).
If you need any more info, please let me know.
--- first crash ---
[504603.250208] INFO: task kswapd0:37 blocked for more than 120 seconds.
[504603.261107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504603.273465] kswapd0 D 0000000000000003 0 37 2 0x00000000
[504603.273473] ffff88034428bc10 0000000000000046 ffff88034428bfd8 ffff88034428a000
[504603.273479] 0000000000013a80 ffff8803442903a0 ffff88034428bfd8 0000000000013a80
[504603.273483] ffff88034572ad80 ffff880344290000 ffffffffffffff10 ffff880343a51e28
[504603.273488] Call Trace:
[504603.273500] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504603.273504] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504603.273541] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504603.273554] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504603.273567] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504603.273573] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504603.273577] [<ffffffff81120420>] balance_pgdat+0x2b0/0x530
[504603.273580] [<ffffffff811207df>] kswapd+0x13f/0x2b0
[504603.273585] [<ffffffff81087d30>] ? autoremove_wake_function+0x0/0x40
[504603.273588] [<ffffffff811206a0>] ? kswapd+0x0/0x2b0
[504603.273591] [<ffffffff81087606>] kthread+0x96/0xa0
[504603.273596] [<ffffffff8100cea4>] kernel_thread_helper+0x4/0x10
[504603.273599] [<ffffffff81087570>] ? kthread+0x0/0xa0
[504603.273603] [<ffffffff8100cea0>] ? kernel_thread_helper+0x0/0x10
[504603.273688] INFO: task apache2:28657 blocked for more than 120 seconds.
[504603.285217] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504603.298145] apache2 D 0000000000000003 0 28657 12480 0x00000000
[504603.298151] ffff8802f91c7a10 0000000000000082 ffff8802f91c7fd8 ffff8802f91c6000
[504603.298155] 0000000000013a80 ffff8802b60703a0 ffff8802f91c7fd8 0000000000013a80
[504603.298160] ffff88034713c440 ffff8802b6070000 0000000000000000 ffff880343a51e28
[504603.298165] Call Trace:
[504603.298174] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504603.298177] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504603.298208] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504603.298221] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504603.298234] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504603.298240] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504603.298243] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504603.298249] [<ffffffff814cefa0>] ? skb_release_data+0x90/0xe0
[504603.298252] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504603.298256] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504603.298262] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504603.298265] [<ffffffff81111f9e>] __get_free_pages+0xe/0x50
[504603.298270] [<ffffffff81063ec6>] dup_task_struct+0x46/0x190
[504603.298273] [<ffffffff8106442e>] copy_process+0xbe/0xe70
[504603.298278] [<ffffffff8104ec1e>] ? __wake_up_sync_key+0x5e/0x80
[504603.298281] [<ffffffff810652bd>] do_fork+0x8d/0x430
[504603.298286] [<ffffffff81166825>] ? __fput+0x155/0x220
[504603.298289] [<ffffffff81166915>] ? fput+0x25/0x30
[504603.298292] [<ffffffff81163240>] ? filp_close+0x60/0x90
[504603.298296] [<ffffffff810150c8>] sys_clone+0x28/0x30
[504603.298300] [<ffffffff8100c3a3>] stub_clone+0x13/0x20
[504603.298303] [<ffffffff8100c082>] ? system_call_fastpath+0x16/0x1b
[504603.298371] INFO: task Interface.cgi:14369 blocked for more than 120 seconds.
[504603.310680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504603.323585] Interface.cgi D 0000000000000001 0 14369 28657 0x00000000
[504603.323590] ffff880046155708 0000000000000086 ffff880046155fd8 ffff880046154000
[504603.323595] 0000000000013a80 ffff8800050edea0 ffff880046155fd8 0000000000013a80
[504603.323600] ffff880347120000 ffff8800050edb00 ffff880046155708 ffff880343a51e28
[504603.323605] Call Trace:
[504603.323612] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504603.323616] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504603.323641] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504603.323654] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504603.323667] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504603.323672] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504603.323675] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504603.323679] [<ffffffff81114d12>] ? prep_new_page+0x142/0x1b0
[504603.323682] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504603.323685] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504603.323691] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504603.323695] [<ffffffff81153095>] new_slab+0x1f5/0x290
[504603.323698] [<ffffffff81154a62>] __slab_alloc+0x1b2/0x390
[504603.323703] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[504603.323706] [<ffffffff81157f7d>] __kmalloc_node_track_caller+0x8d/0x1a0
[504603.323710] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[504603.323713] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[504603.323717] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[504603.323721] [<ffffffff81039459>] ? default_spin_lock_flags+0x9/0x10
[504603.323725] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[504603.323730] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[504603.323734] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[504603.323738] [<ffffffff8116e761>] ? pipe_read+0x2b1/0x4b0
[504603.323742] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[504603.323748] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[504603.323753] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[504603.323756] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[504603.323759] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[504603.323763] [<ffffffff81165771>] sys_write+0x51/0x90
[504603.323766] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[504603.323770] INFO: task php5-cgi:14376 blocked for more than 120 seconds.
[504603.335893] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504603.349265] php5-cgi D 0000000000000001 0 14376 28657 0x00000000
[504603.349270] ffff88003a02da10 0000000000000086 ffff88003a02dfd8 ffff88003a02c000
[504603.349275] 0000000000013a80 ffff88008cfcdea0 ffff88003a02dfd8 0000000000013a80
[504603.349280] ffff880346b8db00 ffff88008cfcdb00 0000000000000000 ffff880343a51e28
[504603.349284] Call Trace:
[504603.349290] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504603.349293] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504603.349314] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504603.349326] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504603.349339] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504603.349343] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504603.349346] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504603.349350] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504603.349353] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504603.349358] [<ffffffff812dc8ee>] ? radix_tree_lookup_slot+0xe/0x10
[504603.349364] [<ffffffff8115ff5d>] ? mem_cgroup_charge_common+0x6d/0x90
[504603.349367] [<ffffffff81119d9d>] ? lru_cache_add_lru+0x2d/0x50
[504603.349371] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504603.349374] [<ffffffff81111f9e>] __get_free_pages+0xe/0x50
[504603.349379] [<ffffffff81063ec6>] dup_task_struct+0x46/0x190
[504603.349382] [<ffffffff8106442e>] copy_process+0xbe/0xe70
[504603.349386] [<ffffffff8113846c>] ? change_protection.clone.4+0x21c/0x250
[504603.349391] [<ffffffff815d3bb8>] ? do_page_fault+0x268/0x560
[504603.349394] [<ffffffff810652bd>] do_fork+0x8d/0x430
[504603.349397] [<ffffffff8113861d>] ? mprotect_fixup+0x17d/0x2d0
[504603.349401] [<ffffffff810150c8>] sys_clone+0x28/0x30
[504603.349405] [<ffffffff8100c3a3>] stub_clone+0x13/0x20
[504603.349408] [<ffffffff8100c082>] ? system_call_fastpath+0x16/0x1b
[504603.349411] INFO: task quota:14379 blocked for more than 120 seconds.
[504603.361741] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504603.375377] quota D 0000000000000000 0 14379 31799 0x00000000
[504603.375382] ffff880064cafca8 0000000000000082 ffff880064caffd8 ffff880064cae000
[504603.375387] 0000000000013a80 ffff88024f7e9a60 ffff880064caffd8 0000000000013a80
[504603.375392] ffff8802dce4db00 ffff88024f7e96c0 0000000000000000 ffff880343a51e28
[504603.375396] Call Trace:
[504603.375402] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504603.375422] [<ffffffffa00cc7d7>] ? xfs_qm_dqget+0x317/0x420 [xfs]
[504603.375426] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504603.375438] [<ffffffffa00cc1a5>] xfs_qm_dqput+0x165/0x1b0 [xfs]
[504603.375451] [<ffffffffa00cefe6>] xfs_qm_scall_getquota+0xc6/0xe0 [xfs]
[504603.375464] [<ffffffffa00d29a6>] xfs_fs_get_dqblk+0x66/0x70 [xfs]
[504603.375469] [<ffffffff811bbeee>] quota_getxquota+0x3e/0x90
[504603.375473] [<ffffffff811bc450>] do_quotactl+0x390/0x510
[504603.375478] [<ffffffff8117a77f>] ? dput+0x8f/0x160
[504603.375481] [<ffffffff811702c1>] ? path_put+0x31/0x40
[504603.375484] [<ffffffff811bc680>] sys_quotactl+0xb0/0x130
[504603.375488] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[504723.370635] INFO: task kswapd0:37 blocked for more than 120 seconds.
[504723.382903] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504723.396847] kswapd0 D 0000000000000003 0 37 2 0x00000000
[504723.396853] ffff88034428bc10 0000000000000046 ffff88034428bfd8 ffff88034428a000
[504723.396858] 0000000000013a80 ffff8803442903a0 ffff88034428bfd8 0000000000013a80
[504723.396863] ffff88034572ad80 ffff880344290000 ffffffffffffff10 ffff880343a51e28
[504723.396868] Call Trace:
[504723.396879] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504723.396883] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504723.396920] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504723.396933] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504723.396946] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504723.396953] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504723.396956] [<ffffffff81120420>] balance_pgdat+0x2b0/0x530
[504723.396960] [<ffffffff811207df>] kswapd+0x13f/0x2b0
[504723.396965] [<ffffffff81087d30>] ? autoremove_wake_function+0x0/0x40
[504723.396968] [<ffffffff811206a0>] ? kswapd+0x0/0x2b0
[504723.396971] [<ffffffff81087606>] kthread+0x96/0xa0
[504723.396976] [<ffffffff8100cea4>] kernel_thread_helper+0x4/0x10
[504723.396979] [<ffffffff81087570>] ? kthread+0x0/0xa0
[504723.396982] [<ffffffff8100cea0>] ? kernel_thread_helper+0x0/0x10
[504723.397065] INFO: task apache2:28657 blocked for more than 120 seconds.
[504723.409887] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504723.424100] apache2 D 0000000000000003 0 28657 12480 0x00000000
[504723.424105] ffff8802f91c7a10 0000000000000082 ffff8802f91c7fd8 ffff8802f91c6000
[504723.424110] 0000000000013a80 ffff8802b60703a0 ffff8802f91c7fd8 0000000000013a80
[504723.424115] ffff88034713c440 ffff8802b6070000 0000000000000000 ffff880343a51e28
[504723.424119] Call Trace:
[504723.424125] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504723.424129] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504723.424149] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504723.424161] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504723.424174] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504723.424178] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504723.424181] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504723.424186] [<ffffffff814cefa0>] ? skb_release_data+0x90/0xe0
[504723.424190] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504723.424193] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504723.424199] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504723.424202] [<ffffffff81111f9e>] __get_free_pages+0xe/0x50
[504723.424207] [<ffffffff81063ec6>] dup_task_struct+0x46/0x190
[504723.424210] [<ffffffff8106442e>] copy_process+0xbe/0xe70
[504723.424214] [<ffffffff8104ec1e>] ? __wake_up_sync_key+0x5e/0x80
[504723.424218] [<ffffffff810652bd>] do_fork+0x8d/0x430
[504723.424222] [<ffffffff81166825>] ? __fput+0x155/0x220
[504723.424226] [<ffffffff81166915>] ? fput+0x25/0x30
[504723.424229] [<ffffffff81163240>] ? filp_close+0x60/0x90
[504723.424233] [<ffffffff810150c8>] sys_clone+0x28/0x30
[504723.424236] [<ffffffff8100c3a3>] stub_clone+0x13/0x20
[504723.424239] [<ffffffff8100c082>] ? system_call_fastpath+0x16/0x1b
[504723.424304] INFO: task Interface.cgi:14369 blocked for more than 120 seconds.
[504723.437737] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504723.452141] Interface.cgi D 0000000000000001 0 14369 28657 0x00000000
[504723.452145] ffff880046155708 0000000000000086 ffff880046155fd8 ffff880046154000
[504723.452150] 0000000000013a80 ffff8800050edea0 ffff880046155fd8 0000000000013a80
[504723.452154] ffff880347120000 ffff8800050edb00 ffff880046155708 ffff880343a51e28
[504723.452159] Call Trace:
[504723.452164] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504723.452168] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504723.452188] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504723.452201] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504723.452214] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504723.452218] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504723.452221] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504723.452224] [<ffffffff81114d12>] ? prep_new_page+0x142/0x1b0
[504723.452227] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504723.452231] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504723.452236] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504723.452240] [<ffffffff81153095>] new_slab+0x1f5/0x290
[504723.452243] [<ffffffff81154a62>] __slab_alloc+0x1b2/0x390
[504723.452247] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[504723.452251] [<ffffffff81157f7d>] __kmalloc_node_track_caller+0x8d/0x1a0
[504723.452254] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[504723.452258] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[504723.452261] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[504723.452265] [<ffffffff81039459>] ? default_spin_lock_flags+0x9/0x10
[504723.452269] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[504723.452274] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[504723.452278] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[504723.452282] [<ffffffff8116e761>] ? pipe_read+0x2b1/0x4b0
[504723.452286] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[504723.452291] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[504723.452296] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[504723.452299] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[504723.452302] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[504723.452305] [<ffffffff81165771>] sys_write+0x51/0x90
[504723.452309] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[504723.452312] INFO: task php5-cgi:14376 blocked for more than 120 seconds.
[504723.465555] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504723.480032] php5-cgi D 0000000000000001 0 14376 28657 0x00000000
[504723.480037] ffff88003a02da10 0000000000000086 ffff88003a02dfd8 ffff88003a02c000
[504723.480042] 0000000000013a80 ffff88008cfcdea0 ffff88003a02dfd8 0000000000013a80
[504723.480046] ffff880346b8db00 ffff88008cfcdb00 0000000000000000 ffff880343a51e28
[504723.480051] Call Trace:
[504723.480056] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504723.480060] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504723.480078] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[504723.480090] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[504723.480103] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
[504723.480107] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[504723.480110] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[504723.480114] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[504723.480117] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[504723.480122] [<ffffffff812dc8ee>] ? radix_tree_lookup_slot+0xe/0x10
[504723.480127] [<ffffffff8115ff5d>] ? mem_cgroup_charge_common+0x6d/0x90
[504723.480131] [<ffffffff81119d9d>] ? lru_cache_add_lru+0x2d/0x50
[504723.480135] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[504723.480138] [<ffffffff81111f9e>] __get_free_pages+0xe/0x50
[504723.480142] [<ffffffff81063ec6>] dup_task_struct+0x46/0x190
[504723.480145] [<ffffffff8106442e>] copy_process+0xbe/0xe70
[504723.480149] [<ffffffff8113846c>] ? change_protection.clone.4+0x21c/0x250
[504723.480154] [<ffffffff815d3bb8>] ? do_page_fault+0x268/0x560
[504723.480157] [<ffffffff810652bd>] do_fork+0x8d/0x430
[504723.480160] [<ffffffff8113861d>] ? mprotect_fixup+0x17d/0x2d0
[504723.480164] [<ffffffff810150c8>] sys_clone+0x28/0x30
[504723.480167] [<ffffffff8100c3a3>] stub_clone+0x13/0x20
[504723.480170] [<ffffffff8100c082>] ? system_call_fastpath+0x16/0x1b
[504723.480174] INFO: task quota:14379 blocked for more than 120 seconds.
[504723.493339] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[504723.508084] quota D 0000000000000000 0 14379 31799 0x00000000
[504723.508089] ffff880064cafca8 0000000000000082 ffff880064caffd8 ffff880064cae000
[504723.508094] 0000000000013a80 ffff88024f7e9a60 ffff880064caffd8 0000000000013a80
[504723.508098] ffff8802dce4db00 ffff88024f7e96c0 0000000000000000 ffff880343a51e28
[504723.508103] Call Trace:
[504723.508108] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[504723.508126] [<ffffffffa00cc7d7>] ? xfs_qm_dqget+0x317/0x420 [xfs]
[504723.508129] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[504723.508142] [<ffffffffa00cc1a5>] xfs_qm_dqput+0x165/0x1b0 [xfs]
[504723.508155] [<ffffffffa00cefe6>] xfs_qm_scall_getquota+0xc6/0xe0 [xfs]
[504723.508168] [<ffffffffa00d29a6>] xfs_fs_get_dqblk+0x66/0x70 [xfs]
[504723.508172] [<ffffffff811bbeee>] quota_getxquota+0x3e/0x90
[504723.508176] [<ffffffff811bc450>] do_quotactl+0x390/0x510
[504723.508181] [<ffffffff8117a77f>] ? dput+0x8f/0x160
[504723.508184] [<ffffffff811702c1>] ? path_put+0x31/0x40
[504723.508187] [<ffffffff811bc680>] sys_quotactl+0xb0/0x130
[504723.508191] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
--- second crash ---
[224162.720509] INFO: task kswapd0:37 blocked for more than 120 seconds.
[224162.732833] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224162.746586] kswapd0 D 0000000000000000 0 37 2 0x00000000
[224162.746593] ffff88034428bc10 0000000000000046 ffff88034428bfd8 ffff88034428a000
[224162.746598] 0000000000013a80 ffff8803466c5ea0 ffff88034428bfd8 0000000000013a80
[224162.746603] ffff8803467fc440 ffff8803466c5b00 ffffffffffffff10 ffff880343d41a28
[224162.746608] Call Trace:
[224162.746619] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224162.746624] [<ffffffff815cfb6e>] ? _raw_spin_lock+0xe/0x20
[224162.746627] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224162.746667] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224162.746680] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224162.746693] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224162.746700] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224162.746703] [<ffffffff81120420>] balance_pgdat+0x2b0/0x530
[224162.746707] [<ffffffff811207df>] kswapd+0x13f/0x2b0
[224162.746712] [<ffffffff81087d30>] ? autoremove_wake_function+0x0/0x40
[224162.746715] [<ffffffff811206a0>] ? kswapd+0x0/0x2b0
[224162.746718] [<ffffffff81087606>] kthread+0x96/0xa0
[224162.746722] [<ffffffff8100cea4>] kernel_thread_helper+0x4/0x10
[224162.746726] [<ffffffff81087570>] ? kthread+0x0/0xa0
[224162.746729] [<ffffffff8100cea0>] ? kernel_thread_helper+0x0/0x10
[224162.746911] INFO: task quota:7861 blocked for more than 120 seconds.
[224162.759045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224162.772610] quota D 0000000000000001 0 7861 13705 0x00000000
[224162.772615] ffff88008f3a3ca8 0000000000000086 ffff88008f3a3fd8 ffff88008f3a2000
[224162.772619] 0000000000013a80 ffff88008dec3120 ffff88008f3a3fd8 0000000000013a80
[224162.772624] ffff8801198a8000 ffff88008dec2d80 ffffffff81060ac7 ffff880343d41a28
[224162.772628] Call Trace:
[224162.772633] [<ffffffff81060ac7>] ? mutex_spin_on_owner+0x97/0xd0
[224162.772638] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224162.772656] [<ffffffffa00d87d7>] ? xfs_qm_dqget+0x317/0x420 [xfs]
[224162.772660] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224162.772672] [<ffffffffa00d81a5>] xfs_qm_dqput+0x165/0x1b0 [xfs]
[224162.772685] [<ffffffffa00dafe6>] xfs_qm_scall_getquota+0xc6/0xe0 [xfs]
[224162.772698] [<ffffffffa00de9a6>] xfs_fs_get_dqblk+0x66/0x70 [xfs]
[224162.772703] [<ffffffff811bbeee>] quota_getxquota+0x3e/0x90
[224162.772707] [<ffffffff811bc450>] do_quotactl+0x390/0x510
[224162.772712] [<ffffffff8117a77f>] ? dput+0x8f/0x160
[224162.772716] [<ffffffff811702c1>] ? path_put+0x31/0x40
[224162.772719] [<ffffffff811bc680>] sys_quotactl+0xb0/0x130
[224162.772723] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224162.772727] INFO: task mutt_dotlock:8253 blocked for more than 120 seconds.
[224162.785572] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224162.799314] mutt_dotlock D 0000000000000002 0 8253 8252 0x00000000
[224162.799318] ffff880078ca9cb8 0000000000000086 ffff880078ca9fd8 ffff880078ca8000
[224162.799323] 0000000000013a80 ffff880075c71a60 ffff880078ca9fd8 0000000000013a80
[224162.799328] ffff88030dae96c0 ffff880075c716c0 ffff880343550000 ffff880343d41a28
[224162.799332] Call Trace:
[224162.799337] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224162.799341] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224162.799358] [<ffffffffa00d81a5>] xfs_qm_dqput+0x165/0x1b0 [xfs]
[224162.799370] [<ffffffffa00d821e>] xfs_qm_dqrele+0x2e/0x70 [xfs]
[224162.799383] [<ffffffffa00dd65f>] xfs_qm_dqdetach+0x3f/0x90 [xfs]
[224162.799405] [<ffffffffa0132a12>] xfs_inactive+0x222/0x470 [xfs]
[224162.799425] [<ffffffffa01407de>] xfs_fs_evict_inode+0x9e/0xf0 [xfs]
[224162.799429] [<ffffffff8117c707>] evict+0x27/0xc0
[224162.799432] [<ffffffff8117d34a>] iput_final+0x19a/0x2a0
[224162.799435] [<ffffffff8117d48b>] iput+0x3b/0x50
[224162.799439] [<ffffffff811728c0>] do_unlinkat+0x110/0x1d0
[224162.799443] [<ffffffff8108eead>] ? commit_creds+0xed/0x1c0
[224162.799447] [<ffffffff81174036>] sys_unlink+0x16/0x20
[224162.799450] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224282.790612] INFO: task kswapd0:37 blocked for more than 120 seconds.
[224282.803102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.817170] kswapd0 D 0000000000000000 0 37 2 0x00000000
[224282.817176] ffff88034428bc10 0000000000000046 ffff88034428bfd8 ffff88034428a000
[224282.817181] 0000000000013a80 ffff8803466c5ea0 ffff88034428bfd8 0000000000013a80
[224282.817186] ffff8803467fc440 ffff8803466c5b00 ffffffffffffff10 ffff880343d41a28
[224282.817191] Call Trace:
[224282.817201] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.817206] [<ffffffff815cfb6e>] ? _raw_spin_lock+0xe/0x20
[224282.817210] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.817247] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.817260] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.817337] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.817344] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.817347] [<ffffffff81120420>] balance_pgdat+0x2b0/0x530
[224282.817350] [<ffffffff811207df>] kswapd+0x13f/0x2b0
[224282.817355] [<ffffffff81087d30>] ? autoremove_wake_function+0x0/0x40
[224282.817359] [<ffffffff811206a0>] ? kswapd+0x0/0x2b0
[224282.817362] [<ffffffff81087606>] kthread+0x96/0xa0
[224282.817366] [<ffffffff8100cea4>] kernel_thread_helper+0x4/0x10
[224282.817370] [<ffffffff81087570>] ? kthread+0x0/0xa0
[224282.817373] [<ffffffff8100cea0>] ? kernel_thread_helper+0x0/0x10
[224282.817392] INFO: task mysqld:1458 blocked for more than 120 seconds.
[224282.829992] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.843909] mysqld D 0000000000000002 0 1458 1 0x00000000
[224282.843914] ffff88033f0dda10 0000000000000086 ffff88033f0ddfd8 ffff88033f0dc000
[224282.843919] 0000000000013a80 ffff88034362b120 ffff88033f0ddfd8 0000000000013a80
[224282.843924] ffff880343a18000 ffff88034362ad80 ffff88033f0dd9f0 ffff880343d41a28
[224282.843928] Call Trace:
[224282.843933] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.843937] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.843956] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.843969] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.843981] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.843986] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.843989] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.843994] [<ffffffff81039459>] ? default_spin_lock_flags+0x9/0x10
[224282.843997] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.844001] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.844006] [<ffffffff8114acdd>] alloc_pages_current+0x9d/0x110
[224282.844010] [<ffffffff81111f9e>] __get_free_pages+0xe/0x50
[224282.844014] [<ffffffff81063ec6>] dup_task_struct+0x46/0x190
[224282.844017] [<ffffffff8106442e>] copy_process+0xbe/0xe70
[224282.844021] [<ffffffff8104ecb3>] ? __wake_up+0x53/0x70
[224282.844025] [<ffffffff815cfb6e>] ? _raw_spin_lock+0xe/0x20
[224282.844028] [<ffffffff810652bd>] do_fork+0x8d/0x430
[224282.844032] [<ffffffff815cfb6e>] ? _raw_spin_lock+0xe/0x20
[224282.844037] [<ffffffff81174980>] ? setfl+0xf0/0x1b0
[224282.844042] [<ffffffff812b0fa0>] ? aa_revalidate_sk+0x80/0x90
[224282.844047] [<ffffffff810150c8>] sys_clone+0x28/0x30
[224282.844050] [<ffffffff8100c3a3>] stub_clone+0x13/0x20
[224282.844053] [<ffffffff8100c082>] ? system_call_fastpath+0x16/0x1b
[224282.844058] INFO: task mysqld:7461 blocked for more than 120 seconds.
[224282.856424] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.870358] mysqld D 0000000000000001 0 7461 1 0x00000000
[224282.870363] ffff88006bcc77c8 0000000000000086 ffff88006bcc7fd8 ffff88006bcc6000
[224282.870368] 0000000000013a80 ffff880067a41a60 ffff88006bcc7fd8 0000000000013a80
[224282.870373] ffff8803472f8000 ffff880067a416c0 ffff88006bcc77c8 ffff880343d41a28
[224282.870377] Call Trace:
[224282.870382] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.870386] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.870408] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.870421] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.870434] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.870438] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.870441] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.870445] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.870448] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.870453] [<ffffffff8115295b>] kmalloc_large_node+0x6b/0xc0
[224282.870457] [<ffffffff81158045>] __kmalloc_node_track_caller+0x155/0x1a0
[224282.870462] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[224282.870466] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[224282.870469] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[224282.870473] [<ffffffff8104ec1e>] ? __wake_up_sync_key+0x5e/0x80
[224282.870476] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[224282.870481] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[224282.870485] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[224282.870489] [<ffffffff811736d9>] ? do_filp_open+0x169/0x670
[224282.870494] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[224282.870498] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[224282.870503] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[224282.870506] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[224282.870509] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[224282.870512] [<ffffffff81165771>] sys_write+0x51/0x90
[224282.870516] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224282.870519] INFO: task mysqld:7470 blocked for more than 120 seconds.
[224282.883047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.897032] mysqld D 0000000000000002 0 7470 1 0x00000000
[224282.897039] ffff880078d1f7c8 0000000000000086 ffff880078d1ffd8 ffff880078d1e000
[224282.897047] 0000000000013a80 ffff880024e95ea0 ffff880078d1ffd8 0000000000013a80
[224282.897054] ffff8800955216c0 ffff880024e95b00 0000000000000000 ffff880343d41a28
[224282.897062] Call Trace:
[224282.897069] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.897074] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.897097] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.897111] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.897124] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.897130] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.897134] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.897139] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.897143] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.897149] [<ffffffff8115295b>] kmalloc_large_node+0x6b/0xc0
[224282.897154] [<ffffffff81158045>] __kmalloc_node_track_caller+0x155/0x1a0
[224282.897159] [<ffffffff81057535>] ? enqueue_entity+0x35/0x1c0
[224282.897164] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[224282.897169] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[224282.897174] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[224282.897178] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[224282.897184] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[224282.897188] [<ffffffff81013a55>] ? native_sched_clock+0x15/0x70
[224282.897194] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[224282.897198] [<ffffffff8105f8b0>] ? wake_up_state+0x10/0x20
[224282.897203] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[224282.897209] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[224282.897214] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[224282.897219] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[224282.897223] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[224282.897227] [<ffffffff81165771>] sys_write+0x51/0x90
[224282.897231] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224282.897236] INFO: task mysqld:28656 blocked for more than 120 seconds.
[224282.910015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.924279] mysqld D 0000000000000000 0 28656 1 0x00000000
[224282.924285] ffff880078d137c8 0000000000000086 ffff880078d13fd8 ffff880078d12000
[224282.924290] 0000000000013a80 ffff880004943120 ffff880078d13fd8 0000000000013a80
[224282.924294] ffff880343a18000 ffff880004942d80 ffff880078d137c8 ffff880343d41a28
[224282.924299] Call Trace:
[224282.924305] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.924308] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.924329] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.924342] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.924355] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.924359] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.924363] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.924366] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.924369] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.924374] [<ffffffff8115295b>] kmalloc_large_node+0x6b/0xc0
[224282.924377] [<ffffffff81158045>] __kmalloc_node_track_caller+0x155/0x1a0
[224282.924381] [<ffffffff814cabfb>] ? sock_wfree+0x5b/0x70
[224282.924384] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[224282.924388] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[224282.924391] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[224282.924395] [<ffffffff814cf0bc>] ? consume_skb+0x2c/0x80
[224282.924398] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[224282.924402] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[224282.924405] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[224282.924410] [<ffffffff811736d9>] ? do_filp_open+0x169/0x670
[224282.924414] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[224282.924418] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[224282.924422] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[224282.924425] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[224282.924429] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[224282.924432] [<ffffffff81165771>] sys_write+0x51/0x90
[224282.924436] [<ffffffff81056b2c>] ? do_sched_setscheduler+0x7c/0x80
[224282.924439] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224282.924443] INFO: task mysqld:7848 blocked for more than 120 seconds.
[224282.937308] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.951644] mysqld D 0000000000000003 0 7848 1 0x00000000
[224282.951651] ffff88000eb837c8 0000000000000086 ffff88000eb83fd8 ffff88000eb82000
[224282.951659] 0000000000013a80 ffff8803091803a0 ffff88000eb83fd8 0000000000013a80
[224282.951666] ffff88034713c440 ffff880309180000 0000000000000000 ffff880343d41a28
[224282.951674] Call Trace:
[224282.951680] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.951685] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.951706] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.951719] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.951733] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.951738] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.951743] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.951747] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.951752] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.951757] [<ffffffff8115295b>] kmalloc_large_node+0x6b/0xc0
[224282.951762] [<ffffffff81158045>] __kmalloc_node_track_caller+0x155/0x1a0
[224282.951766] [<ffffffff814cabfb>] ? sock_wfree+0x5b/0x70
[224282.951771] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[224282.951776] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[224282.951780] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[224282.951785] [<ffffffff8117c1a9>] ? touch_atime+0x69/0x190
[224282.951790] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[224282.951794] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[224282.951799] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[224282.951804] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[224282.951809] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[224282.951814] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[224282.951818] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[224282.951823] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[224282.951827] [<ffffffff81165771>] sys_write+0x51/0x90
[224282.951832] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
[224282.951836] INFO: task mysqld:7852 blocked for more than 120 seconds.
[224282.964878] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[224282.979407] mysqld D 0000000000000001 0 7852 1 0x00000000
[224282.979411] ffff88008f37b7c8 0000000000000086 ffff88008f37bfd8 ffff88008f37a000
[224282.979419] 0000000000013a80 ffff880037119a60 ffff88008f37bfd8 0000000000013a80
[224282.979426] ffff880343414440 ffff8800371196c0 0000000000000000 ffff880343d41a28
[224282.979434] Call Trace:
[224282.979440] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
[224282.979445] [<ffffffff815ce303>] mutex_lock+0x23/0x50
[224282.979464] [<ffffffffa00db709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
[224282.979477] [<ffffffffa00dbaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
[224282.979491] [<ffffffffa00dbb69>] xfs_qm_shake+0x59/0x70 [xfs]
[224282.979497] [<ffffffff8111d619>] shrink_slab+0x89/0x180
[224282.979501] [<ffffffff8111fca7>] do_try_to_free_pages+0x277/0x370
[224282.979505] [<ffffffff8111fff4>] try_to_free_pages+0x84/0x100
[224282.979510] [<ffffffff811157d4>] __alloc_pages_nodemask+0x404/0x6d0
[224282.979515] [<ffffffff8115295b>] kmalloc_large_node+0x6b/0xc0
[224282.979520] [<ffffffff81158045>] __kmalloc_node_track_caller+0x155/0x1a0
[224282.979524] [<ffffffff814ca7aa>] ? sock_alloc_send_pskb+0xea/0x250
[224282.979529] [<ffffffff814cf6d3>] __alloc_skb+0x83/0x170
[224282.979534] [<ffffffff814ca7aa>] sock_alloc_send_pskb+0xea/0x250
[224282.979538] [<ffffffff8104ec1e>] ? __wake_up_sync_key+0x5e/0x80
[224282.979543] [<ffffffff814ca925>] sock_alloc_send_skb+0x15/0x20
[224282.979548] [<ffffffff81566c1d>] unix_stream_sendmsg+0x32d/0x470
[224282.979552] [<ffffffff814c5f7e>] sock_aio_write+0x14e/0x160
[224282.979558] [<ffffffff8113aa9d>] ? page_add_new_anon_rmap+0x8d/0xa0
[224282.979563] [<ffffffff81164cc2>] do_sync_write+0xd2/0x110
[224282.979568] [<ffffffff812afcd8>] ? apparmor_file_permission+0x18/0x20
[224282.979573] [<ffffffff81279f0c>] ? security_file_permission+0x2c/0xb0
[224282.979577] [<ffffffff811650fd>] ? rw_verify_area+0x5d/0xe0
[224282.979582] [<ffffffff811654ed>] vfs_write+0x16d/0x190
[224282.979586] [<ffffffff81165771>] sys_write+0x51/0x90
[224282.979590] [<ffffffff8100c082>] system_call_fastpath+0x16/0x1b
--
Malcolm Scott
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: XFS deadlock in 2.6.37
2011-01-20 17:08 XFS deadlock in 2.6.37 Malcolm Scott
@ 2011-01-20 17:27 ` William Pitcock
2011-01-21 5:28 ` [PATCH] " Dave Chinner
1 sibling, 0 replies; 4+ messages in thread
From: William Pitcock @ 2011-01-20 17:27 UTC (permalink / raw)
To: Malcolm Scott; +Cc: linux-kernel
Hi,
On Thu, 20 Jan 2011 17:08:45 +0000 (GMT)
Malcolm Scott <lkml@malc.org.uk> wrote:
> Hi all,
>
> I've had the following deadlock happen twice on a 2.6.37 system with
> several XFS filesystems (including root) and no swap (may be
> relevant, considering that kswapd is one task involved here). Some
> minor filesystem corruption resulted (but maybe only because the root
> fs couldn't be synced/umounted).
>
kswapd also handles swapping pages in and out of the filesystem cache
as part of the VM subsystem. I have a hunch that if you do a bisect
of commits relating to the VM subsystem, you will find the commit that
is causing this.
Also, are you building with BKL enabled or disabled?
William
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH] Re: XFS deadlock in 2.6.37
2011-01-20 17:08 XFS deadlock in 2.6.37 Malcolm Scott
2011-01-20 17:27 ` William Pitcock
@ 2011-01-21 5:28 ` Dave Chinner
2011-01-25 17:13 ` Malcolm Scott
1 sibling, 1 reply; 4+ messages in thread
From: Dave Chinner @ 2011-01-21 5:28 UTC (permalink / raw)
To: Malcolm Scott; +Cc: linux-kernel, xfs
[cc xfs@os.sgi.com]
On Thu, Jan 20, 2011 at 05:08:45PM +0000, Malcolm Scott wrote:
> Hi all,
>
> I've had the following deadlock happen twice on a 2.6.37 system with
> several XFS filesystems (including root) and no swap (may be
> relevant, considering that kswapd is one task involved here). Some
> minor filesystem corruption resulted (but maybe only because the
> root fs couldn't be synced/umounted).
>
> If you need any more info, please let me know.
>
> --- first crash ---
>
> [504603.250208] INFO: task kswapd0:37 blocked for more than 120 seconds.
> [504603.261107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [504603.273465] kswapd0 D 0000000000000003 0 37 2 0x00000000
> [504603.273473] ffff88034428bc10 0000000000000046 ffff88034428bfd8 ffff88034428a000
> [504603.273479] 0000000000013a80 ffff8803442903a0 ffff88034428bfd8 0000000000013a80
> [504603.273483] ffff88034572ad80 ffff880344290000 ffffffffffffff10 ffff880343a51e28
> [504603.273488] Call Trace:
> [504603.273500] [<ffffffff815ce917>] __mutex_lock_slowpath+0xf7/0x180
> [504603.273504] [<ffffffff815ce303>] mutex_lock+0x23/0x50
> [504603.273541] [<ffffffffa00cf709>] xfs_qm_dqreclaim_one+0x29/0x350 [xfs]
> [504603.273554] [<ffffffffa00cfaed>] xfs_qm_shake_freelist+0x1d/0x40 [xfs]
> [504603.273567] [<ffffffffa00cfb69>] xfs_qm_shake+0x59/0x70 [xfs]
> [504603.273573] [<ffffffff8111d619>] shrink_slab+0x89/0x180
> [504603.273577] [<ffffffff81120420>] balance_pgdat+0x2b0/0x530
> [504603.273580] [<ffffffff811207df>] kswapd+0x13f/0x2b0
> [504603.273585] [<ffffffff81087d30>] ? autoremove_wake_function+0x0/0x40
> [504603.273588] [<ffffffff811206a0>] ? kswapd+0x0/0x2b0
> [504603.273591] [<ffffffff81087606>] kthread+0x96/0xa0
> [504603.273596] [<ffffffff8100cea4>] kernel_thread_helper+0x4/0x10
> [504603.273599] [<ffffffff81087570>] ? kthread+0x0/0xa0
> [504603.273603] [<ffffffff8100cea0>] ? kernel_thread_helper+0x0/0x10
[snip]
Looks like everything is hung up on the freelist lock. Can you
test the patch below?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
xfs: fix dquot shaker deadlock
From: Dave Chinner <dchinner@redhat.com>
Commit 368e136 ("xfs: remove duplicate code from dquot reclaim") fails
to unlock the dquot freelist when the number of loop restarts is
exceeded in xfs_qm_dqreclaim_one(). This causes hangs in memory
reclaim. Remove the bogus loop exit check that causes the problem.
Reported-by: Malcolm Scott <lkml@malc.org.uk>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
fs/xfs/quota/xfs_qm.c | 2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/fs/xfs/quota/xfs_qm.c b/fs/xfs/quota/xfs_qm.c
index f8e854b..9431c56 100644
--- a/fs/xfs/quota/xfs_qm.c
+++ b/fs/xfs/quota/xfs_qm.c
@@ -1992,8 +1992,6 @@ dqfunlock:
xfs_dqunlock(dqp);
if (dqpout)
break;
- if (restarts >= XFS_QM_RECLAIM_MAX_RESTARTS)
- return NULL;
}
mutex_unlock(&xfs_Gqm->qm_dqfrlist_lock);
return dqpout;
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] Re: XFS deadlock in 2.6.37
2011-01-21 5:28 ` [PATCH] " Dave Chinner
@ 2011-01-25 17:13 ` Malcolm Scott
0 siblings, 0 replies; 4+ messages in thread
From: Malcolm Scott @ 2011-01-25 17:13 UTC (permalink / raw)
To: Dave Chinner; +Cc: linux-kernel, xfs
On Fri, 21 Jan 2011, Dave Chinner wrote:
> Looks like everything is hung up on the freelist lock. Can you
> test the patch below?
So far I haven't seen that crash after applying your patch; thanks. (It
previously took several days to occur on a busy box, though, so not 100%
sure it's gone.)
However, there's now a new crash -- not sure how related this is but at
some of the stack traces involve XFS. Any ideas?
Unlike the last one, I could apparently recover the kernel with sysrq-i.
INFO: task nfsd:3923 blocked for more than 120 seconds.
[221162.675428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.691015] ffff88030a5d79f0 0000000000000046 ffff88030a5d7980 ffff880002b97ff8
[221162.706250] 0000000000013a00 ffff880339cfad80 ffff880339cfb118 ffff88030a5d7fd8
[221162.721488] ffff880339cfb120 0000000000013a00 ffff88030a5d6010 0000000000013a00
[221162.736739] Call Trace:
[221162.746904] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.760862] [<ffffffffa00bf23e>] ? exportfs_get_name+0x13e/0x160 [exportfs]
[221162.775580] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.788271] [<ffffffffa00bf452>] reconnect_path+0x1f2/0x300 [exportfs]
[221162.802483] [<ffffffffa00bf7aa>] exportfs_decode_fh+0xea/0x940 [exportfs]
[221162.816907] [<ffffffffa04907c0>] ? nfsd_acceptable+0x0/0x120 [nfsd]
[221162.830732] [<ffffffffa03ea06f>] ? cache_check+0x6f/0x210 [sunrpc]
[221162.844380] [<ffffffffa0496842>] ? exp_find_key+0x62/0xb0 [nfsd]
[221162.857770] [<ffffffff8115399c>] ? kmem_cache_alloc_notrace+0x9c/0xb0
[221162.871504] [<ffffffff812a741d>] ? aa_dup_task_context+0x3d/0x70
[221162.871509] [<ffffffff812acf60>] ? apparmor_cred_prepare+0x40/0x60
[221162.871519] [<ffffffffa0490ba4>] nfsd_set_fh_dentry+0x214/0x3d0 [nfsd]
[221162.871523] [<ffffffff81112505>] ? __alloc_pages_slowpath+0x1c5/0x5c0
[221162.871530] [<ffffffffa0490f25>] fh_verify+0x1c5/0x280 [nfsd]
[221162.871537] [<ffffffffa04917a8>] nfsd_access+0x38/0x160 [nfsd]
[221162.871546] [<ffffffffa049bf87>] nfsd3_proc_access+0x77/0xd0 [nfsd]
[221162.871556] [<ffffffffa048d43e>] nfsd_dispatch+0xfe/0x250 [nfsd]
[221162.871571] [<ffffffffa03df6c4>] svc_process_common+0x344/0x630 [sunrpc]
[221162.871576] [<ffffffff8105c5f0>] ? default_wake_function+0x0/0x20
[221162.871582] [<ffffffffa048dae0>] ? nfsd+0x0/0x170 [nfsd]
[221162.871593] [<ffffffffa03dfac0>] svc_process+0x110/0x150 [sunrpc]
[221162.871600] [<ffffffffa048dba5>] nfsd+0xc5/0x170 [nfsd]
[221162.871604] [<ffffffff81085a56>] kthread+0x96/0xa0
[221162.871608] [<ffffffff8100cf24>] kernel_thread_helper+0x4/0x10
[221162.871611] [<ffffffff810859c0>] ? kthread+0x0/0xa0
[221162.871614] [<ffffffff8100cf20>] ? kernel_thread_helper+0x0/0x10
[221162.871672] INFO: task php5-cgi:32524 blocked for more than 120 seconds.
[221162.871674] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.871680] ffff880057f4fc88 0000000000000086 ffff880057f4fbe8 ffffffff815c9160
[221162.871684] 0000000000013a00 ffff88002bc22d80 ffff88002bc23118 ffff880057f4ffd8
[221162.871687] ffff88002bc23120 0000000000013a00 ffff880057f4e010 0000000000013a00
[221162.871691] Call Trace:
[221162.871695] [<ffffffff815c9160>] ? io_schedule+0x90/0xc0
[221162.871700] [<ffffffff8110b5a3>] ? wait_on_page_bit+0x73/0x80
[221162.871703] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.871707] [<ffffffff8108e281>] ? in_group_p+0x31/0x40
[221162.871710] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.871713] [<ffffffff8116f802>] do_lookup+0x102/0x180
[221162.871716] [<ffffffff8116fb1d>] do_last+0x29d/0x420
[221162.871719] [<ffffffff81171a13>] do_filp_open+0x203/0x5f0
[221162.871723] [<ffffffff8112fc39>] ? handle_mm_fault+0x1b9/0x420
[221162.871728] [<ffffffff8117d1fa>] ? alloc_fd+0x10a/0x150
[221162.871732] [<ffffffff81160c79>] do_sys_open+0x69/0x150
[221162.871735] [<ffffffff81160da0>] sys_open+0x20/0x30
[221162.871739] [<ffffffff8100c102>] system_call_fastpath+0x16/0x1b
[221162.871743] INFO: task sshd:32537 blocked for more than 120 seconds.
[221162.871745] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.871750] ffff88007c09bc88 0000000000000082 ffff88007c09bbe8 ffffffff815c9160
[221162.871754] 0000000000013a00 ffff88027d098000 ffff88027d098398 ffff88007c09bfd8
[221162.871757] ffff88027d0983a0 0000000000013a00 ffff88007c09a010 0000000000013a00
[221162.871761] Call Trace:
[221162.871763] [<ffffffff815c9160>] ? io_schedule+0x90/0xc0
[221162.871767] [<ffffffff8110b5a3>] ? wait_on_page_bit+0x73/0x80
[221162.871770] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.871773] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.871775] [<ffffffff8116f802>] do_lookup+0x102/0x180
[221162.871778] [<ffffffff8116fb1d>] do_last+0x29d/0x420
[221162.871781] [<ffffffff81171a13>] do_filp_open+0x203/0x5f0
[221162.871784] [<ffffffff8112fc39>] ? handle_mm_fault+0x1b9/0x420
[221162.871787] [<ffffffff8117d1fa>] ? alloc_fd+0x10a/0x150
[221162.871790] [<ffffffff81160c79>] do_sys_open+0x69/0x150
[221162.871793] [<ffffffff81160da0>] sys_open+0x20/0x30
[221162.871796] [<ffffffff8100c102>] system_call_fastpath+0x16/0x1b
[221162.871799] INFO: task sshd:32541 blocked for more than 120 seconds.
[221162.871801] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.871806] ffff880056c7fc88 0000000000000082 ffff880056c7fbe8 ffffffff00000000
[221162.871809] 0000000000013a00 ffff88027d09ad80 ffff88027d09b118 ffff880056c7ffd8
[221162.871813] ffff88027d09b120 0000000000013a00 ffff880056c7e010 0000000000013a00
[221162.871816] Call Trace:
[221162.871819] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.871822] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.871825] [<ffffffff8116f802>] do_lookup+0x102/0x180
[221162.871828] [<ffffffff8116fb1d>] do_last+0x29d/0x420
[221162.871830] [<ffffffff81171a13>] do_filp_open+0x203/0x5f0
[221162.871833] [<ffffffff8112fc39>] ? handle_mm_fault+0x1b9/0x420
[221162.871836] [<ffffffff8117d1fa>] ? alloc_fd+0x10a/0x150
[221162.871839] [<ffffffff81160c79>] do_sys_open+0x69/0x150
[221162.871842] [<ffffffff81160da0>] sys_open+0x20/0x30
[221162.871845] [<ffffffff8100c102>] system_call_fastpath+0x16/0x1b
[221162.871849] INFO: task sshd:32547 blocked for more than 120 seconds.
[221162.871850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.871855] ffff88009935bc88 0000000000000086 ffff88009935bbe8 ffffffff00000000
[221162.871859] 0000000000013a00 ffff8802279596c0 ffff880227959a58 ffff88009935bfd8
[221162.871862] ffff880227959a60 0000000000013a00 ffff88009935a010 0000000000013a00
[221162.871865] Call Trace:
[221162.871868] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.871871] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.871874] [<ffffffff8116f802>] do_lookup+0x102/0x180
[221162.871876] [<ffffffff8116fb1d>] do_last+0x29d/0x420
[221162.871879] [<ffffffff81171a13>] do_filp_open+0x203/0x5f0
[221162.871882] [<ffffffff8112fc39>] ? handle_mm_fault+0x1b9/0x420
[221162.871885] [<ffffffff8117d1fa>] ? alloc_fd+0x10a/0x150
[221162.871888] [<ffffffff81160c79>] do_sys_open+0x69/0x150
[221162.871891] [<ffffffff81160da0>] sys_open+0x20/0x30
[221162.871894] [<ffffffff8100c102>] system_call_fastpath+0x16/0x1b
[221162.871896] INFO: task sshd:32548 blocked for more than 120 seconds.
[221162.871898] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221162.871903] ffff88006cabda98 0000000000000082 ffff88006cabda38 ffffffff811787d6
[221162.871906] 0000000000013a00 ffff880066494440 ffff8800664947d8 ffff88006cabdfd8
[221162.871910] ffff8800664947e0 0000000000013a00 ffff88006cabc010 0000000000013a00
[221162.871913] Call Trace:
[221162.871916] [<ffffffff811787d6>] ? __d_lookup+0x136/0x150
[221162.871919] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221162.871922] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221162.871925] [<ffffffff8116f802>] do_lookup+0x102/0x180
[221162.871927] [<ffffffff8116fb1d>] do_last+0x29d/0x420
[221162.871930] [<ffffffff81171a13>] do_filp_open+0x203/0x5f0
[221162.871933] [<ffffffff811627fa>] ? do_sync_read+0xda/0x120
[221162.871936] [<ffffffff811174ed>] ? lru_cache_add_lru+0x2d/0x50
[221162.871941] [<ffffffff8116ab1d>] open_exec+0x3d/0x110
[221162.871945] [<ffffffff811af99d>] load_elf_binary+0xdcd/0xeb0
[221162.871948] [<ffffffff811303c9>] ? get_user_pages+0x49/0x50
[221162.871951] [<ffffffff8116979c>] ? get_arg_page+0x5c/0x100
[221162.871954] [<ffffffff8116a042>] search_binary_handler+0xe2/0x300
[221162.871957] [<ffffffff8116ae27>] do_execve+0x237/0x340
[221162.871961] [<ffffffff8101474a>] sys_execve+0x4a/0x80
[221162.871964] [<ffffffff8100c55c>] stub_execve+0x6c/0xc0
[221282.870468] INFO: task nfsd:3923 blocked for more than 120 seconds.
[221282.882094] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221282.895378] ffff88030a5d79f0 0000000000000046 ffff88030a5d7980 ffff880002b97ff8
[221282.908408] 0000000000013a00 ffff880339cfad80 ffff880339cfb118 ffff88030a5d7fd8
[221282.921529] ffff880339cfb120 0000000000013a00 ffff88030a5d6010 0000000000013a00
[221282.921533] Call Trace:
[221282.921544] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221282.921558] [<ffffffffa00bf23e>] ? exportfs_get_name+0x13e/0x160 [exportfs]
[221282.921561] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221282.921565] [<ffffffffa00bf452>] reconnect_path+0x1f2/0x300 [exportfs]
[221282.921569] [<ffffffffa00bf7aa>] exportfs_decode_fh+0xea/0x940 [exportfs]
[221282.921580] [<ffffffffa04907c0>] ? nfsd_acceptable+0x0/0x120 [nfsd]
[221282.921600] [<ffffffffa03ea06f>] ? cache_check+0x6f/0x210 [sunrpc]
[221282.921608] [<ffffffffa0496842>] ? exp_find_key+0x62/0xb0 [nfsd]
[221282.921613] [<ffffffff8115399c>] ? kmem_cache_alloc_notrace+0x9c/0xb0
[221282.921618] [<ffffffff812a741d>] ? aa_dup_task_context+0x3d/0x70
[221282.921622] [<ffffffff812acf60>] ? apparmor_cred_prepare+0x40/0x60
[221282.921629] [<ffffffffa0490ba4>] nfsd_set_fh_dentry+0x214/0x3d0 [nfsd]
[221282.921633] [<ffffffff81112505>] ? __alloc_pages_slowpath+0x1c5/0x5c0
[221282.921640] [<ffffffffa0490f25>] fh_verify+0x1c5/0x280 [nfsd]
[221282.921647] [<ffffffffa04917a8>] nfsd_access+0x38/0x160 [nfsd]
[221282.921655] [<ffffffffa049bf87>] nfsd3_proc_access+0x77/0xd0 [nfsd]
[221282.921662] [<ffffffffa048d43e>] nfsd_dispatch+0xfe/0x250 [nfsd]
[221282.921674] [<ffffffffa03df6c4>] svc_process_common+0x344/0x630 [sunrpc]
[221282.921679] [<ffffffff8105c5f0>] ? default_wake_function+0x0/0x20
[221282.921685] [<ffffffffa048dae0>] ? nfsd+0x0/0x170 [nfsd]
[221282.921696] [<ffffffffa03dfac0>] svc_process+0x110/0x150 [sunrpc]
[221282.921702] [<ffffffffa048dba5>] nfsd+0xc5/0x170 [nfsd]
[221282.921707] [<ffffffff81085a56>] kthread+0x96/0xa0
[221282.921711] [<ffffffff8100cf24>] kernel_thread_helper+0x4/0x10
[221282.921714] [<ffffffff810859c0>] ? kthread+0x0/0xa0
[221282.921717] [<ffffffff8100cf20>] ? kernel_thread_helper+0x0/0x10
[221282.921755] INFO: task kworker/0:2:21740 blocked for more than 120 seconds.
[221282.921756] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221282.921762] ffff880345031ac0 0000000000000046 ffff880345031a10 ffffffff812db486
[221282.921765] 0000000000013a00 ffff88004c5bc440 ffff88004c5bc7d8 ffff880345031fd8
[221282.921768] ffff88004c5bc7e0 0000000000013a00 ffff880345030010 0000000000013a00
[221282.921772] Call Trace:
[221282.921776] [<ffffffff812db486>] ? rb_erase+0xd6/0x160
[221282.921782] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221282.921787] [<ffffffff81051ed2>] ? enqueue_entity+0x132/0x1b0
[221282.921789] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221282.921825] [<ffffffffa01400ca>] xfs_file_aio_write+0x4fa/0xac0 [xfs]
[221282.921829] [<ffffffff8104ace0>] ? tg_nop+0x0/0x10
[221282.921832] [<ffffffff81051cc8>] ? enqueue_sleeper+0x158/0x230
[221282.921836] [<ffffffff811626da>] do_sync_write+0xda/0x120
[221282.921839] [<ffffffff8105c2a3>] ? try_to_wake_up+0xc3/0x410
[221282.921844] [<ffffffff81090749>] ? ktime_get_ts+0xa9/0xe0
[221282.921848] [<ffffffff810aab5b>] do_acct_process+0x3fb/0x500
[221282.921851] [<ffffffff810aacde>] acct_process+0x7e/0xb0
[221282.921854] [<ffffffff81068fd5>] do_exit+0x2a5/0x400
[221282.921857] [<ffffffff81081400>] ? worker_thread+0x0/0x410
[221282.921860] [<ffffffff81085a4e>] kthread+0x8e/0xa0
[221282.921863] [<ffffffff8100cf24>] kernel_thread_helper+0x4/0x10
[221282.921866] [<ffffffff810859c0>] ? kthread+0x0/0xa0
[221282.921868] [<ffffffff8100cf20>] ? kernel_thread_helper+0x0/0x10
[221282.921872] INFO: task kworker/3:0:16949 blocked for more than 120 seconds.
[221282.921873] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221282.921879] ffff88021acd9ac0 0000000000000046 0000000000000001 ffff88009e0d3a00
[221282.921882] 0000000000013a00 ffff8800020696c0 ffff880002069a58 ffff88021acd9fd8
[221282.921885] ffff880002069a60 0000000000013a00 ffff88021acd8010 0000000000013a00
[221282.921889] Call Trace:
[221282.921892] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221282.921895] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221282.921915] [<ffffffffa01400ca>] xfs_file_aio_write+0x4fa/0xac0 [xfs]
[221282.921919] [<ffffffff81038649>] ? default_spin_lock_flags+0x9/0x10
[221282.921923] [<ffffffff81013459>] ? sched_clock+0x9/0x10
[221282.921926] [<ffffffff812db11d>] ? rb_insert_color+0x9d/0x160
[221282.921929] [<ffffffff811626da>] do_sync_write+0xda/0x120
[221282.921932] [<ffffffff8105c2a3>] ? try_to_wake_up+0xc3/0x410
[221282.921935] [<ffffffff81090749>] ? ktime_get_ts+0xa9/0xe0
[221282.921938] [<ffffffff810aab5b>] do_acct_process+0x3fb/0x500
[221282.921941] [<ffffffff810aacde>] acct_process+0x7e/0xb0
[221282.921944] [<ffffffff81068fd5>] do_exit+0x2a5/0x400
[221282.921947] [<ffffffff81081400>] ? worker_thread+0x0/0x410
[221282.921950] [<ffffffff81085a4e>] kthread+0x8e/0xa0
[221282.921952] [<ffffffff8100cf24>] kernel_thread_helper+0x4/0x10
[221282.921955] [<ffffffff810859c0>] ? kthread+0x0/0xa0
[221282.921958] [<ffffffff8100cf20>] ? kernel_thread_helper+0x0/0x10
[221282.921961] INFO: task kworker/1:2:29640 blocked for more than 120 seconds.
[221282.921962] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[221282.921968] ffff880144665ac0 0000000000000046 ffff880144665a10 ffffffff812db486
[221282.921971] 0000000000013a00 ffff8800025b96c0 ffff8800025b9a58 ffff880144665fd8
[221282.921974] ffff8800025b9a60 0000000000013a00 ffff880144664010 0000000000013a00
[221282.921978] Call Trace:
[221282.921980] [<ffffffff812db486>] ? rb_erase+0xd6/0x160
[221282.921984] [<ffffffff815c9e87>] __mutex_lock_slowpath+0xf7/0x180
[221282.921987] [<ffffffff815c9893>] mutex_lock+0x23/0x50
[221282.922007] [<ffffffffa01400ca>] xfs_file_aio_write+0x4fa/0xac0 [xfs]
[221282.922010] [<ffffffff812dcaf9>] ? put_dec+0x59/0x60
[221282.922030] [<ffffffffa012c01b>] ? xfs_icsb_count+0x8b/0xc0 [xfs]
[221282.922034] [<ffffffff81038649>] ? default_spin_lock_flags+0x9/0x10
[221282.922037] [<ffffffff815caf4f>] ? _raw_spin_lock_irqsave+0x2f/0x40
[221282.922040] [<ffffffff811626da>] do_sync_write+0xda/0x120
[221282.922044] [<ffffffff81090749>] ? ktime_get_ts+0xa9/0xe0
[221282.922047] [<ffffffff810aab5b>] do_acct_process+0x3fb/0x500
[221282.922050] [<ffffffff810aacde>] acct_process+0x7e/0xb0
[221282.922053] [<ffffffff81068fd5>] do_exit+0x2a5/0x400
[221282.922056] [<ffffffff81081400>] ? worker_thread+0x0/0x410
[221282.922059] [<ffffffff81085a4e>] kthread+0x8e/0xa0
[221282.922062] [<ffffffff8100cf24>] kernel_thread_helper+0x4/0x10
[221282.922065] [<ffffffff810859c0>] ? kthread+0x0/0xa0
[221282.922067] [<ffffffff8100cf20>] ? kernel_thread_helper+0x0/0x10
--
Malcolm Scott
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-01-25 17:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-20 17:08 XFS deadlock in 2.6.37 Malcolm Scott
2011-01-20 17:27 ` William Pitcock
2011-01-21 5:28 ` [PATCH] " Dave Chinner
2011-01-25 17:13 ` Malcolm Scott
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).