xfs deadlocks since upgrading to 4.1.15

xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe]

Hello,

since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle 
call traces show xfs.

Attention all those are down top instead of top down (sorry for the 
wrong order).

NR 1:

2016-01-14 00:17:23     [<ffffffff93501ca5>] rtnl_lock+0x15/0x20
2016-01-14 00:17:23     [<ffffffff93501cf6>] rtnetlink_rcv+0x16/0x40
2016-01-14 00:17:23     [<ffffffff935252aa>] netlink_unicast+0x13a/0x1d0
2016-01-14 00:17:23     [<ffffffff9352554a>] netlink_sendmsg+0x20a/0x600
2016-01-14 00:17:23     [<ffffffff934d7d61>] SYSC_sendto+0x111/0x170
2016-01-14 00:17:23     [<ffffffff9314e263>] ? 
context_tracking_user_exit+0x13/0x20
2016-01-14 00:17:23     [<ffffffff930134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:23     [<ffffffff934d82ae>] SyS_sendto+0xe/0x10
2016-01-14 00:17:23     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:22     [<ffffffff932e449f>] xlog_cil_push+0x22f/0x3f0
2016-01-14 00:17:22     [<ffffffff932e4675>] xlog_cil_push_work+0x15/0x20
2016-01-14 00:17:22     [<ffffffff930986bf>] process_one_work+0x14f/0x410
2016-01-14 00:17:22     [<ffffffff930989ac>] 
process_scheduled_works+0x2c/0x40
2016-01-14 00:17:22     [<ffffffff93098c4a>] worker_thread+0x28a/0x510
2016-01-14 00:17:22     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:22     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:22     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:22     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:22     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     INFO: task qpsmtpd p:25 :::12850 blocked for 
more than 120 seconds.
2016-01-14 00:17:22     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:22     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:22     qpsmtpd p:25 :: D ffff88071f807c48 0 12850 6124 
0x00000000
2016-01-14 00:17:22     ffff88071f807c48 ffff8805aabf0000 
ffff8806c5780000 ffff88083ffd7000
2016-01-14 00:17:22     ffff88071f808000 ffffffff93ce6444 
ffff8806c5780000 00000000ffffffff
2016-01-14 00:17:22     ffffffff93ce6448 ffff88071f807c68 
ffffffff9363c167 ffff88042b9f0000
2016-01-14 00:17:22     Call Trace:
2016-01-14 00:17:22     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:22     [<ffffffff9363c3de>] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:22     [<ffffffff9363dfd5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:22     [<ffffffff9363e06b>] mutex_lock+0x1b/0x2f
2016-01-14 00:17:21     [<ffffffff931a7617>] __kmalloc+0x1c7/0x240
2016-01-14 00:17:21     [<ffffffff931e0224>] alloc_fdmem+0x24/0x40
2016-01-14 00:17:21     [<ffffffff931e02b7>] alloc_fdtable+0x77/0x100
2016-01-14 00:17:21     [<ffffffff931e0af2>] dup_fd+0x262/0x330
2016-01-14 00:17:21     [<ffffffff9307ec9a>] 
copy_process.part.37+0x76a/0x1a80
2016-01-14 00:17:21     [<ffffffff931190ec>] ? 
acct_account_cputime+0x1c/0x20
2016-01-14 00:17:21     [<ffffffff9308014c>] do_fork+0xbc/0x350
2016-01-14 00:17:21     [<ffffffff930134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:21     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:21     [<ffffffff93080466>] SyS_clone+0x16/0x20
2016-01-14 00:17:21     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:21     INFO: task kworker/7:0:23773 blocked for more 
than 120 seconds.
2016-01-14 00:17:21     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:21     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:21     kworker/7:0 D ffff8805243bbb38 0 23773 2 0x00000000
2016-01-14 00:17:21     Workqueue: xfs-cil/sdb1 xlog_cil_push_work
2016-01-14 00:17:21     ffff8805243bbb38 ffff88082d941d50 
ffff880790560000 0000000000000000
2016-01-14 00:17:21     ffff8805243bc000 ffff88010af7c0b8 
ffff88042c067d28 ffff880790560000
2016-01-14 00:17:21     ffff88042c067d00 ffff8805243bbb58 
ffffffff9363c167 ffff88042c067c00
2016-01-14 00:17:21     Call Trace:
2016-01-14 00:17:21     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:21     [<ffffffff932e241b>] 
xlog_state_get_iclog_space+0xdb/0x2e0
2016-01-14 00:17:21     [<ffffffff932dfe14>] ? xlog_bdstrat+0x34/0x70
2016-01-14 00:17:21     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:21     [<ffffffff932e291e>] xlog_write+0x18e/0x700
2016-01-14 00:17:20     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:20     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:20     [<ffffffffc03efb24>] ? aac_fib_send+0x2c4/0x4f0 
[aacraid]
2016-01-14 00:17:20     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:20     [<ffffffff932b4b87>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:20     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:20     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:20     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:20     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:20     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:20     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:20     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:20     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:20     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:20     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:20     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:20     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:20     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:20     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:20     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:20     [<ffffffff9316d468>] ? zone_statistics+0x98/0xc0
2016-01-14 00:17:20     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:20     [<ffffffff931e0224>] ? alloc_fdmem+0x24/0x40
2016-01-14 00:17:19     [<ffffffff93355fa9>] ida_pre_get+0x69/0x100
2016-01-14 00:17:19     [<ffffffff93356d9e>] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:19     [<ffffffff930a7c9e>] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:19     [<ffffffff93097adb>] create_worker+0x4b/0x1f0
2016-01-14 00:17:19     [<ffffffff930e1e0a>] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:19     [<ffffffff93098daf>] worker_thread+0x3ef/0x510
2016-01-14 00:17:19     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:19     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:19     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:19     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:19     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     INFO: task httpd:22880 blocked for more than 120 
seconds.
2016-01-14 00:17:19     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:19     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:19     httpd D ffff88005557b108 0 22880 22644 0x00000000
2016-01-14 00:17:19     ffff88005557b108 ffff88042dbdbaa0 
ffff8807e65a57f0 ffff88005557b1b8
2016-01-14 00:17:19     ffff88005557c000 ffff88005557b270 
7fffffffffffffff ffff8807e65a57f0
2016-01-14 00:17:19     ffffffffffffffff ffff88005557b128 
ffffffff9363c167 0000000000000292
2016-01-14 00:17:19     Call Trace:
2016-01-14 00:17:19     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:19     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:19     [<ffffffff933506b5>] ? cfq_set_request+0xa5/0x4c0
2016-01-14 00:17:19     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:19     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:19     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:18     [<ffffffff9334c148>] ? 
__cfq_set_active_queue+0xf8/0x230
2016-01-14 00:17:18     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:18     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:18     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:18     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:18     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:18     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:18     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:18     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:18     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:18     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:18     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:18     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:18     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:18     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:18     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:18     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:18     [<ffffffff930ad1f3>] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:18     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:18     [<ffffffff93355fa9>] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:18     [<ffffffff930c2f43>] ? __wake_up+0x53/0x70
2016-01-14 00:17:18     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:18     [<ffffffff930b42f0>] ? pick_next_entity+0x90/0x170
2016-01-14 00:17:17     [<ffffffff93098daf>] worker_thread+0x3ef/0x510
2016-01-14 00:17:17     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:17     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:17     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:17     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:17     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     INFO: task kworker/7:0H:7845 blocked for more 
than 120 seconds.
2016-01-14 00:17:17     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:17     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:17     kworker/7:0H D ffff8806ec7ff108 0 7845 2 0x00000000
2016-01-14 00:17:17     ffff8806ec7ff108 ffff88044840d7f0 
ffff88082762baa0 ffff88044840d7f0
2016-01-14 00:17:17     ffff8806ec800000 ffff8806ec7ff270 
7fffffffffffffff ffff88082762baa0
2016-01-14 00:17:17     ffffffffffffffff ffff8806ec7ff128 
ffffffff9363c167 ffff8806ec7ff148
2016-01-14 00:17:17     Call Trace:
2016-01-14 00:17:17     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:17     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:17     [<ffffffff930a98b6>] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-14 00:17:17     [<ffffffff930ad1f3>] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:17     [<ffffffff9335090f>] ? cfq_set_request+0x2ff/0x4c0
2016-01-14 00:17:17     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:17     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:17     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:17     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:17     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:16     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:16     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:16     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:16     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:16     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:16     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:16     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:16     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:16     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:16     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:16     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:16     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:16     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:16     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:16     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:16     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:16     [<ffffffff93355fa9>] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:16     [<ffffffff9300d775>] ? native_sched_clock+0x35/0x80
2016-01-14 00:17:16     [<ffffffff93355fa9>] ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [<ffffffff93356d9e>] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:16     [<ffffffff930a7c9e>] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:16     [<ffffffff93097adb>] create_worker+0x4b/0x1f0
2016-01-14 00:17:16     [<ffffffff930e1e0a>] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:15     [<ffffffff931f4c58>] do_fsync+0x38/0x70
2016-01-14 00:17:15     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:15     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:15     [<ffffffff931f4ef3>] SyS_fdatasync+0x13/0x20
2016-01-14 00:17:15     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:15     INFO: task kworker/7:2:22195 blocked for more 
than 120 seconds.
2016-01-14 00:17:15     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:15     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:15     kworker/7:2 D ffff88009ac93108 0 22195 2 0x00000000
2016-01-14 00:17:15     ffff88009ac93108 ffff88082d941d50 
ffff88044840d7f0 0000000950d64c61
2016-01-14 00:17:15     ffff88009ac94000 ffff88009ac93270 
7fffffffffffffff ffff88044840d7f0
2016-01-14 00:17:15     ffffffffffffffff ffff88009ac93128 
ffffffff9363c167 74612e656e6f6568
2016-01-14 00:17:15     Call Trace:
2016-01-14 00:17:15     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:15     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:15     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:15     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:15     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:15     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:15     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:15     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:15     [<ffffffff932b4b87>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:15     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:15     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:14     [<ffffffff9354d671>] ? tcp_write_xmit+0x2b1/0xda0
2016-01-14 00:17:14     [<ffffffff934da9a1>] 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:14     [<ffffffff934db9b3>] sk_alloc+0x33/0xb0
2016-01-14 00:17:14     [<ffffffff93590231>] unix_create1+0x51/0x1a0
2016-01-14 00:17:14     [<ffffffff935903f3>] unix_create+0x73/0x90
2016-01-14 00:17:14     [<ffffffff934d6710>] __sock_create+0xf0/0x1f0
2016-01-14 00:17:14     [<ffffffff930d95f8>] ? rcu_eqs_enter+0x68/0x90
2016-01-14 00:17:14     [<ffffffff934d6870>] sock_create+0x30/0x40
2016-01-14 00:17:14     [<ffffffff934d803c>] SyS_socketpair+0x6c/0x1d0
2016-01-14 00:17:14     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:14     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:14     INFO: task imap:26593 blocked for more than 120 
seconds.
2016-01-14 00:17:14     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:14     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:14     imap D ffff880675a5fdc8 0 26593 4498 0x00000000
2016-01-14 00:17:14     ffff880675a5fdc8 ffff88082d943aa0 
ffff8807abab0000 ffff8806b0702f00
2016-01-14 00:17:14     ffff880675a60000 ffff88042b1f9928 
ffff88042b1f9800 ffff88042bba4c18
2016-01-14 00:17:14     ffff8807abab0000 ffff880675a5fde8 
ffffffff9363c167 0000fddf00010800
2016-01-14 00:17:14     Call Trace:
2016-01-14 00:17:14     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:14     [<ffffffff932e3433>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-14 00:17:14     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:14     [<ffffffff932c5d96>] xfs_dir_fsync+0x86/0xf0
2016-01-14 00:17:14     [<ffffffff931f4a56>] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:14     [<ffffffff931f4aec>] vfs_fsync+0x1c/0x20
2016-01-14 00:17:13     [<ffffffff9363dfd5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:13     [<ffffffff9363e06b>] mutex_lock+0x1b/0x2f
2016-01-14 00:17:13     [<ffffffff932cab77>] 
xfs_reclaim_inodes_ag+0x307/0x3d0
2016-01-14 00:17:13     [<ffffffff931d8cf0>] ? spin_unlock+0x10/0x10
2016-01-14 00:17:13     [<ffffffff930d9d1f>] ? __call_rcu+0x3f/0x360
2016-01-14 00:17:13     [<ffffffff931df3c9>] ? inode_lru_isolate+0x89/0x1c0
2016-01-14 00:17:13     [<ffffffff93177e48>] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-14 00:17:13     [<ffffffff931df340>] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [<ffffffff931df340>] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:13     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:13     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:13     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:13     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:13     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:13     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:13     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:13     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:13     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:13     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:13     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:13     [<ffffffff934da9a1>] ? 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:13     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:12     [<ffffffff932e354b>] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-14 00:17:12     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:12     [<ffffffff932c626f>] xfs_file_fsync+0x18f/0x1e0
2016-01-14 00:17:12     [<ffffffff931f4a56>] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:12     [<ffffffff931f4aec>] vfs_fsync+0x1c/0x20
2016-01-14 00:17:12     [<ffffffff931f4c58>] do_fsync+0x38/0x70
2016-01-14 00:17:12     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:12     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:12     [<ffffffff931f4ed0>] SyS_fsync+0x10/0x20
2016-01-14 00:17:12     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:12     INFO: task pop3-login:7921 blocked for more than 
120 seconds.
2016-01-14 00:17:12     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:12     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:12     pop3-login D ffff88082c5234f8 0 7921 4498 0x00000000
2016-01-14 00:17:12     ffff88082c5234f8 ffff88042dbdbaa0 
ffff880827443aa0 ffff88082c523568
2016-01-14 00:17:12     ffff88082c524000 ffff88082bb63b6c 
ffff880827443aa0 00000000ffffffff
2016-01-14 00:17:12     ffff88082bb63b70 ffff88082c523518 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:12     [<ffffffff9363c3de>] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:11     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:11     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:11     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:11     [<ffffffff930e14bf>] ? 
try_to_del_timer_sync+0x4f/0x70
2016-01-14 00:17:11     [<ffffffff932e2feb>] _xfs_log_force+0x6b/0x280
2016-01-14 00:17:11     [<ffffffff932c37e0>] ? 
xfs_buf_delwri_submit_nowait+0x20/0x30
2016-01-14 00:17:11     [<ffffffff932e322a>] xfs_log_force+0x2a/0x90
2016-01-14 00:17:11     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [<ffffffff932edca3>] xfsaild+0x143/0x5f0
2016-01-14 00:17:11     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:11     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:11     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:11     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     INFO: task mysqld:7538 blocked for more than 120 
seconds.
2016-01-14 00:17:11     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:11     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:11     mysqld D ffff8803e708bd98 0 7538 3771 0x00000000
2016-01-14 00:17:11     ffff8803e708bd98 ffff88082d941d50 
ffff8803f1ebd7f0 ffff88042bba4c00
2016-01-14 00:17:11     ffff8803e708c000 ffff88042b1f9928 
ffff88042b1f9800 0000000000000000
2016-01-14 00:17:11     ffff8803f1ebd7f0 ffff8803e708bdb8 
ffffffff9363c167 ffff8803f1ebd7f0
2016-01-14 00:17:10     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:10     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:10     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:10     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:10     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     INFO: task xfsaild/sdb1:2068 blocked for more 
than 120 seconds.
2016-01-14 00:17:10     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:10     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:10     xfsaild/sdb1 D ffff88042529fad8 0 2068 2 0x00000000
2016-01-14 00:17:10     ffff88042529fad8 ffff88082d948000 
ffff8800b94ad7f0 ffff88082a8401d0
2016-01-14 00:17:10     ffff8804252a0000 ffff88042529fc40 
7fffffffffffffff ffff8800b94ad7f0
2016-01-14 00:17:10     ffffffffffffffff ffff88042529faf8 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:10     Call Trace:
2016-01-14 00:17:10     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:10     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:10     [<ffffffff93326f0c>] ? part_round_stats+0x5c/0x70
2016-01-14 00:17:10     [<ffffffff93239439>] ? kernfs_path_locked+0x49/0xa0
2016-01-14 00:17:10     [<ffffffff93324a22>] ? elv_rb_add+0x62/0x70
2016-01-14 00:17:10     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:10     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     INFO: task xfsaild/sda3:598 blocked for more 
than 120 seconds.
2016-01-14 00:17:09     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:09     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:09     xfsaild/sda3 D ffff88042a8d7d18 0 598 2 0x00000000
2016-01-14 00:17:09     ffff88042a8d7d18 ffff880102c93aa0 
ffff88042c248000 0000000000000282
2016-01-14 00:17:09     ffff88042a8d8000 ffff88042c248000 
0000000000000000 ffff88042b1f9800
2016-01-14 00:17:09     ffff88042bba4100 ffff88042a8d7d38 
ffffffff9363c167 ffff88042b1f9928
2016-01-14 00:17:09     Call Trace:
2016-01-14 00:17:09     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:09     [<ffffffff932e3109>] _xfs_log_force+0x189/0x280
2016-01-14 00:17:09     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     [<ffffffff932e322a>] xfs_log_force+0x2a/0x90
2016-01-14 00:17:09     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:09     [<ffffffff932edca3>] xfsaild+0x143/0x5f0


Nr 2:

2016-01-15 02:01:04     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:04     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:04     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:04     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:04     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:04     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:04     [<ffffffff8e2d1dac>] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:04     [<ffffffff8e2e122e>] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:04     [<ffffffff8e2d54d7>] xfs_ifree+0x27/0xd0
2016-01-15 02:01:04     [<ffffffff8e2ef776>] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:04     [<ffffffff8e2d5641>] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:04     [<ffffffff8e2d5846>] xfs_inactive+0x96/0x150
2016-01-15 02:01:04     [<ffffffff8e2da5cf>] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:04     [<ffffffff8e1de7b0>] evict+0xc0/0x1a0
2016-01-15 02:01:04     [<ffffffff8e1defdb>] iput+0x18b/0x1f0
2016-01-15 02:01:04     [<ffffffff8e1da678>] __dentry_kill+0x198/0x200
2016-01-15 02:01:04     [<ffffffff8e1dace0>] dput+0x1c0/0x220
2016-01-15 02:01:04     [<ffffffff8e1c5a24>] __fput+0x164/0x1e0
2016-01-15 02:01:04     [<ffffffff8e1c5aee>] ____fput+0xe/0x10
2016-01-15 02:01:04     [<ffffffff8e09d5ff>] task_work_run+0xbf/0x100
2016-01-15 02:01:04     [<ffffffff8e003bf8>] do_notify_resume+0x88/0xa0
2016-01-15 02:01:04     [<ffffffff8e640d7c>] int_signal+0x12/0x17
2016-01-15 02:01:03     [<ffffffff8e2d5846>] xfs_inactive+0x96/0x150
2016-01-15 02:01:03     [<ffffffff8e2da5cf>] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:03     [<ffffffff8e1de7b0>] evict+0xc0/0x1a0
2016-01-15 02:01:03     [<ffffffff8e1defdb>] iput+0x18b/0x1f0
2016-01-15 02:01:03     [<ffffffff8e1da678>] __dentry_kill+0x198/0x200
2016-01-15 02:01:03     [<ffffffff8e1dace0>] dput+0x1c0/0x220
2016-01-15 02:01:03     [<ffffffff8e1c5a24>] __fput+0x164/0x1e0
2016-01-15 02:01:03     [<ffffffff8e1c5aee>] ____fput+0xe/0x10
2016-01-15 02:01:03     [<ffffffff8e09d5ff>] task_work_run+0xbf/0x100
2016-01-15 02:01:03     [<ffffffff8e003bf8>] do_notify_resume+0x88/0xa0
2016-01-15 02:01:03     [<ffffffff8e640d7c>] int_signal+0x12/0x17
2016-01-15 02:01:03     INFO: task php5.4:9661 blocked for more than 120 
seconds.
2016-01-15 02:01:03     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:03     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:03     php5.4 D ffff8800189278e8 0 9661 15202 0x00000004
2016-01-15 02:01:03     ffff8800189278e8 ffff880624309d50 
ffff88000b4a57f0 ffff880000000002
2016-01-15 02:01:03     ffff880018928000 7fffffffffffffff 
ffff88000b4a57f0 0000000000000002
2016-01-15 02:01:03     0000000000000000 ffff880018927908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:03     Call Trace:
2016-01-15 02:01:03     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:03     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:03     [<ffffffff8e32cee7>] ? blk_peek_request+0x87/0x290
2016-01-15 02:01:03     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:03     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:03     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:03     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:02     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:02     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:02     php5.4 D ffff8800017338e8 0 9657 15202 0x00000004
2016-01-15 02:01:02     ffff8800017338e8 ffff8806242f0000 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     ffff880001734000 7fffffffffffffff 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     0000000000000000 ffff880001733908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:02     Call Trace:
2016-01-15 02:01:02     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:02     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:02     [<ffffffff8e0b25a8>] ? __enqueue_entity+0x78/0x80
2016-01-15 02:01:02     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:02     [<ffffffff8e16d478>] ? zone_statistics+0x98/0xc0
2016-01-15 02:01:02     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:02     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:02     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:02     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:02     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:02     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:02     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:02     [<ffffffff8e2d1dac>] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:02     [<ffffffff8e2e122e>] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:02     [<ffffffff8e2d54d7>] xfs_ifree+0x27/0xd0
2016-01-15 02:01:02     [<ffffffff8e2ef776>] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:02     [<ffffffff8e2d5641>] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:01     [<ffffffff8e2b2a23>] xfs_ialloc_read_agi+0x43/0xe0
2016-01-15 02:01:01     [<ffffffff8e2b3421>] xfs_dialloc+0xf1/0x2d0
2016-01-15 02:01:01     [<ffffffff8e2d35eb>] xfs_ialloc+0x5b/0x550
2016-01-15 02:01:01     [<ffffffff8e2df9b7>] ? kmem_zone_alloc+0x77/0x100
2016-01-15 02:01:01     [<ffffffff8e2f1c77>] ? 
xfs_trans_mod_dquot+0x47/0x190
2016-01-15 02:01:01     [<ffffffff8e2d3b4a>] xfs_dir_ialloc+0x6a/0x290
2016-01-15 02:01:01     [<ffffffff8e2d408b>] xfs_create+0x2bb/0x6e0
2016-01-15 02:01:01     [<ffffffff8e00d775>] ? native_sched_clock+0x35/0x80
2016-01-15 02:01:01     [<ffffffff8e21cab1>] ? get_cached_acl+0x21/0x70
2016-01-15 02:01:01     [<ffffffff8e2d09b3>] xfs_generic_create+0xd3/0x2b0
2016-01-15 02:01:01     [<ffffffff8e1dc4ed>] ? __d_lookup+0x11d/0x160
2016-01-15 02:01:01     [<ffffffff8e2d0bc4>] xfs_vn_mknod+0x14/0x20
2016-01-15 02:01:01     [<ffffffff8e2d0c03>] xfs_vn_create+0x13/0x20
2016-01-15 02:01:01     [<ffffffff8e1cf307>] vfs_create+0x77/0xc0
2016-01-15 02:01:01     [<ffffffff8e1d076d>] do_last.isra.57+0x73d/0xd30
2016-01-15 02:01:01     [<ffffffff8e1d1084>] ? path_init+0xc4/0x430
2016-01-15 02:01:01     [<ffffffff8e1d2012>] path_openat+0x92/0x5a0
2016-01-15 02:01:01     [<ffffffff8e1d3f78>] ? user_path_at_empty+0x68/0xa0
2016-01-15 02:01:01     [<ffffffff8e1d4063>] do_filp_open+0x43/0xa0
2016-01-15 02:01:01     [<ffffffff8e1e0f24>] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:01:01     [<ffffffff8e1c2dec>] do_sys_open+0x13c/0x230
2016-01-15 02:01:01     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:01:01     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:01:01     [<ffffffff8e1c2f02>] SyS_open+0x22/0x30
2016-01-15 02:01:01     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:01:01     INFO: task php5.4:9657 blocked for more than 120 
seconds.
2016-01-15 02:01:00     INFO: task php5.2:6976 blocked for more than 120 
seconds.
2016-01-15 02:01:00     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:00     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:00     php5.2 D ffff88015bd87568 0 6976 15202 0x00000004
2016-01-15 02:01:00     ffff88015bd87568 ffff8806242c0000 
ffff8801938e0000 ffff8800374ded00
2016-01-15 02:01:00     ffff88015bd88000 7fffffffffffffff 
ffff8801938e0000 0000000000000002
2016-01-15 02:01:00     0000000000000000 ffff88015bd87588 
ffffffff8e63c967 00000000000001ed
2016-01-15 02:01:00     Call Trace:
2016-01-15 02:01:00     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:00     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:00     [<ffffffff8e0b873c>] ? dequeue_entity+0x13c/0x580
2016-01-15 02:01:00     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:00     [<ffffffff8e0b1700>] ? 
arch_vtime_task_switch+0x40/0x90
2016-01-15 02:01:00     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:00     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:00     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:00     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:00     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:00     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:00     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:00:59     INFO: task kworker/4:1H:6823 blocked for more 
than 120 seconds.
2016-01-15 02:00:59     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:59     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:59     kworker/4:1H D ffff880302813c88 0 6823 2 0x00000000
2016-01-15 02:00:59     Workqueue: xfs-log/sda3 xfs_log_worker
2016-01-15 02:00:59     ffff880302813c88 ffff8806243ed7f0 
ffff8800038b0000 ffff880621729400
2016-01-15 02:00:59     ffff880302814000 ffff8800bb179128 
ffff8800bb179000 0000000000000000
2016-01-15 02:00:59     ffff8800038b0000 ffff880302813ca8 
ffffffff8e63c967 ffff8800038b0000
2016-01-15 02:00:59     Call Trace:
2016-01-15 02:00:59     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:59     [<ffffffff8e2e359b>] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-15 02:00:59     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:59     [<ffffffff8e2df093>] xfs_trans_commit+0x1e3/0x220
2016-01-15 02:00:59     [<ffffffff8e2b8b60>] xfs_sync_sb+0x60/0xa0
2016-01-15 02:00:59     [<ffffffff8e2e1bb4>] ? 
xfs_log_need_covered+0x74/0x100
2016-01-15 02:00:59     [<ffffffff8e2e332d>] xfs_log_worker+0x4d/0x50
2016-01-15 02:00:59     [<ffffffff8e0986bf>] process_one_work+0x14f/0x410
2016-01-15 02:00:59     [<ffffffff8e098ad9>] worker_thread+0x119/0x510
2016-01-15 02:00:59     [<ffffffff8e63c3d1>] ? __schedule+0x4f1/0x990
2016-01-15 02:00:59     [<ffffffff8e0989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-15 02:00:59     [<ffffffff8e09f1f9>] kthread+0xc9/0xe0
2016-01-15 02:00:59     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:59     [<ffffffff8e640fa2>] ret_from_fork+0x42/0x70
2016-01-15 02:00:59     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:58     [<ffffffff8e1df3d9>] ? inode_lru_isolate+0x89/0x1c0
2016-01-15 02:00:58     [<ffffffff8e177e58>] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-15 02:00:58     [<ffffffff8e1df350>] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [<ffffffff8e1df350>] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:58     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:58     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:58     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:58     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:58     [<ffffffff8e1646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:58     [<ffffffff8e164b8a>] try_to_free_pages+0xda/0x160
2016-01-15 02:00:58     [<ffffffff8e6364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:58     [<ffffffff8e1581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:58     [<ffffffff8e158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:58     [<ffffffff8e07e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:58     [<ffffffff8e07e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:58     [<ffffffff8e1a7f63>] ? kmem_cache_alloc+0x193/0x1f0
2016-01-15 02:00:58     [<ffffffff8e1190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:58     [<ffffffff8e0b0f99>] ? account_user_time+0x99/0xb0
2016-01-15 02:00:58     [<ffffffff8e08014c>] do_fork+0xbc/0x350
2016-01-15 02:00:58     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:58     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:58     [<ffffffff8e080466>] SyS_clone+0x16/0x20
2016-01-15 02:00:58     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:57     [<ffffffff8e013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:57     [<ffffffff8e1f4ee0>] SyS_fsync+0x10/0x20
2016-01-15 02:00:57     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     INFO: task auth:9631 blocked for more than 120 
seconds.
2016-01-15 02:00:57     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:57     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:57     auth D ffff8809eea33588 0 9631 9616 0x00000004
2016-01-15 02:00:57     ffff8809eea33588 ffff8806242c1d50 
ffff880b97f19d50 ffff8809eea33568
2016-01-15 02:00:57     ffff8809eea34000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:57     ffff880b97f19d50 ffff8809eea335a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:57     Call Trace:
2016-01-15 02:00:57     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:57     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:57     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:57     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:57     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:57     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:57     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:57     [<ffffffff8e0d8cb6>] ? 
__call_rcu_nocb_enqueue+0xe6/0xf0
2016-01-15 02:00:56     [<ffffffff8e2c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [<ffffffff8e1f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [<ffffffff8e1f4afc>] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [<ffffffff8e1f4c68>] do_fsync+0x38/0x70
2016-01-15 02:00:56     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:56     [<ffffffff8e013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:56     [<ffffffff8e1f4ee0>] SyS_fsync+0x10/0x20
2016-01-15 02:00:56     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:56     INFO: task mysqld:20600 blocked for more than 
120 seconds.
2016-01-15 02:00:56     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:56     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:56     mysqld D ffff8809218ffd98 0 20600 5893 0x00000000
2016-01-15 02:00:56     ffff8809218ffd98 ffff8806242d8000 
ffff880123ef57f0 0000000000000000
2016-01-15 02:00:56     ffff880921900000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:56     ffff880123ef57f0 ffff8809218ffdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:56     Call Trace:
2016-01-15 02:00:56     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:56     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:56     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:56     [<ffffffff8e2c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [<ffffffff8e1f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [<ffffffff8e1f4afc>] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [<ffffffff8e1f4c68>] do_fsync+0x38/0x70
2016-01-15 02:00:55     [<ffffffff8e1646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:55     [<ffffffff8e164b8a>] try_to_free_pages+0xda/0x160
2016-01-15 02:00:55     [<ffffffff8e6364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:55     [<ffffffff8e1581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:55     [<ffffffff8e158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:55     [<ffffffff8e07e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:55     [<ffffffff8e07e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:55     [<ffffffff8e1190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:55     [<ffffffff8e0b0f99>] ? account_user_time+0x99/0xb0
2016-01-15 02:00:55     [<ffffffff8e08014c>] do_fork+0xbc/0x350
2016-01-15 02:00:55     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:55     [<ffffffff8e080466>] SyS_clone+0x16/0x20
2016-01-15 02:00:55     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:55     INFO: task mysqld:7069 blocked for more than 120 
seconds.
2016-01-15 02:00:55     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:55     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:55     mysqld D ffff880f0abbbd98 0 7069 5893 0x00000000
2016-01-15 02:00:55     ffff880f0abbbd98 ffff8806242c0000 
ffff880f0ab53aa0 0000000000000000
2016-01-15 02:00:55     ffff880f0abbc000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:55     ffff880f0ab53aa0 ffff880f0abbbdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:55     Call Trace:
2016-01-15 02:00:55     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:55     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:55     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:54     INFO: task mysqld:6683 blocked for more than 120 
seconds.
2016-01-15 02:00:54     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:54     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:54     mysqld D ffff880f0b2f3588 0 6683 5893 0x00000000
2016-01-15 02:00:54     ffff880f0b2f3588 ffff8806242dbaa0 
ffff880f09913aa0 000000000000254e
2016-01-15 02:00:54     ffff880f0b2f4000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:54     ffff880f09913aa0 ffff880f0b2f35a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:54     Call Trace:
2016-01-15 02:00:54     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:54     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:54     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:54     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:54     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:54     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:54     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:54     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:54     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:54     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:54     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:53     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:53     mysqld D ffff88064549bbc8 0 21561 5534 0x00000000
2016-01-15 02:00:53     ffff88064549bbc8 ffff88062426d7f0 
ffff88000b4cbaa0 0000000000000040
2016-01-15 02:00:53     ffff88064549c000 ffff880f1631b674 
ffff88000b4cbaa0 00000000ffffffff
2016-01-15 02:00:53     ffff880f1631b678 ffff88064549bbe8 
ffffffff8e63c967 00000000000282c2
2016-01-15 02:00:53     Call Trace:
2016-01-15 02:00:53     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:53     [<ffffffff8e63cbde>] 
schedule_preempt_disabled+0xe/0x10
2016-01-15 02:00:53     [<ffffffff8e63e7e5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-15 02:00:53     [<ffffffff8e63e87b>] mutex_lock+0x1b/0x2f
2016-01-15 02:00:53     [<ffffffff8e1d028b>] do_last.isra.57+0x25b/0xd30
2016-01-15 02:00:53     [<ffffffff8e1d1084>] ? path_init+0xc4/0x430
2016-01-15 02:00:53     [<ffffffff8e1d2012>] path_openat+0x92/0x5a0
2016-01-15 02:00:53     [<ffffffff8e17d817>] ? handle_mm_fault+0xb47/0x1860
2016-01-15 02:00:53     [<ffffffff8e1d4063>] do_filp_open+0x43/0xa0
2016-01-15 02:00:53     [<ffffffff8e1e0f24>] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:00:53     [<ffffffff8e1c2dec>] do_sys_open+0x13c/0x230
2016-01-15 02:00:53     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:53     [<ffffffff8e1c2f02>] SyS_open+0x22/0x30
2016-01-15 02:00:52     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:52     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:52     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:52     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:52     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:52     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:52     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:52     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:52     [<ffffffff8e165287>] kswapd+0x527/0x980
2016-01-15 02:00:52     [<ffffffff8e164d60>] ? 
mem_cgroup_shrink_node_zone+0x150/0x150
2016-01-15 02:00:52     [<ffffffff8e09f1f9>] kthread+0xc9/0xe0
2016-01-15 02:00:52     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     [<ffffffff8e640fa2>] ret_from_fork+0x42/0x70
2016-01-15 02:00:52     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     INFO: task mysqld:21561 blocked for more than 
120 seconds.
2016-01-15 02:00:52     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:51     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:51     kswapd1 D ffff880622e87848 0 195 2 0x00000000
2016-01-15 02:00:51     ffff880622e87848 ffff8806242d8000 
ffff880622e33aa0 0000000000000000
2016-01-15 02:00:51     ffff880622e88000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:51     ffff880622e33aa0 ffff880622e87868 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:51     Call Trace:
2016-01-15 02:00:51     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:51     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:51     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:51     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:51     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:51     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:50     INFO: task kswapd1:195 blocked for more than 120 
seconds.
2016-01-15 02:00:50     Not tainted 4.1.15+68-ph #1

Nr 3:
2016-01-12 03:18:50     [<ffffffffb107e61b>] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:50     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:50     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:50     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:50     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:50     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:50     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:50     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:49     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:49     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:49     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:49     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:49     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:49     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:49     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:49     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:49     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:49     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:49     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:49     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:49     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:49     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:49     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:49     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:49     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:49     [<ffffffffb104dca7>] ? do_page_fault+0x37/0x90
2016-01-12 03:18:49     [<ffffffffb11a794a>] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:48     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:48     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:48     sshd D ffff880859e6f198 0 6171 1 0x00000000
2016-01-12 03:18:48     ffff880859e6f198 ffff88085c2e1d50 
ffff880855d53aa0 ffff880859e6f248
2016-01-12 03:18:48     ffff880859e70000 ffff880859e6f300 
7fffffffffffffff ffff880855d53aa0
2016-01-12 03:18:48     ffffffffffffffff ffff880859e6f1b8 
ffffffffb163c967 ffff880859e6f298
2016-01-12 03:18:48     Call Trace:
2016-01-12 03:18:48     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:48     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:48     [<ffffffffb13506e5>] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:48     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:48     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:48     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:48     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:48     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:48     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:48     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:48     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:48     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:47     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:47     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:47     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:47     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:47     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:47     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:47     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:47     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:47     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:47     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:47     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:47     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:47     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:47     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:47     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:47     [<ffffffffb108044c>] sys_vfork+0x1c/0x20
2016-01-12 03:18:47     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:47     INFO: task sshd:6171 blocked for more than 120 
seconds.
2016-01-12 03:18:46     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:46     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:46     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:46     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:46     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:46     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:46     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:46     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:46     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:46     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:46     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:46     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:46     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:46     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:46     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:46     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:45     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:45     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:45     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:45     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:45     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:45     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:45     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:45     INFO: task qmail-rspawn:6145 blocked for more 
than 120 seconds.
2016-01-12 03:18:45     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:45     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:45     qmail-rspawn D ffff88085af93348 0 6145 6125 
0x00000002
2016-01-12 03:18:45     ffff88085af93348 ffff88085c300000 
ffff8808554c9d50 0000000000000000
2016-01-12 03:18:45     ffff88085af94000 ffff88085af934b0 
7fffffffffffffff ffff8808554c9d50
2016-01-12 03:18:45     ffffffffffffffff ffff88085af93368 
ffffffffb163c967 ffff88085af933e8
2016-01-12 03:18:45     Call Trace:
2016-01-12 03:18:45     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:45     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:45     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:44     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:44     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:44     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:44     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:44     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:44     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:44     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:44     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:44     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:44     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:44     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:44     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:44     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:44     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:44     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:44     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:44     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:43     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:43     INFO: task zabbix_agentd:6130 blocked for more 
than 120 seconds.
2016-01-12 03:18:43     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:43     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:43     zabbix_agentd D ffff881053bc7348 0 6130 6118 
0x00000004
2016-01-12 03:18:43     ffff881053bc7348 ffff88085c300000 
ffff88105a4d9d50 ffff881053bc73d8
2016-01-12 03:18:43     ffff881053bc8000 ffff881053bc74b0 
7fffffffffffffff ffff88105a4d9d50
2016-01-12 03:18:43     ffffffffffffffff ffff881053bc7368 
ffffffffb163c967 ffff881053bc7368
2016-01-12 03:18:43     Call Trace:
2016-01-12 03:18:43     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:43     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:43     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:43     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:43     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:43     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:43     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:43     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:43     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:42     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:42     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:42     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:42     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:42     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:42     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:42     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:42     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:42     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:42     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:42     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:42     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:42     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:42     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:42     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:42     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:42     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:42     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:41     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:41     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:41     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:41     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:41     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:41     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:41     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:41     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:41     [<ffffffffb1326f7c>] ? part_round_stats+0x5c/0x70
2016-01-12 03:18:41     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:41     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:41     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:41     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:41     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:41     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:41     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:40     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:40     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:40     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:40     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:40     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:40     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:40     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:40     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:40     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:40     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:40     INFO: task zabbix_agentd:6129 blocked for more 
than 120 seconds.
2016-01-12 03:18:40     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:40     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:40     zabbix_agentd D ffff881053bd7348 0 6129 6118 
0x00000006
2016-01-12 03:18:40     ffff881053bd7348 ffff88085c2e1d50 
ffff88105340d7f0 ffffea0017e65180
2016-01-12 03:18:40     ffff881053bd8000 ffff881053bd74b0 
7fffffffffffffff ffff88105340d7f0
2016-01-12 03:18:40     ffffffffffffffff ffff881053bd7368 
ffffffffb163c967 ffff88085fa59650
2016-01-12 03:18:40     Call Trace:
2016-01-12 03:18:40     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:39     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:39     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:39     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:39     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:39     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:39     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:39     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:39     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:39     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:39     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:39     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:39     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:39     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:39     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:39     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:39     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:38     INFO: task zabbix_agentd:6128 blocked for more 
than 120 seconds.
2016-01-12 03:18:38     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:38     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:38     zabbix_agentd D ffff88105341f348 0 6128 6118 
0x00000004
2016-01-12 03:18:38     ffff88105341f348 ffff88085c2e57f0 
ffff880858739d50 ffffea00009e2100
2016-01-12 03:18:38     ffff881053420000 ffff88105341f4b0 
7fffffffffffffff ffff880858739d50
2016-01-12 03:18:38     ffffffffffffffff ffff88105341f368 
ffffffffb163c967 ffff88085fad9650
2016-01-12 03:18:38     Call Trace:
2016-01-12 03:18:38     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:38     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:38     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:38     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:38     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:38     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:38     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:38     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:38     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:38     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:38     [<ffffffffb11a659c>] ? __free_slab+0xcc/0x1c0
2016-01-12 03:18:37     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:37     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:37     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:37     [<ffffffffb14d4b8d>] ? sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [<ffffffffb10a98b6>] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-12 03:18:37     [<ffffffffb11d6f71>] ? set_fd_set+0x21/0x30
2016-01-12 03:18:37     [<ffffffffb11d7f42>] ? core_sys_select+0x242/0x2f0
2016-01-12 03:18:37     [<ffffffffb11a7f63>] kmem_cache_alloc+0x193/0x1f0
2016-01-12 03:18:37     [<ffffffffb14d4b8d>] sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [<ffffffffb11ddab6>] alloc_inode+0x26/0xa0
2016-01-12 03:18:37     [<ffffffffb11df84a>] new_inode_pseudo+0x1a/0x70
2016-01-12 03:18:37     [<ffffffffb14d4e0e>] sock_alloc+0x1e/0x80
2016-01-12 03:18:37     [<ffffffffb14d69ac>] SYSC_accept4+0x7c/0x210
2016-01-12 03:18:37     [<ffffffffb10b159d>] ? vtime_account_user+0x5d/0x70
2016-01-12 03:18:37     [<ffffffffb114e243>] ? 
context_tracking_user_exit+0x13/0x20
2016-01-12 03:18:37     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:37     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:37     [<ffffffffb14d8240>] SyS_accept+0x10/0x20
2016-01-12 03:18:37     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:36     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:36     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:36     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:36     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:36     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:36     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:36     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:36     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:36     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:36     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:36     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:36     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:36     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:36     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:36     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:36     [<ffffffffb11d7220>] ? __pollwait+0xf0/0xf0
2016-01-12 03:18:36     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:35     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:35     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:35     INFO: task nrpe:6076 blocked for more than 120 
seconds.
2016-01-12 03:18:35     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:35     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:35     nrpe D ffff880853e6b1a8 0 6076 1 0x00000002
2016-01-12 03:18:35     ffff880853e6b1a8 ffffffffb1c134a0 
ffff8808554e57f0 ffff88105f5d6240
2016-01-12 03:18:35     ffff880853e6c000 ffff880853e6b310 
7fffffffffffffff ffff8808554e57f0
2016-01-12 03:18:35     ffffffffffffffff ffff880853e6b1c8 
ffffffffb163c967 0000000000000003
2016-01-12 03:18:35     Call Trace:
2016-01-12 03:18:35     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:35     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:35     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:35     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:35     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:35     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:35     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:35     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:34     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:34     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:34     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:34     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:34     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:34     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:34     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:34     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:34     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:34     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:34     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:34     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:34     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:34     [<ffffffffb11c8910>] ? cp_new_stat+0x140/0x160
2016-01-12 03:18:34     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:34     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:34     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:34     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:33     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:33     [<ffffffffb13506e5>] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:33     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:33     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:33     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:33     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:33     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:33     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:33     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:33     [<ffffffffb132cb37>] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:33     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:33     [<ffffffffb14722b0>] ? 
scsi_prep_state_check.isra.22+0x80/0x80
2016-01-12 03:18:33     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:33     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:33     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:33     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:33     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:32     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:32     [<ffffffffb12c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-12 03:18:32     [<ffffffffb11f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-12 03:18:32     [<ffffffffb11f4afc>] vfs_fsync+0x1c/0x20
2016-01-12 03:18:32     [<ffffffffb11f4c68>] do_fsync+0x38/0x70
2016-01-12 03:18:32     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:32     [<ffffffffb1013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-12 03:18:32     [<ffffffffb11f4ee0>] SyS_fsync+0x10/0x20
2016-01-12 03:18:32     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:32     INFO: task cron:6006 blocked for more than 120 
seconds.
2016-01-12 03:18:32     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:32     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:32     cron D ffff881053b57348 0 6006 1 0x00000000
2016-01-12 03:18:32     ffff881053b57348 ffff880858738000 
ffff88105a8057f0 ffff8810595db080
2016-01-12 03:18:32     ffff881053b58000 ffff881053b574b0 
7fffffffffffffff ffff88105a8057f0
2016-01-12 03:18:32     ffffffffffffffff ffff881053b57368 
ffffffffb163c967 ffff880000055000
2016-01-12 03:18:32     Call Trace:
2016-01-12 03:18:32     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:31     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:31     INFO: task mysqld:6762 blocked for more than 120 
seconds.
2016-01-12 03:18:31     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:31     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:31     mysqld D ffff880843887b58 0 6762 5590 0x00000000
2016-01-12 03:18:31     ffff880843887b58 ffff88085c301d50 
ffff88084385baa0 ffff880572ac09d0
2016-01-12 03:18:31     ffff880843888000 ffff880843887cc0 
7fffffffffffffff ffff88084385baa0
2016-01-12 03:18:31     ffffffffffffffff ffff880843887b78 
ffffffffb163c967 0000000000000100
2016-01-12 03:18:31     Call Trace:
2016-01-12 03:18:31     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:31     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:31     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:31     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:31     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:31     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:31     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:31     [<ffffffffb12bb940>] ? 
xfs_submit_ioend_bio.isra.14+0x40/0x40
2016-01-12 03:18:31     [<ffffffffb1639f03>] ? 
xfs_file_dio_aio_write+0x2a4/0x2f8
2016-01-12 03:18:30     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:30     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:30     [<ffffffffb100268b>] ? __switch_to+0x1eb/0x590
2016-01-12 03:18:30     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:30     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:30     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:30     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:30     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:30     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:30     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [<ffffffffb11a794a>] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:30     [<ffffffffb107e61b>] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:30     [<ffffffffb11dd9f3>] ? inode_init_always+0xe3/0x180
2016-01-12 03:18:30     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:30     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:30     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:30     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:29     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:29     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:29     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:29     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:29     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:29     [<ffffffffb132cb37>] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:29     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:29     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:29     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:29     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:29     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:29     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:29     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:29     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:29     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:29     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:28     [<ffffffffb109f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:28     INFO: task proftpd:5592 blocked for more than 
120 seconds.
2016-01-12 03:18:28     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:28     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:28     proftpd D ffff88105487b198 0 5592 1 0x00000006
2016-01-12 03:18:28     ffff88105487b198 ffffffffb1c134a0 
ffff8808587b9d50 ffff88105487b1d8
2016-01-12 03:18:28     ffff88105487c000 ffff88105487b300 
7fffffffffffffff ffff8808587b9d50
2016-01-12 03:18:28     ffffffffffffffff ffff88105487b1b8 
ffffffffb163c967 ffff88105b9881d8
2016-01-12 03:18:28     Call Trace:
2016-01-12 03:18:28     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:28     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:28     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:28     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:28     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:27     [<ffffffffb12b6c79>] ? xfs_imap_to_bp+0xc9/0xe0
2016-01-12 03:18:27     [<ffffffffb12e327a>] xfs_log_force+0x2a/0x90
2016-01-12 03:18:27     [<ffffffffb12d68c2>] xfs_iflush+0x202/0x210
2016-01-12 03:18:27     [<ffffffffb10c9872>] ? up+0x32/0x50
2016-01-12 03:18:27     [<ffffffffb12e723a>] xfs_inode_item_push+0xda/0x140
2016-01-12 03:18:27     [<ffffffffb12edbb0>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [<ffffffffb12ede7e>] xfsaild+0x2ce/0x5f0
2016-01-12 03:18:27     [<ffffffffb163c3d1>] ? __schedule+0x4f1/0x990
2016-01-12 03:18:27     [<ffffffffb12edbb0>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [<ffffffffb109f1f9>] kthread+0xc9/0xe0
2016-01-12 03:18:27     [<ffffffffb109f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:27     [<ffffffffb1640fa2>] ret_from_fork+0x42/0x70
2016-01-12 03:18:26     [<ffffffffb10ad197>] ? try_to_wake_up+0x197/0x340
2016-01-12 03:18:26     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:26     [<ffffffffb10ad395>] ? wake_up_process+0x15/0x20
2016-01-12 03:18:26     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:26     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:26     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:26     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:26     [<ffffffffb12c2f77>] ? xfs_buf_read_map+0x37/0x110
2016-01-12 03:18:26     [<ffffffffb12e303b>] _xfs_log_force+0x6b/0x280
2016-01-12 03:18:25     ffff8810566e7a28 ffff88085c31d7f0 
ffff880859430000 000000000000137e
2016-01-12 03:18:25     ffff8810566e8000 ffff8810566e7b90 
7fffffffffffffff ffff880859430000
2016-01-12 03:18:25     ffffffffffffffff ffff8810566e7a48 
ffffffffb163c967 ffff88105b988060
2016-01-12 03:18:25     Call Trace:
2016-01-12 03:18:25     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:25     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:25     [<ffffffffb100d7c9>] ? sched_clock+0x9/0x10
2016-01-12 03:18:25     [<ffffffffb10b0ae5>] ? sched_clock_cpu+0x85/0xc0
2016-01-12 03:18:24     INFO: task xfsaild/sda3:730 blocked for more 
than 120 seconds.
2016-01-12 03:18:24     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:24     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:24     xfsaild/sda3 D ffff8810566e7a28 0 730 2 0x00000000



I can give you many more if they may help.


Greets,
Stefan

xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe]

Hello,

since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle 
call traces show xfs.

Attention all those are down top instead of top down (sorry for the 
wrong order).

NR 1:

2016-01-14 00:17:23     [<ffffffff93501ca5>] rtnl_lock+0x15/0x20
2016-01-14 00:17:23     [<ffffffff93501cf6>] rtnetlink_rcv+0x16/0x40
2016-01-14 00:17:23     [<ffffffff935252aa>] netlink_unicast+0x13a/0x1d0
2016-01-14 00:17:23     [<ffffffff9352554a>] netlink_sendmsg+0x20a/0x600
2016-01-14 00:17:23     [<ffffffff934d7d61>] SYSC_sendto+0x111/0x170
2016-01-14 00:17:23     [<ffffffff9314e263>] ? 
context_tracking_user_exit+0x13/0x20
2016-01-14 00:17:23     [<ffffffff930134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:23     [<ffffffff934d82ae>] SyS_sendto+0xe/0x10
2016-01-14 00:17:23     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:22     [<ffffffff932e449f>] xlog_cil_push+0x22f/0x3f0
2016-01-14 00:17:22     [<ffffffff932e4675>] xlog_cil_push_work+0x15/0x20
2016-01-14 00:17:22     [<ffffffff930986bf>] process_one_work+0x14f/0x410
2016-01-14 00:17:22     [<ffffffff930989ac>] 
process_scheduled_works+0x2c/0x40
2016-01-14 00:17:22     [<ffffffff93098c4a>] worker_thread+0x28a/0x510
2016-01-14 00:17:22     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:22     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:22     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:22     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:22     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:22     INFO: task qpsmtpd p:25 :::12850 blocked for 
more than 120 seconds.
2016-01-14 00:17:22     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:22     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:22     qpsmtpd p:25 :: D ffff88071f807c48 0 12850 6124 
0x00000000
2016-01-14 00:17:22     ffff88071f807c48 ffff8805aabf0000 
ffff8806c5780000 ffff88083ffd7000
2016-01-14 00:17:22     ffff88071f808000 ffffffff93ce6444 
ffff8806c5780000 00000000ffffffff
2016-01-14 00:17:22     ffffffff93ce6448 ffff88071f807c68 
ffffffff9363c167 ffff88042b9f0000
2016-01-14 00:17:22     Call Trace:
2016-01-14 00:17:22     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:22     [<ffffffff9363c3de>] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:22     [<ffffffff9363dfd5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:22     [<ffffffff9363e06b>] mutex_lock+0x1b/0x2f
2016-01-14 00:17:21     [<ffffffff931a7617>] __kmalloc+0x1c7/0x240
2016-01-14 00:17:21     [<ffffffff931e0224>] alloc_fdmem+0x24/0x40
2016-01-14 00:17:21     [<ffffffff931e02b7>] alloc_fdtable+0x77/0x100
2016-01-14 00:17:21     [<ffffffff931e0af2>] dup_fd+0x262/0x330
2016-01-14 00:17:21     [<ffffffff9307ec9a>] 
copy_process.part.37+0x76a/0x1a80
2016-01-14 00:17:21     [<ffffffff931190ec>] ? 
acct_account_cputime+0x1c/0x20
2016-01-14 00:17:21     [<ffffffff9308014c>] do_fork+0xbc/0x350
2016-01-14 00:17:21     [<ffffffff930134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-14 00:17:21     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:21     [<ffffffff93080466>] SyS_clone+0x16/0x20
2016-01-14 00:17:21     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:21     INFO: task kworker/7:0:23773 blocked for more 
than 120 seconds.
2016-01-14 00:17:21     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:21     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:21     kworker/7:0 D ffff8805243bbb38 0 23773 2 0x00000000
2016-01-14 00:17:21     Workqueue: xfs-cil/sdb1 xlog_cil_push_work
2016-01-14 00:17:21     ffff8805243bbb38 ffff88082d941d50 
ffff880790560000 0000000000000000
2016-01-14 00:17:21     ffff8805243bc000 ffff88010af7c0b8 
ffff88042c067d28 ffff880790560000
2016-01-14 00:17:21     ffff88042c067d00 ffff8805243bbb58 
ffffffff9363c167 ffff88042c067c00
2016-01-14 00:17:21     Call Trace:
2016-01-14 00:17:21     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:21     [<ffffffff932e241b>] 
xlog_state_get_iclog_space+0xdb/0x2e0
2016-01-14 00:17:21     [<ffffffff932dfe14>] ? xlog_bdstrat+0x34/0x70
2016-01-14 00:17:21     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:21     [<ffffffff932e291e>] xlog_write+0x18e/0x700
2016-01-14 00:17:20     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:20     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:20     [<ffffffffc03efb24>] ? aac_fib_send+0x2c4/0x4f0 
[aacraid]
2016-01-14 00:17:20     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:20     [<ffffffff932b4b87>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:20     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:20     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:20     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:20     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:20     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:20     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:20     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:20     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:20     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:20     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:20     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:20     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:20     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:20     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:20     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:20     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:20     [<ffffffff9316d468>] ? zone_statistics+0x98/0xc0
2016-01-14 00:17:20     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:20     [<ffffffff931e0224>] ? alloc_fdmem+0x24/0x40
2016-01-14 00:17:19     [<ffffffff93355fa9>] ida_pre_get+0x69/0x100
2016-01-14 00:17:19     [<ffffffff93356d9e>] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:19     [<ffffffff930a7c9e>] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:19     [<ffffffff93097adb>] create_worker+0x4b/0x1f0
2016-01-14 00:17:19     [<ffffffff930e1e0a>] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:19     [<ffffffff93098daf>] worker_thread+0x3ef/0x510
2016-01-14 00:17:19     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:19     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:19     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:19     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:19     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:19     INFO: task httpd:22880 blocked for more than 120 
seconds.
2016-01-14 00:17:19     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:19     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:19     httpd D ffff88005557b108 0 22880 22644 0x00000000
2016-01-14 00:17:19     ffff88005557b108 ffff88042dbdbaa0 
ffff8807e65a57f0 ffff88005557b1b8
2016-01-14 00:17:19     ffff88005557c000 ffff88005557b270 
7fffffffffffffff ffff8807e65a57f0
2016-01-14 00:17:19     ffffffffffffffff ffff88005557b128 
ffffffff9363c167 0000000000000292
2016-01-14 00:17:19     Call Trace:
2016-01-14 00:17:19     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:19     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:19     [<ffffffff933506b5>] ? cfq_set_request+0xa5/0x4c0
2016-01-14 00:17:19     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:19     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:19     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:18     [<ffffffff9334c148>] ? 
__cfq_set_active_queue+0xf8/0x230
2016-01-14 00:17:18     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:18     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:18     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:18     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:18     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:18     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:18     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:18     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:18     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:18     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:18     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:18     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:18     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:18     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:18     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:18     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:18     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:18     [<ffffffff930ad1f3>] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:18     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:18     [<ffffffff93355fa9>] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:18     [<ffffffff930c2f43>] ? __wake_up+0x53/0x70
2016-01-14 00:17:18     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:18     [<ffffffff930b42f0>] ? pick_next_entity+0x90/0x170
2016-01-14 00:17:17     [<ffffffff93098daf>] worker_thread+0x3ef/0x510
2016-01-14 00:17:17     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:17     [<ffffffff930989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-14 00:17:17     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:17     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:17     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:17     INFO: task kworker/7:0H:7845 blocked for more 
than 120 seconds.
2016-01-14 00:17:17     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:17     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:17     kworker/7:0H D ffff8806ec7ff108 0 7845 2 0x00000000
2016-01-14 00:17:17     ffff8806ec7ff108 ffff88044840d7f0 
ffff88082762baa0 ffff88044840d7f0
2016-01-14 00:17:17     ffff8806ec800000 ffff8806ec7ff270 
7fffffffffffffff ffff88082762baa0
2016-01-14 00:17:17     ffffffffffffffff ffff8806ec7ff128 
ffffffff9363c167 ffff8806ec7ff148
2016-01-14 00:17:17     Call Trace:
2016-01-14 00:17:17     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:17     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:17     [<ffffffff930a98b6>] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-14 00:17:17     [<ffffffff930ad1f3>] ? try_to_wake_up+0x1f3/0x330
2016-01-14 00:17:17     [<ffffffff9335090f>] ? cfq_set_request+0x2ff/0x4c0
2016-01-14 00:17:17     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:17     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:17     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:17     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:17     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:16     [<ffffffff932d595e>] xfs_iunpin_wait+0xae/0x140
2016-01-14 00:17:16     [<ffffffff930c2be0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-14 00:17:16     [<ffffffff932ca610>] xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:16     [<ffffffff932caae7>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-14 00:17:16     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:16     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:16     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:16     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:16     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:16     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:16     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:16     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:16     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:16     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:16     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:16     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:16     [<ffffffff93355fa9>] ? ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:16     [<ffffffff9300d775>] ? native_sched_clock+0x35/0x80
2016-01-14 00:17:16     [<ffffffff93355fa9>] ida_pre_get+0x69/0x100
2016-01-14 00:17:16     [<ffffffff93356d9e>] ida_simple_get+0x4e/0xf0
2016-01-14 00:17:16     [<ffffffff930a7c9e>] ? 
finish_task_switch+0x11e/0x1a0
2016-01-14 00:17:16     [<ffffffff93097adb>] create_worker+0x4b/0x1f0
2016-01-14 00:17:16     [<ffffffff930e1e0a>] ? mod_timer+0x14a/0x1c0
2016-01-14 00:17:15     [<ffffffff931f4c58>] do_fsync+0x38/0x70
2016-01-14 00:17:15     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:15     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:15     [<ffffffff931f4ef3>] SyS_fdatasync+0x13/0x20
2016-01-14 00:17:15     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:15     INFO: task kworker/7:2:22195 blocked for more 
than 120 seconds.
2016-01-14 00:17:15     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:15     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:15     kworker/7:2 D ffff88009ac93108 0 22195 2 0x00000000
2016-01-14 00:17:15     ffff88009ac93108 ffff88082d941d50 
ffff88044840d7f0 0000000950d64c61
2016-01-14 00:17:15     ffff88009ac94000 ffff88009ac93270 
7fffffffffffffff ffff88044840d7f0
2016-01-14 00:17:15     ffffffffffffffff ffff88009ac93128 
ffffffff9363c167 74612e656e6f6568
2016-01-14 00:17:15     Call Trace:
2016-01-14 00:17:15     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:15     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:15     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:15     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:15     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:15     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:15     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:15     [<ffffffff932e3338>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-14 00:17:15     [<ffffffff932b4b87>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-14 00:17:15     [<ffffffff932e35db>] xfs_log_force_lsn+0x3b/0xa0
2016-01-14 00:17:15     [<ffffffff932ca610>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-14 00:17:14     [<ffffffff9354d671>] ? tcp_write_xmit+0x2b1/0xda0
2016-01-14 00:17:14     [<ffffffff934da9a1>] 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:14     [<ffffffff934db9b3>] sk_alloc+0x33/0xb0
2016-01-14 00:17:14     [<ffffffff93590231>] unix_create1+0x51/0x1a0
2016-01-14 00:17:14     [<ffffffff935903f3>] unix_create+0x73/0x90
2016-01-14 00:17:14     [<ffffffff934d6710>] __sock_create+0xf0/0x1f0
2016-01-14 00:17:14     [<ffffffff930d95f8>] ? rcu_eqs_enter+0x68/0x90
2016-01-14 00:17:14     [<ffffffff934d6870>] sock_create+0x30/0x40
2016-01-14 00:17:14     [<ffffffff934d803c>] SyS_socketpair+0x6c/0x1d0
2016-01-14 00:17:14     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:14     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:14     INFO: task imap:26593 blocked for more than 120 
seconds.
2016-01-14 00:17:14     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:14     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:14     imap D ffff880675a5fdc8 0 26593 4498 0x00000000
2016-01-14 00:17:14     ffff880675a5fdc8 ffff88082d943aa0 
ffff8807abab0000 ffff8806b0702f00
2016-01-14 00:17:14     ffff880675a60000 ffff88042b1f9928 
ffff88042b1f9800 ffff88042bba4c18
2016-01-14 00:17:14     ffff8807abab0000 ffff880675a5fde8 
ffffffff9363c167 0000fddf00010800
2016-01-14 00:17:14     Call Trace:
2016-01-14 00:17:14     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:14     [<ffffffff932e3433>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-14 00:17:14     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:14     [<ffffffff932c5d96>] xfs_dir_fsync+0x86/0xf0
2016-01-14 00:17:14     [<ffffffff931f4a56>] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:14     [<ffffffff931f4aec>] vfs_fsync+0x1c/0x20
2016-01-14 00:17:13     [<ffffffff9363dfd5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-14 00:17:13     [<ffffffff9363e06b>] mutex_lock+0x1b/0x2f
2016-01-14 00:17:13     [<ffffffff932cab77>] 
xfs_reclaim_inodes_ag+0x307/0x3d0
2016-01-14 00:17:13     [<ffffffff931d8cf0>] ? spin_unlock+0x10/0x10
2016-01-14 00:17:13     [<ffffffff930d9d1f>] ? __call_rcu+0x3f/0x360
2016-01-14 00:17:13     [<ffffffff931df3c9>] ? inode_lru_isolate+0x89/0x1c0
2016-01-14 00:17:13     [<ffffffff93177e48>] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-14 00:17:13     [<ffffffff931df340>] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [<ffffffff931df340>] ? 
insert_inode_locked+0x180/0x180
2016-01-14 00:17:13     [<ffffffff932cb673>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-14 00:17:13     [<ffffffff932da169>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-14 00:17:13     [<ffffffff931c7313>] super_cache_scan+0x1a3/0x1b0
2016-01-14 00:17:13     [<ffffffff9315fb06>] shrink_slab+0x216/0x3d0
2016-01-14 00:17:13     [<ffffffff93164399>] shrink_zone+0x2b9/0x2d0
2016-01-14 00:17:13     [<ffffffff9316470a>] 
do_try_to_free_pages+0x16a/0x400
2016-01-14 00:17:13     [<ffffffff93164baa>] try_to_free_pages+0xda/0x160
2016-01-14 00:17:13     [<ffffffff93635ceb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-14 00:17:13     [<ffffffff931581e0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-14 00:17:13     [<ffffffff9319cc0c>] alloc_pages_current+0x9c/0x110
2016-01-14 00:17:13     [<ffffffff931a59cb>] new_slab+0x32b/0x450
2016-01-14 00:17:13     [<ffffffff936376f8>] __slab_alloc+0x2de/0x433
2016-01-14 00:17:13     [<ffffffff934da9a1>] ? 
sk_prot_alloc.isra.38+0x41/0x170
2016-01-14 00:17:13     [<ffffffff931a7f53>] kmem_cache_alloc+0x193/0x1f0
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:12     [<ffffffff932e354b>] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-14 00:17:12     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:12     [<ffffffff932c626f>] xfs_file_fsync+0x18f/0x1e0
2016-01-14 00:17:12     [<ffffffff931f4a56>] vfs_fsync_range+0x46/0xc0
2016-01-14 00:17:12     [<ffffffff931f4aec>] vfs_fsync+0x1c/0x20
2016-01-14 00:17:12     [<ffffffff931f4c58>] do_fsync+0x38/0x70
2016-01-14 00:17:12     [<ffffffff9314e243>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-14 00:17:12     [<ffffffff93013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-14 00:17:12     [<ffffffff931f4ed0>] SyS_fsync+0x10/0x20
2016-01-14 00:17:12     [<ffffffff936403ae>] system_call_fastpath+0x12/0x71
2016-01-14 00:17:12     INFO: task pop3-login:7921 blocked for more than 
120 seconds.
2016-01-14 00:17:12     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:12     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:12     pop3-login D ffff88082c5234f8 0 7921 4498 0x00000000
2016-01-14 00:17:12     ffff88082c5234f8 ffff88042dbdbaa0 
ffff880827443aa0 ffff88082c523568
2016-01-14 00:17:12     ffff88082c524000 ffff88082bb63b6c 
ffff880827443aa0 00000000ffffffff
2016-01-14 00:17:12     ffff88082bb63b70 ffff88082c523518 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:12     Call Trace:
2016-01-14 00:17:12     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:12     [<ffffffff9363c3de>] 
schedule_preempt_disabled+0xe/0x10
2016-01-14 00:17:11     [<ffffffff93099b7f>] flush_work+0xff/0x1d0
2016-01-14 00:17:11     [<ffffffff93095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-14 00:17:11     [<ffffffff932e4c3e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-14 00:17:11     [<ffffffff930e14bf>] ? 
try_to_del_timer_sync+0x4f/0x70
2016-01-14 00:17:11     [<ffffffff932e2feb>] _xfs_log_force+0x6b/0x280
2016-01-14 00:17:11     [<ffffffff932c37e0>] ? 
xfs_buf_delwri_submit_nowait+0x20/0x30
2016-01-14 00:17:11     [<ffffffff932e322a>] xfs_log_force+0x2a/0x90
2016-01-14 00:17:11     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [<ffffffff932edca3>] xfsaild+0x143/0x5f0
2016-01-14 00:17:11     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:11     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:11     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:11     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:11     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:11     INFO: task mysqld:7538 blocked for more than 120 
seconds.
2016-01-14 00:17:11     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:11     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:11     mysqld D ffff8803e708bd98 0 7538 3771 0x00000000
2016-01-14 00:17:11     ffff8803e708bd98 ffff88082d941d50 
ffff8803f1ebd7f0 ffff88042bba4c00
2016-01-14 00:17:11     ffff8803e708c000 ffff88042b1f9928 
ffff88042b1f9800 0000000000000000
2016-01-14 00:17:11     ffff8803f1ebd7f0 ffff8803e708bdb8 
ffffffff9363c167 ffff8803f1ebd7f0
2016-01-14 00:17:10     [<ffffffff9363bbd1>] ? __schedule+0x4f1/0x990
2016-01-14 00:17:10     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:10     [<ffffffff9309f1f9>] kthread+0xc9/0xe0
2016-01-14 00:17:10     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     [<ffffffff936407a2>] ret_from_fork+0x42/0x70
2016-01-14 00:17:10     [<ffffffff9309f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-14 00:17:10     INFO: task xfsaild/sdb1:2068 blocked for more 
than 120 seconds.
2016-01-14 00:17:10     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:10     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:10     xfsaild/sdb1 D ffff88042529fad8 0 2068 2 0x00000000
2016-01-14 00:17:10     ffff88042529fad8 ffff88082d948000 
ffff8800b94ad7f0 ffff88082a8401d0
2016-01-14 00:17:10     ffff8804252a0000 ffff88042529fc40 
7fffffffffffffff ffff8800b94ad7f0
2016-01-14 00:17:10     ffffffffffffffff ffff88042529faf8 
ffffffff9363c167 0000000000000000
2016-01-14 00:17:10     Call Trace:
2016-01-14 00:17:10     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:10     [<ffffffff9363ee7c>] schedule_timeout+0x1fc/0x260
2016-01-14 00:17:10     [<ffffffff93326f0c>] ? part_round_stats+0x5c/0x70
2016-01-14 00:17:10     [<ffffffff93239439>] ? kernfs_path_locked+0x49/0xa0
2016-01-14 00:17:10     [<ffffffff93324a22>] ? elv_rb_add+0x62/0x70
2016-01-14 00:17:10     [<ffffffff9363cc9f>] wait_for_completion+0x9f/0x170
2016-01-14 00:17:10     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     INFO: task xfsaild/sda3:598 blocked for more 
than 120 seconds.
2016-01-14 00:17:09     Not tainted 4.1.14+63-ph #1
2016-01-14 00:17:09     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-14 00:17:09     xfsaild/sda3 D ffff88042a8d7d18 0 598 2 0x00000000
2016-01-14 00:17:09     ffff88042a8d7d18 ffff880102c93aa0 
ffff88042c248000 0000000000000282
2016-01-14 00:17:09     ffff88042a8d8000 ffff88042c248000 
0000000000000000 ffff88042b1f9800
2016-01-14 00:17:09     ffff88042bba4100 ffff88042a8d7d38 
ffffffff9363c167 ffff88042b1f9928
2016-01-14 00:17:09     Call Trace:
2016-01-14 00:17:09     [<ffffffff9363c167>] schedule+0x37/0x90
2016-01-14 00:17:09     [<ffffffff932e3109>] _xfs_log_force+0x189/0x280
2016-01-14 00:17:09     [<ffffffff930ad330>] ? try_to_wake_up+0x330/0x330
2016-01-14 00:17:09     [<ffffffff932e322a>] xfs_log_force+0x2a/0x90
2016-01-14 00:17:09     [<ffffffff932edb60>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-14 00:17:09     [<ffffffff932edca3>] xfsaild+0x143/0x5f0


Nr 2:

2016-01-15 02:01:04     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:04     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:04     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:04     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:04     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:04     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:04     [<ffffffff8e2d1dac>] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:04     [<ffffffff8e2e122e>] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:04     [<ffffffff8e2d54d7>] xfs_ifree+0x27/0xd0
2016-01-15 02:01:04     [<ffffffff8e2ef776>] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:04     [<ffffffff8e2d5641>] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:04     [<ffffffff8e2d5846>] xfs_inactive+0x96/0x150
2016-01-15 02:01:04     [<ffffffff8e2da5cf>] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:04     [<ffffffff8e1de7b0>] evict+0xc0/0x1a0
2016-01-15 02:01:04     [<ffffffff8e1defdb>] iput+0x18b/0x1f0
2016-01-15 02:01:04     [<ffffffff8e1da678>] __dentry_kill+0x198/0x200
2016-01-15 02:01:04     [<ffffffff8e1dace0>] dput+0x1c0/0x220
2016-01-15 02:01:04     [<ffffffff8e1c5a24>] __fput+0x164/0x1e0
2016-01-15 02:01:04     [<ffffffff8e1c5aee>] ____fput+0xe/0x10
2016-01-15 02:01:04     [<ffffffff8e09d5ff>] task_work_run+0xbf/0x100
2016-01-15 02:01:04     [<ffffffff8e003bf8>] do_notify_resume+0x88/0xa0
2016-01-15 02:01:04     [<ffffffff8e640d7c>] int_signal+0x12/0x17
2016-01-15 02:01:03     [<ffffffff8e2d5846>] xfs_inactive+0x96/0x150
2016-01-15 02:01:03     [<ffffffff8e2da5cf>] xfs_fs_evict_inode+0x6f/0xb0
2016-01-15 02:01:03     [<ffffffff8e1de7b0>] evict+0xc0/0x1a0
2016-01-15 02:01:03     [<ffffffff8e1defdb>] iput+0x18b/0x1f0
2016-01-15 02:01:03     [<ffffffff8e1da678>] __dentry_kill+0x198/0x200
2016-01-15 02:01:03     [<ffffffff8e1dace0>] dput+0x1c0/0x220
2016-01-15 02:01:03     [<ffffffff8e1c5a24>] __fput+0x164/0x1e0
2016-01-15 02:01:03     [<ffffffff8e1c5aee>] ____fput+0xe/0x10
2016-01-15 02:01:03     [<ffffffff8e09d5ff>] task_work_run+0xbf/0x100
2016-01-15 02:01:03     [<ffffffff8e003bf8>] do_notify_resume+0x88/0xa0
2016-01-15 02:01:03     [<ffffffff8e640d7c>] int_signal+0x12/0x17
2016-01-15 02:01:03     INFO: task php5.4:9661 blocked for more than 120 
seconds.
2016-01-15 02:01:03     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:03     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:03     php5.4 D ffff8800189278e8 0 9661 15202 0x00000004
2016-01-15 02:01:03     ffff8800189278e8 ffff880624309d50 
ffff88000b4a57f0 ffff880000000002
2016-01-15 02:01:03     ffff880018928000 7fffffffffffffff 
ffff88000b4a57f0 0000000000000002
2016-01-15 02:01:03     0000000000000000 ffff880018927908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:03     Call Trace:
2016-01-15 02:01:03     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:03     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:03     [<ffffffff8e32cee7>] ? blk_peek_request+0x87/0x290
2016-01-15 02:01:03     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:03     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:03     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:03     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:02     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:02     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:02     php5.4 D ffff8800017338e8 0 9657 15202 0x00000004
2016-01-15 02:01:02     ffff8800017338e8 ffff8806242f0000 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     ffff880001734000 7fffffffffffffff 
ffff8800010a1d50 0000000000000002
2016-01-15 02:01:02     0000000000000000 ffff880001733908 
ffffffff8e63c967 0000000000000000
2016-01-15 02:01:02     Call Trace:
2016-01-15 02:01:02     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:02     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:02     [<ffffffff8e0b25a8>] ? __enqueue_entity+0x78/0x80
2016-01-15 02:01:02     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:02     [<ffffffff8e16d478>] ? zone_statistics+0x98/0xc0
2016-01-15 02:01:02     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:02     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:02     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:02     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:02     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:02     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:02     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:02     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:01:02     [<ffffffff8e2d1dac>] xfs_iunlink_remove+0x4c/0x420
2016-01-15 02:01:02     [<ffffffff8e2e122e>] ? 
xlog_grant_head_check+0x4e/0xf0
2016-01-15 02:01:02     [<ffffffff8e2d54d7>] xfs_ifree+0x27/0xd0
2016-01-15 02:01:02     [<ffffffff8e2ef776>] ? xfs_trans_ijoin+0x36/0x60
2016-01-15 02:01:02     [<ffffffff8e2d5641>] xfs_inactive_ifree+0xc1/0x230
2016-01-15 02:01:01     [<ffffffff8e2b2a23>] xfs_ialloc_read_agi+0x43/0xe0
2016-01-15 02:01:01     [<ffffffff8e2b3421>] xfs_dialloc+0xf1/0x2d0
2016-01-15 02:01:01     [<ffffffff8e2d35eb>] xfs_ialloc+0x5b/0x550
2016-01-15 02:01:01     [<ffffffff8e2df9b7>] ? kmem_zone_alloc+0x77/0x100
2016-01-15 02:01:01     [<ffffffff8e2f1c77>] ? 
xfs_trans_mod_dquot+0x47/0x190
2016-01-15 02:01:01     [<ffffffff8e2d3b4a>] xfs_dir_ialloc+0x6a/0x290
2016-01-15 02:01:01     [<ffffffff8e2d408b>] xfs_create+0x2bb/0x6e0
2016-01-15 02:01:01     [<ffffffff8e00d775>] ? native_sched_clock+0x35/0x80
2016-01-15 02:01:01     [<ffffffff8e21cab1>] ? get_cached_acl+0x21/0x70
2016-01-15 02:01:01     [<ffffffff8e2d09b3>] xfs_generic_create+0xd3/0x2b0
2016-01-15 02:01:01     [<ffffffff8e1dc4ed>] ? __d_lookup+0x11d/0x160
2016-01-15 02:01:01     [<ffffffff8e2d0bc4>] xfs_vn_mknod+0x14/0x20
2016-01-15 02:01:01     [<ffffffff8e2d0c03>] xfs_vn_create+0x13/0x20
2016-01-15 02:01:01     [<ffffffff8e1cf307>] vfs_create+0x77/0xc0
2016-01-15 02:01:01     [<ffffffff8e1d076d>] do_last.isra.57+0x73d/0xd30
2016-01-15 02:01:01     [<ffffffff8e1d1084>] ? path_init+0xc4/0x430
2016-01-15 02:01:01     [<ffffffff8e1d2012>] path_openat+0x92/0x5a0
2016-01-15 02:01:01     [<ffffffff8e1d3f78>] ? user_path_at_empty+0x68/0xa0
2016-01-15 02:01:01     [<ffffffff8e1d4063>] do_filp_open+0x43/0xa0
2016-01-15 02:01:01     [<ffffffff8e1e0f24>] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:01:01     [<ffffffff8e1c2dec>] do_sys_open+0x13c/0x230
2016-01-15 02:01:01     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:01:01     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:01:01     [<ffffffff8e1c2f02>] SyS_open+0x22/0x30
2016-01-15 02:01:01     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:01:01     INFO: task php5.4:9657 blocked for more than 120 
seconds.
2016-01-15 02:01:00     INFO: task php5.2:6976 blocked for more than 120 
seconds.
2016-01-15 02:01:00     Not tainted 4.1.15+68-ph #1
2016-01-15 02:01:00     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:01:00     php5.2 D ffff88015bd87568 0 6976 15202 0x00000004
2016-01-15 02:01:00     ffff88015bd87568 ffff8806242c0000 
ffff8801938e0000 ffff8800374ded00
2016-01-15 02:01:00     ffff88015bd88000 7fffffffffffffff 
ffff8801938e0000 0000000000000002
2016-01-15 02:01:00     0000000000000000 ffff88015bd87588 
ffffffff8e63c967 00000000000001ed
2016-01-15 02:01:00     Call Trace:
2016-01-15 02:01:00     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:01:00     [<ffffffff8e63f68c>] schedule_timeout+0x1fc/0x260
2016-01-15 02:01:00     [<ffffffff8e0b873c>] ? dequeue_entity+0x13c/0x580
2016-01-15 02:01:00     [<ffffffff8e63e93e>] __down_common+0xaf/0x118
2016-01-15 02:01:00     [<ffffffff8e0b1700>] ? 
arch_vtime_task_switch+0x40/0x90
2016-01-15 02:01:00     [<ffffffff8e2c1f04>] ? _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [<ffffffff8e63ea1a>] __down+0x1d/0x1f
2016-01-15 02:01:00     [<ffffffff8e0c9831>] down+0x41/0x50
2016-01-15 02:01:00     [<ffffffff8e2c1d00>] xfs_buf_lock+0x40/0xe0
2016-01-15 02:01:00     [<ffffffff8e2c1f04>] _xfs_buf_find+0x164/0x2a0
2016-01-15 02:01:00     [<ffffffff8e2c20f5>] xfs_buf_get_map+0x35/0x180
2016-01-15 02:01:00     [<ffffffff8e2c2f77>] xfs_buf_read_map+0x37/0x110
2016-01-15 02:01:00     [<ffffffff8e2eef29>] 
xfs_trans_read_buf_map+0xf9/0x280
2016-01-15 02:01:00     [<ffffffff8e2b2985>] xfs_read_agi+0x95/0xf0
2016-01-15 02:00:59     INFO: task kworker/4:1H:6823 blocked for more 
than 120 seconds.
2016-01-15 02:00:59     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:59     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:59     kworker/4:1H D ffff880302813c88 0 6823 2 0x00000000
2016-01-15 02:00:59     Workqueue: xfs-log/sda3 xfs_log_worker
2016-01-15 02:00:59     ffff880302813c88 ffff8806243ed7f0 
ffff8800038b0000 ffff880621729400
2016-01-15 02:00:59     ffff880302814000 ffff8800bb179128 
ffff8800bb179000 0000000000000000
2016-01-15 02:00:59     ffff8800038b0000 ffff880302813ca8 
ffffffff8e63c967 ffff8800038b0000
2016-01-15 02:00:59     Call Trace:
2016-01-15 02:00:59     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:59     [<ffffffff8e2e359b>] _xfs_log_force_lsn+0x26b/0x2c0
2016-01-15 02:00:59     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:59     [<ffffffff8e2df093>] xfs_trans_commit+0x1e3/0x220
2016-01-15 02:00:59     [<ffffffff8e2b8b60>] xfs_sync_sb+0x60/0xa0
2016-01-15 02:00:59     [<ffffffff8e2e1bb4>] ? 
xfs_log_need_covered+0x74/0x100
2016-01-15 02:00:59     [<ffffffff8e2e332d>] xfs_log_worker+0x4d/0x50
2016-01-15 02:00:59     [<ffffffff8e0986bf>] process_one_work+0x14f/0x410
2016-01-15 02:00:59     [<ffffffff8e098ad9>] worker_thread+0x119/0x510
2016-01-15 02:00:59     [<ffffffff8e63c3d1>] ? __schedule+0x4f1/0x990
2016-01-15 02:00:59     [<ffffffff8e0989c0>] ? 
process_scheduled_works+0x40/0x40
2016-01-15 02:00:59     [<ffffffff8e09f1f9>] kthread+0xc9/0xe0
2016-01-15 02:00:59     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:59     [<ffffffff8e640fa2>] ret_from_fork+0x42/0x70
2016-01-15 02:00:59     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:58     [<ffffffff8e1df3d9>] ? inode_lru_isolate+0x89/0x1c0
2016-01-15 02:00:58     [<ffffffff8e177e58>] ? 
__list_lru_walk_one.isra.6+0x158/0x170
2016-01-15 02:00:58     [<ffffffff8e1df350>] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [<ffffffff8e1df350>] ? 
insert_inode_locked+0x180/0x180
2016-01-15 02:00:58     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:58     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:58     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:58     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:58     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:58     [<ffffffff8e1646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:58     [<ffffffff8e164b8a>] try_to_free_pages+0xda/0x160
2016-01-15 02:00:58     [<ffffffff8e6364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:58     [<ffffffff8e1581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:58     [<ffffffff8e158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:58     [<ffffffff8e07e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:58     [<ffffffff8e07e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:58     [<ffffffff8e1a7f63>] ? kmem_cache_alloc+0x193/0x1f0
2016-01-15 02:00:58     [<ffffffff8e1190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:58     [<ffffffff8e0b0f99>] ? account_user_time+0x99/0xb0
2016-01-15 02:00:58     [<ffffffff8e08014c>] do_fork+0xbc/0x350
2016-01-15 02:00:58     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:58     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:58     [<ffffffff8e080466>] SyS_clone+0x16/0x20
2016-01-15 02:00:58     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:57     [<ffffffff8e013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:57     [<ffffffff8e1f4ee0>] SyS_fsync+0x10/0x20
2016-01-15 02:00:57     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:57     INFO: task auth:9631 blocked for more than 120 
seconds.
2016-01-15 02:00:57     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:57     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:57     auth D ffff8809eea33588 0 9631 9616 0x00000004
2016-01-15 02:00:57     ffff8809eea33588 ffff8806242c1d50 
ffff880b97f19d50 ffff8809eea33568
2016-01-15 02:00:57     ffff8809eea34000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:57     ffff880b97f19d50 ffff8809eea335a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:57     Call Trace:
2016-01-15 02:00:57     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:57     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:57     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:57     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:57     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:57     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:57     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:57     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:57     [<ffffffff8e0d8cb6>] ? 
__call_rcu_nocb_enqueue+0xe6/0xf0
2016-01-15 02:00:56     [<ffffffff8e2c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [<ffffffff8e1f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [<ffffffff8e1f4afc>] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [<ffffffff8e1f4c68>] do_fsync+0x38/0x70
2016-01-15 02:00:56     [<ffffffff8e14e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-15 02:00:56     [<ffffffff8e013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-15 02:00:56     [<ffffffff8e1f4ee0>] SyS_fsync+0x10/0x20
2016-01-15 02:00:56     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:56     INFO: task mysqld:20600 blocked for more than 
120 seconds.
2016-01-15 02:00:56     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:56     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:56     mysqld D ffff8809218ffd98 0 20600 5893 0x00000000
2016-01-15 02:00:56     ffff8809218ffd98 ffff8806242d8000 
ffff880123ef57f0 0000000000000000
2016-01-15 02:00:56     ffff880921900000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:56     ffff880123ef57f0 ffff8809218ffdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:56     Call Trace:
2016-01-15 02:00:56     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:56     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:56     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:56     [<ffffffff8e2c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-15 02:00:56     [<ffffffff8e1f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-15 02:00:56     [<ffffffff8e1f4afc>] vfs_fsync+0x1c/0x20
2016-01-15 02:00:56     [<ffffffff8e1f4c68>] do_fsync+0x38/0x70
2016-01-15 02:00:55     [<ffffffff8e1646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-15 02:00:55     [<ffffffff8e164b8a>] try_to_free_pages+0xda/0x160
2016-01-15 02:00:55     [<ffffffff8e6364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-15 02:00:55     [<ffffffff8e1581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-15 02:00:55     [<ffffffff8e158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-15 02:00:55     [<ffffffff8e07e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-15 02:00:55     [<ffffffff8e07e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-15 02:00:55     [<ffffffff8e1190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-15 02:00:55     [<ffffffff8e0b0f99>] ? account_user_time+0x99/0xb0
2016-01-15 02:00:55     [<ffffffff8e08014c>] do_fork+0xbc/0x350
2016-01-15 02:00:55     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:55     [<ffffffff8e080466>] SyS_clone+0x16/0x20
2016-01-15 02:00:55     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:55     INFO: task mysqld:7069 blocked for more than 120 
seconds.
2016-01-15 02:00:55     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:55     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:55     mysqld D ffff880f0abbbd98 0 7069 5893 0x00000000
2016-01-15 02:00:55     ffff880f0abbbd98 ffff8806242c0000 
ffff880f0ab53aa0 0000000000000000
2016-01-15 02:00:55     ffff880f0abbc000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:55     ffff880f0ab53aa0 ffff880f0abbbdb8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:55     Call Trace:
2016-01-15 02:00:55     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:55     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:55     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [<ffffffff8e640bae>] system_call_fastpath+0x12/0x71
2016-01-15 02:00:54     INFO: task mysqld:6683 blocked for more than 120 
seconds.
2016-01-15 02:00:54     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:54     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:54     mysqld D ffff880f0b2f3588 0 6683 5893 0x00000000
2016-01-15 02:00:54     ffff880f0b2f3588 ffff8806242dbaa0 
ffff880f09913aa0 000000000000254e
2016-01-15 02:00:54     ffff880f0b2f4000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:54     ffff880f09913aa0 ffff880f0b2f35a8 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:54     Call Trace:
2016-01-15 02:00:54     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:54     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:54     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:54     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:54     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:54     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:54     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:54     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:54     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:54     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:54     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:54     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:54     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:53     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:53     mysqld D ffff88064549bbc8 0 21561 5534 0x00000000
2016-01-15 02:00:53     ffff88064549bbc8 ffff88062426d7f0 
ffff88000b4cbaa0 0000000000000040
2016-01-15 02:00:53     ffff88064549c000 ffff880f1631b674 
ffff88000b4cbaa0 00000000ffffffff
2016-01-15 02:00:53     ffff880f1631b678 ffff88064549bbe8 
ffffffff8e63c967 00000000000282c2
2016-01-15 02:00:53     Call Trace:
2016-01-15 02:00:53     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:53     [<ffffffff8e63cbde>] 
schedule_preempt_disabled+0xe/0x10
2016-01-15 02:00:53     [<ffffffff8e63e7e5>] 
__mutex_lock_slowpath+0x95/0x110
2016-01-15 02:00:53     [<ffffffff8e63e87b>] mutex_lock+0x1b/0x2f
2016-01-15 02:00:53     [<ffffffff8e1d028b>] do_last.isra.57+0x25b/0xd30
2016-01-15 02:00:53     [<ffffffff8e1d1084>] ? path_init+0xc4/0x430
2016-01-15 02:00:53     [<ffffffff8e1d2012>] path_openat+0x92/0x5a0
2016-01-15 02:00:53     [<ffffffff8e17d817>] ? handle_mm_fault+0xb47/0x1860
2016-01-15 02:00:53     [<ffffffff8e1d4063>] do_filp_open+0x43/0xa0
2016-01-15 02:00:53     [<ffffffff8e1e0f24>] ? __alloc_fd+0x134/0x1b0
2016-01-15 02:00:53     [<ffffffff8e1c2dec>] do_sys_open+0x13c/0x230
2016-01-15 02:00:53     [<ffffffff8e0134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-15 02:00:53     [<ffffffff8e1c2f02>] SyS_open+0x22/0x30
2016-01-15 02:00:52     [<ffffffff8e0c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-15 02:00:52     [<ffffffff8e2ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:52     [<ffffffff8e2cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-15 02:00:52     [<ffffffff8e2cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-15 02:00:52     [<ffffffff8e2da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-15 02:00:52     [<ffffffff8e1c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-15 02:00:52     [<ffffffff8e15fae6>] shrink_slab+0x216/0x3d0
2016-01-15 02:00:52     [<ffffffff8e164379>] shrink_zone+0x2b9/0x2d0
2016-01-15 02:00:52     [<ffffffff8e165287>] kswapd+0x527/0x980
2016-01-15 02:00:52     [<ffffffff8e164d60>] ? 
mem_cgroup_shrink_node_zone+0x150/0x150
2016-01-15 02:00:52     [<ffffffff8e09f1f9>] kthread+0xc9/0xe0
2016-01-15 02:00:52     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     [<ffffffff8e640fa2>] ret_from_fork+0x42/0x70
2016-01-15 02:00:52     [<ffffffff8e09f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-15 02:00:52     INFO: task mysqld:21561 blocked for more than 
120 seconds.
2016-01-15 02:00:52     Not tainted 4.1.15+68-ph #1
2016-01-15 02:00:51     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-15 02:00:51     kswapd1 D ffff880622e87848 0 195 2 0x00000000
2016-01-15 02:00:51     ffff880622e87848 ffff8806242d8000 
ffff880622e33aa0 0000000000000000
2016-01-15 02:00:51     ffff880622e88000 ffff8800bb179128 
ffff8800bb179000 ffff880621729418
2016-01-15 02:00:51     ffff880622e33aa0 ffff880622e87868 
ffffffff8e63c967 0002577b00042c00
2016-01-15 02:00:51     Call Trace:
2016-01-15 02:00:51     [<ffffffff8e63c967>] schedule+0x37/0x90
2016-01-15 02:00:51     [<ffffffff8e2e3483>] _xfs_log_force_lsn+0x153/0x2c0
2016-01-15 02:00:51     [<ffffffff8e0ad340>] ? try_to_wake_up+0x340/0x340
2016-01-15 02:00:51     [<ffffffff8e2e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-15 02:00:51     [<ffffffff8e2ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-15 02:00:51     [<ffffffff8e2d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-15 02:00:50     INFO: task kswapd1:195 blocked for more than 120 
seconds.
2016-01-15 02:00:50     Not tainted 4.1.15+68-ph #1

Nr 3:
2016-01-12 03:18:50     [<ffffffffb107e61b>] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:50     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:50     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:50     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:50     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:50     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:50     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:50     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:49     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:49     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:49     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:49     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:49     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:49     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:49     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:49     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:49     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:49     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:49     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:49     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:49     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:49     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:49     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:49     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:49     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:49     [<ffffffffb104dca7>] ? do_page_fault+0x37/0x90
2016-01-12 03:18:49     [<ffffffffb11a794a>] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:48     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:48     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:48     sshd D ffff880859e6f198 0 6171 1 0x00000000
2016-01-12 03:18:48     ffff880859e6f198 ffff88085c2e1d50 
ffff880855d53aa0 ffff880859e6f248
2016-01-12 03:18:48     ffff880859e70000 ffff880859e6f300 
7fffffffffffffff ffff880855d53aa0
2016-01-12 03:18:48     ffffffffffffffff ffff880859e6f1b8 
ffffffffb163c967 ffff880859e6f298
2016-01-12 03:18:48     Call Trace:
2016-01-12 03:18:48     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:48     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:48     [<ffffffffb13506e5>] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:48     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:48     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:48     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:48     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:48     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:48     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:48     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:48     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:48     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:47     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:47     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:47     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:47     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:47     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:47     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:47     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:47     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:47     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:47     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:47     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:47     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:47     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:47     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:47     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:47     [<ffffffffb108044c>] sys_vfork+0x1c/0x20
2016-01-12 03:18:47     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:47     INFO: task sshd:6171 blocked for more than 120 
seconds.
2016-01-12 03:18:46     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:46     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:46     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:46     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:46     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:46     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:46     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:46     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:46     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:46     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:46     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:46     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:46     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:46     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:46     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:46     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:46     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:45     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:45     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:45     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:45     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:45     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:45     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:45     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:45     INFO: task qmail-rspawn:6145 blocked for more 
than 120 seconds.
2016-01-12 03:18:45     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:45     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:45     qmail-rspawn D ffff88085af93348 0 6145 6125 
0x00000002
2016-01-12 03:18:45     ffff88085af93348 ffff88085c300000 
ffff8808554c9d50 0000000000000000
2016-01-12 03:18:45     ffff88085af94000 ffff88085af934b0 
7fffffffffffffff ffff8808554c9d50
2016-01-12 03:18:45     ffffffffffffffff ffff88085af93368 
ffffffffb163c967 ffff88085af933e8
2016-01-12 03:18:45     Call Trace:
2016-01-12 03:18:45     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:45     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:45     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:44     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:44     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:44     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:44     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:44     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:44     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:44     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:44     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:44     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:44     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:44     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:44     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:44     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:44     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:44     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:44     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:44     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:44     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:43     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:43     INFO: task zabbix_agentd:6130 blocked for more 
than 120 seconds.
2016-01-12 03:18:43     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:43     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:43     zabbix_agentd D ffff881053bc7348 0 6130 6118 
0x00000004
2016-01-12 03:18:43     ffff881053bc7348 ffff88085c300000 
ffff88105a4d9d50 ffff881053bc73d8
2016-01-12 03:18:43     ffff881053bc8000 ffff881053bc74b0 
7fffffffffffffff ffff88105a4d9d50
2016-01-12 03:18:43     ffffffffffffffff ffff881053bc7368 
ffffffffb163c967 ffff881053bc7368
2016-01-12 03:18:43     Call Trace:
2016-01-12 03:18:43     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:43     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:43     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:43     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:43     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:43     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:43     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:43     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:43     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:42     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:42     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:42     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:42     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:42     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:42     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:42     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:42     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:42     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:42     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:42     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:42     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:42     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:42     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:42     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:42     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:42     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:42     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:41     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:41     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:41     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:41     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:41     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:41     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:41     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:41     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:41     [<ffffffffb1326f7c>] ? part_round_stats+0x5c/0x70
2016-01-12 03:18:41     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:41     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:41     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:41     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:41     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:41     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:41     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:41     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:40     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:40     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:40     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:40     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:40     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:40     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:40     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:40     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:40     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:40     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:40     INFO: task zabbix_agentd:6129 blocked for more 
than 120 seconds.
2016-01-12 03:18:40     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:40     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:40     zabbix_agentd D ffff881053bd7348 0 6129 6118 
0x00000006
2016-01-12 03:18:40     ffff881053bd7348 ffff88085c2e1d50 
ffff88105340d7f0 ffffea0017e65180
2016-01-12 03:18:40     ffff881053bd8000 ffff881053bd74b0 
7fffffffffffffff ffff88105340d7f0
2016-01-12 03:18:40     ffffffffffffffff ffff881053bd7368 
ffffffffb163c967 ffff88085fa59650
2016-01-12 03:18:40     Call Trace:
2016-01-12 03:18:40     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:39     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:39     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:39     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:39     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:39     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:39     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:39     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:39     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:39     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:39     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:39     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:39     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:39     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:39     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:39     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:39     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:39     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:38     INFO: task zabbix_agentd:6128 blocked for more 
than 120 seconds.
2016-01-12 03:18:38     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:38     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:38     zabbix_agentd D ffff88105341f348 0 6128 6118 
0x00000004
2016-01-12 03:18:38     ffff88105341f348 ffff88085c2e57f0 
ffff880858739d50 ffffea00009e2100
2016-01-12 03:18:38     ffff881053420000 ffff88105341f4b0 
7fffffffffffffff ffff880858739d50
2016-01-12 03:18:38     ffffffffffffffff ffff88105341f368 
ffffffffb163c967 ffff88085fad9650
2016-01-12 03:18:38     Call Trace:
2016-01-12 03:18:38     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:38     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:38     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:38     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:38     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:38     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:38     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:38     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:38     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:38     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:38     [<ffffffffb11a659c>] ? __free_slab+0xcc/0x1c0
2016-01-12 03:18:37     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:37     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:37     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:37     [<ffffffffb14d4b8d>] ? sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [<ffffffffb10a98b6>] ? 
ttwu_do_activate.constprop.91+0x66/0x70
2016-01-12 03:18:37     [<ffffffffb11d6f71>] ? set_fd_set+0x21/0x30
2016-01-12 03:18:37     [<ffffffffb11d7f42>] ? core_sys_select+0x242/0x2f0
2016-01-12 03:18:37     [<ffffffffb11a7f63>] kmem_cache_alloc+0x193/0x1f0
2016-01-12 03:18:37     [<ffffffffb14d4b8d>] sock_alloc_inode+0x1d/0xc0
2016-01-12 03:18:37     [<ffffffffb11ddab6>] alloc_inode+0x26/0xa0
2016-01-12 03:18:37     [<ffffffffb11df84a>] new_inode_pseudo+0x1a/0x70
2016-01-12 03:18:37     [<ffffffffb14d4e0e>] sock_alloc+0x1e/0x80
2016-01-12 03:18:37     [<ffffffffb14d69ac>] SYSC_accept4+0x7c/0x210
2016-01-12 03:18:37     [<ffffffffb10b159d>] ? vtime_account_user+0x5d/0x70
2016-01-12 03:18:37     [<ffffffffb114e243>] ? 
context_tracking_user_exit+0x13/0x20
2016-01-12 03:18:37     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:37     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:37     [<ffffffffb14d8240>] SyS_accept+0x10/0x20
2016-01-12 03:18:37     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:36     [<ffffffffb1156957>] ? __free_pages+0x27/0x30
2016-01-12 03:18:36     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:36     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:36     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:36     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:36     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:36     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:36     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:36     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:36     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:36     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:36     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:36     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:36     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:36     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:36     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:36     [<ffffffffb11d7220>] ? __pollwait+0xf0/0xf0
2016-01-12 03:18:36     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:35     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:35     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:35     INFO: task nrpe:6076 blocked for more than 120 
seconds.
2016-01-12 03:18:35     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:35     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:35     nrpe D ffff880853e6b1a8 0 6076 1 0x00000002
2016-01-12 03:18:35     ffff880853e6b1a8 ffffffffb1c134a0 
ffff8808554e57f0 ffff88105f5d6240
2016-01-12 03:18:35     ffff880853e6c000 ffff880853e6b310 
7fffffffffffffff ffff8808554e57f0
2016-01-12 03:18:35     ffffffffffffffff ffff880853e6b1c8 
ffffffffb163c967 0000000000000003
2016-01-12 03:18:35     Call Trace:
2016-01-12 03:18:35     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:35     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:35     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:35     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:35     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:35     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:35     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:35     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:34     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:34     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:34     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:34     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:34     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:34     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:34     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:34     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:34     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:34     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:34     [<ffffffffb1158235>] 
alloc_kmem_pages_node+0x65/0x130
2016-01-12 03:18:34     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:34     [<ffffffffb107e639>] 
copy_process.part.37+0x109/0x1a80
2016-01-12 03:18:34     [<ffffffffb11c8910>] ? cp_new_stat+0x140/0x160
2016-01-12 03:18:34     [<ffffffffb11190cc>] ? 
acct_account_cputime+0x1c/0x20
2016-01-12 03:18:34     [<ffffffffb10b0f99>] ? account_user_time+0x99/0xb0
2016-01-12 03:18:34     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:34     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:33     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:33     [<ffffffffb13506e5>] ? cfq_set_request+0xa5/0x4c0
2016-01-12 03:18:33     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:33     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:33     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:33     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:33     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:33     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:33     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:33     [<ffffffffb132cb37>] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:33     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:33     [<ffffffffb14722b0>] ? 
scsi_prep_state_check.isra.22+0x80/0x80
2016-01-12 03:18:33     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:33     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:33     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:33     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:33     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:32     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:32     [<ffffffffb12c62bf>] xfs_file_fsync+0x18f/0x1e0
2016-01-12 03:18:32     [<ffffffffb11f4a66>] vfs_fsync_range+0x46/0xc0
2016-01-12 03:18:32     [<ffffffffb11f4afc>] vfs_fsync+0x1c/0x20
2016-01-12 03:18:32     [<ffffffffb11f4c68>] do_fsync+0x38/0x70
2016-01-12 03:18:32     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:32     [<ffffffffb1013805>] ? syscall_trace_leave+0x95/0xf0
2016-01-12 03:18:32     [<ffffffffb11f4ee0>] SyS_fsync+0x10/0x20
2016-01-12 03:18:32     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:32     INFO: task cron:6006 blocked for more than 120 
seconds.
2016-01-12 03:18:32     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:32     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:32     cron D ffff881053b57348 0 6006 1 0x00000000
2016-01-12 03:18:32     ffff881053b57348 ffff880858738000 
ffff88105a8057f0 ffff8810595db080
2016-01-12 03:18:32     ffff881053b58000 ffff881053b574b0 
7fffffffffffffff ffff88105a8057f0
2016-01-12 03:18:32     ffffffffffffffff ffff881053b57368 
ffffffffb163c967 ffff880000055000
2016-01-12 03:18:32     Call Trace:
2016-01-12 03:18:32     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:31     [<ffffffffb1640bae>] system_call_fastpath+0x12/0x71
2016-01-12 03:18:31     INFO: task mysqld:6762 blocked for more than 120 
seconds.
2016-01-12 03:18:31     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:31     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:31     mysqld D ffff880843887b58 0 6762 5590 0x00000000
2016-01-12 03:18:31     ffff880843887b58 ffff88085c301d50 
ffff88084385baa0 ffff880572ac09d0
2016-01-12 03:18:31     ffff880843888000 ffff880843887cc0 
7fffffffffffffff ffff88084385baa0
2016-01-12 03:18:31     ffffffffffffffff ffff880843887b78 
ffffffffb163c967 0000000000000100
2016-01-12 03:18:31     Call Trace:
2016-01-12 03:18:31     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:31     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:31     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:31     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:31     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:31     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:31     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:31     [<ffffffffb12bb940>] ? 
xfs_submit_ioend_bio.isra.14+0x40/0x40
2016-01-12 03:18:31     [<ffffffffb1639f03>] ? 
xfs_file_dio_aio_write+0x2a4/0x2f8
2016-01-12 03:18:30     [<ffffffffb1164379>] shrink_zone+0x2b9/0x2d0
2016-01-12 03:18:30     [<ffffffffb11646ea>] 
do_try_to_free_pages+0x16a/0x400
2016-01-12 03:18:30     [<ffffffffb100268b>] ? __switch_to+0x1eb/0x590
2016-01-12 03:18:30     [<ffffffffb1164b8a>] try_to_free_pages+0xda/0x160
2016-01-12 03:18:30     [<ffffffffb16364eb>] 
__alloc_pages_slowpath+0x3ff/0x697
2016-01-12 03:18:30     [<ffffffffb11581c0>] 
__alloc_pages_nodemask+0x230/0x240
2016-01-12 03:18:30     [<ffffffffb119cc1c>] alloc_pages_current+0x9c/0x110
2016-01-12 03:18:30     [<ffffffffb11a59db>] new_slab+0x32b/0x450
2016-01-12 03:18:30     [<ffffffffb1637ef8>] __slab_alloc+0x2de/0x433
2016-01-12 03:18:30     [<ffffffffb107e61b>] ? 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [<ffffffffb11a794a>] 
kmem_cache_alloc_node+0x8a/0x240
2016-01-12 03:18:30     [<ffffffffb107e61b>] 
copy_process.part.37+0xeb/0x1a80
2016-01-12 03:18:30     [<ffffffffb11117a2>] ? make_kgid+0x12/0x20
2016-01-12 03:18:30     [<ffffffffb11dd9f3>] ? inode_init_always+0xe3/0x180
2016-01-12 03:18:30     [<ffffffffb108014c>] do_fork+0xbc/0x350
2016-01-12 03:18:30     [<ffffffffb10134b8>] ? 
syscall_trace_enter_phase1+0xf8/0x160
2016-01-12 03:18:30     [<ffffffffb114e223>] ? 
context_tracking_user_enter+0x13/0x20
2016-01-12 03:18:30     [<ffffffffb1080466>] SyS_clone+0x16/0x20
2016-01-12 03:18:29     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:29     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:29     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:29     [<ffffffffb11520da>] ? mempool_alloc+0x5a/0x150
2016-01-12 03:18:29     [<ffffffffc02346be>] ? 
MR_BuildRaidContext+0x2fe/0x6d0 [megaraid_sas]
2016-01-12 03:18:29     [<ffffffffb132cb37>] ? 
blk_attempt_plug_merge+0xd7/0xf0
2016-01-12 03:18:29     [<ffffffffb12e3388>] _xfs_log_force_lsn+0x58/0x2c0
2016-01-12 03:18:29     [<ffffffffb12b4bd7>] ? 
xfs_iext_bno_to_ext+0x97/0x170
2016-01-12 03:18:29     [<ffffffffb12e362b>] xfs_log_force_lsn+0x3b/0xa0
2016-01-12 03:18:29     [<ffffffffb12ca660>] ? xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [<ffffffffb12d59ae>] xfs_iunpin_wait+0xae/0x140
2016-01-12 03:18:29     [<ffffffffb10c2bc0>] ? 
wake_atomic_t_function+0x40/0x40
2016-01-12 03:18:29     [<ffffffffb12ca660>] xfs_reclaim_inode+0xd0/0x330
2016-01-12 03:18:29     [<ffffffffb12cab37>] 
xfs_reclaim_inodes_ag+0x277/0x3d0
2016-01-12 03:18:29     [<ffffffffb12cb6c3>] xfs_reclaim_inodes_nr+0x33/0x40
2016-01-12 03:18:29     [<ffffffffb12da1b9>] 
xfs_fs_free_cached_objects+0x19/0x20
2016-01-12 03:18:29     [<ffffffffb11c7323>] super_cache_scan+0x1a3/0x1b0
2016-01-12 03:18:29     [<ffffffffb115fae6>] shrink_slab+0x216/0x3d0
2016-01-12 03:18:28     [<ffffffffb109f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:28     INFO: task proftpd:5592 blocked for more than 
120 seconds.
2016-01-12 03:18:28     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:28     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:28     proftpd D ffff88105487b198 0 5592 1 0x00000006
2016-01-12 03:18:28     ffff88105487b198 ffffffffb1c134a0 
ffff8808587b9d50 ffff88105487b1d8
2016-01-12 03:18:28     ffff88105487c000 ffff88105487b300 
7fffffffffffffff ffff8808587b9d50
2016-01-12 03:18:28     ffffffffffffffff ffff88105487b1b8 
ffffffffb163c967 ffff88105b9881d8
2016-01-12 03:18:28     Call Trace:
2016-01-12 03:18:28     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:28     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:28     [<ffffffffb1637af4>] ? 
get_partial_node.isra.64+0x1bf/0x1dc
2016-01-12 03:18:28     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:28     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:27     [<ffffffffb12b6c79>] ? xfs_imap_to_bp+0xc9/0xe0
2016-01-12 03:18:27     [<ffffffffb12e327a>] xfs_log_force+0x2a/0x90
2016-01-12 03:18:27     [<ffffffffb12d68c2>] xfs_iflush+0x202/0x210
2016-01-12 03:18:27     [<ffffffffb10c9872>] ? up+0x32/0x50
2016-01-12 03:18:27     [<ffffffffb12e723a>] xfs_inode_item_push+0xda/0x140
2016-01-12 03:18:27     [<ffffffffb12edbb0>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [<ffffffffb12ede7e>] xfsaild+0x2ce/0x5f0
2016-01-12 03:18:27     [<ffffffffb163c3d1>] ? __schedule+0x4f1/0x990
2016-01-12 03:18:27     [<ffffffffb12edbb0>] ? 
xfs_trans_ail_cursor_first+0x90/0x90
2016-01-12 03:18:27     [<ffffffffb109f1f9>] kthread+0xc9/0xe0
2016-01-12 03:18:27     [<ffffffffb109f130>] ? 
kthread_create_on_node+0x1a0/0x1a0
2016-01-12 03:18:27     [<ffffffffb1640fa2>] ret_from_fork+0x42/0x70
2016-01-12 03:18:26     [<ffffffffb10ad197>] ? try_to_wake_up+0x197/0x340
2016-01-12 03:18:26     [<ffffffffb163d4af>] wait_for_completion+0x9f/0x170
2016-01-12 03:18:26     [<ffffffffb10ad395>] ? wake_up_process+0x15/0x20
2016-01-12 03:18:26     [<ffffffffb10ad340>] ? try_to_wake_up+0x340/0x340
2016-01-12 03:18:26     [<ffffffffb1099b7f>] flush_work+0xff/0x1d0
2016-01-12 03:18:26     [<ffffffffb1095e70>] ? 
worker_detach_from_pool+0xd0/0xd0
2016-01-12 03:18:26     [<ffffffffb12e4c8e>] xlog_cil_force_lsn+0x7e/0x200
2016-01-12 03:18:26     [<ffffffffb12c2f77>] ? xfs_buf_read_map+0x37/0x110
2016-01-12 03:18:26     [<ffffffffb12e303b>] _xfs_log_force+0x6b/0x280
2016-01-12 03:18:25     ffff8810566e7a28 ffff88085c31d7f0 
ffff880859430000 000000000000137e
2016-01-12 03:18:25     ffff8810566e8000 ffff8810566e7b90 
7fffffffffffffff ffff880859430000
2016-01-12 03:18:25     ffffffffffffffff ffff8810566e7a48 
ffffffffb163c967 ffff88105b988060
2016-01-12 03:18:25     Call Trace:
2016-01-12 03:18:25     [<ffffffffb163c967>] schedule+0x37/0x90
2016-01-12 03:18:25     [<ffffffffb163f68c>] schedule_timeout+0x1fc/0x260
2016-01-12 03:18:25     [<ffffffffb100d7c9>] ? sched_clock+0x9/0x10
2016-01-12 03:18:25     [<ffffffffb10b0ae5>] ? sched_clock_cpu+0x85/0xc0
2016-01-12 03:18:24     INFO: task xfsaild/sda3:730 blocked for more 
than 120 seconds.
2016-01-12 03:18:24     Not tainted 4.1.15+68-ph #1
2016-01-12 03:18:24     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2016-01-12 03:18:24     xfsaild/sda3 D ffff8810566e7a28 0 730 2 0x00000000



I can give you many more if they may help.


Greets,
Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs deadlocks since upgrading to 4.1.15

[Dave Chinner]

On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> Hello,
> 
> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> call traces show xfs.
> 
> Attention all those are down top instead of top down (sorry for the
> wrong order).

Can you repost them with line wrapping turned off, so they are
readable without causing beeding eye syndrome?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: xfs deadlocks since upgrading to 4.1.15

[Dave Chinner]

On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> Hello,
> 
> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> call traces show xfs.
> 
> Attention all those are down top instead of top down (sorry for the
> wrong order).

Can you repost them with line wrapping turned off, so they are
readable without causing beeding eye syndrome?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe]

Hi Dave,

Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>> Hello,
>>
>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>> call traces show xfs.
>>
>> Attention all those are down top instead of top down (sorry for the
>> wrong order).
>
> Can you repost them with line wrapping turned off, so they are
> readable without causing beeding eye syndrome?
i'm sorry. i was a bit optimistic that someone can read that.

Here are the traces in correct order, without timestamps but at pastebin 
which seems to be easier at least with thunderbird as it has no easy to 
use option to disable wrapping.

Nr 1:
http://pastebin.com/raw/Fuxd86TP

Nr 2:
http://pastebin.com/raw/Dp7mEcKi

Nr 3:
http://pastebin.com/raw/BY3gUwzx

Greets,
Stefan


> Cheers,
>
> Dave.
>

Re: xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe]

Hi Dave,

Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>> Hello,
>>
>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>> call traces show xfs.
>>
>> Attention all those are down top instead of top down (sorry for the
>> wrong order).
>
> Can you repost them with line wrapping turned off, so they are
> readable without causing beeding eye syndrome?
i'm sorry. i was a bit optimistic that someone can read that.

Here are the traces in correct order, without timestamps but at pastebin 
which seems to be easier at least with thunderbird as it has no easy to 
use option to disable wrapping.

Nr 1:
http://pastebin.com/raw/Fuxd86TP

Nr 2:
http://pastebin.com/raw/Dp7mEcKi

Nr 3:
http://pastebin.com/raw/BY3gUwzx

Greets,
Stefan


> Cheers,
>
> Dave.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs deadlocks since upgrading to 4.1.15

[Dave Chinner]

On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
> Hi Dave,
> 
> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> >On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> >>Hello,
> >>
> >>since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> >>call traces show xfs.
> >>
> >>Attention all those are down top instead of top down (sorry for the
> >>wrong order).
> >
> >Can you repost them with line wrapping turned off, so they are
> >readable without causing beeding eye syndrome?
> i'm sorry. i was a bit optimistic that someone can read that.
> 
> Here are the traces in correct order, without timestamps but at
> pastebin which seems to be easier at least with thunderbird as it
> has no easy to use option to disable wrapping.
> 
> Nr 1:
> http://pastebin.com/raw/Fuxd86TP
> 
> Nr 2:
> http://pastebin.com/raw/Dp7mEcKi
> 
> Nr 3:
> http://pastebin.com/raw/BY3gUwzx

Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
fix it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: xfs deadlocks since upgrading to 4.1.15

[Dave Chinner]

On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
> Hi Dave,
> 
> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
> >On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
> >>Hello,
> >>
> >>since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
> >>call traces show xfs.
> >>
> >>Attention all those are down top instead of top down (sorry for the
> >>wrong order).
> >
> >Can you repost them with line wrapping turned off, so they are
> >readable without causing beeding eye syndrome?
> i'm sorry. i was a bit optimistic that someone can read that.
> 
> Here are the traces in correct order, without timestamps but at
> pastebin which seems to be easier at least with thunderbird as it
> has no easy to use option to disable wrapping.
> 
> Nr 1:
> http://pastebin.com/raw/Fuxd86TP
> 
> Nr 2:
> http://pastebin.com/raw/Dp7mEcKi
> 
> Nr 3:
> http://pastebin.com/raw/BY3gUwzx

Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
fix it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe - Profihost AG]

Am 17.01.2016 um 20:48 schrieb Dave Chinner:
> On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
>> Hi Dave,
>>
>> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
>>> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>>>> Hello,
>>>>
>>>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>>>> call traces show xfs.
>>>>
>>>> Attention all those are down top instead of top down (sorry for the
>>>> wrong order).
>>>
>>> Can you repost them with line wrapping turned off, so they are
>>> readable without causing beeding eye syndrome?
>> i'm sorry. i was a bit optimistic that someone can read that.
>>
>> Here are the traces in correct order, without timestamps but at
>> pastebin which seems to be easier at least with thunderbird as it
>> has no easy to use option to disable wrapping.
>>
>> Nr 1:
>> http://pastebin.com/raw/Fuxd86TP
>>
>> Nr 2:
>> http://pastebin.com/raw/Dp7mEcKi
>>
>> Nr 3:
>> http://pastebin.com/raw/BY3gUwzx
> 
> Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
> fix it.

Thx!

Greets,
Stefan

> Cheers,
> 
> Dave.
> 

Re: xfs deadlocks since upgrading to 4.1.15

[Stefan Priebe - Profihost AG]

Am 17.01.2016 um 20:48 schrieb Dave Chinner:
> On Sat, Jan 16, 2016 at 08:17:00PM +0100, Stefan Priebe wrote:
>> Hi Dave,
>>
>> Am 15.01.2016 um 23:55 schrieb Dave Chinner:
>>> On Fri, Jan 15, 2016 at 10:07:58AM +0100, Stefan Priebe wrote:
>>>> Hello,
>>>>
>>>> since we upgraded from 3.18 to 4.1.15 i've seen many deadlocks. Alle
>>>> call traces show xfs.
>>>>
>>>> Attention all those are down top instead of top down (sorry for the
>>>> wrong order).
>>>
>>> Can you repost them with line wrapping turned off, so they are
>>> readable without causing beeding eye syndrome?
>> i'm sorry. i was a bit optimistic that someone can read that.
>>
>> Here are the traces in correct order, without timestamps but at
>> pastebin which seems to be easier at least with thunderbird as it
>> has no easy to use option to disable wrapping.
>>
>> Nr 1:
>> http://pastebin.com/raw/Fuxd86TP
>>
>> Nr 2:
>> http://pastebin.com/raw/Dp7mEcKi
>>
>> Nr 3:
>> http://pastebin.com/raw/BY3gUwzx
> 
> Commit 7a29ac4 ("xfs: give all workqueues rescuer threads") should
> fix it.

Thx!

Greets,
Stefan

> Cheers,
> 
> Dave.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs