Tasks blocking forever with XFS stack traces

Tasks blocking forever with XFS stack traces

[Sitsofe Wheeler]

Hi,

We have a system that has been seeing tasks with XFS calls in their
stacks. Once these tasks start hanging with uninterruptible sleep any
write I/O to the directory they were doing I/O to will also hang
forever. The I/O they doing is being done to a bind mounted directory
atop an XFS filesystem on top an MD device (the MD device seems to be
still functional and isn't offline). The kernel is fairly old but I
thought I'd post a stack in case anyone can describe this or has seen
it before:

kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for
more than 120 seconds.
kernel: [425684.110800]       Tainted: G           OE
4.15.0-64-generic #73-Ubuntu
kernel: [425684.111164] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [425684.111568] kworker/u162:0  D    0 58843      2 0x80000080
kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126)
kernel: [425684.111585] Call Trace:
kernel: [425684.111595]  __schedule+0x24e/0x880
kernel: [425684.111664]  ? xfs_map_blocks+0x82/0x250 [xfs]
kernel: [425684.111668]  schedule+0x2c/0x80
kernel: [425684.111671]  rwsem_down_read_failed+0xf0/0x160
kernel: [425684.111675]  ? bitmap_startwrite+0x9f/0x1f0
kernel: [425684.111679]  call_rwsem_down_read_failed+0x18/0x30
kernel: [425684.111682]  ? call_rwsem_down_read_failed+0x18/0x30
kernel: [425684.111685]  down_read+0x20/0x40
kernel: [425684.111736]  xfs_ilock+0xd5/0x100 [xfs]
kernel: [425684.111782]  xfs_map_blocks+0x82/0x250 [xfs]
kernel: [425684.111823]  xfs_do_writepage+0x167/0x6a0 [xfs]
kernel: [425684.111830]  ? clear_page_dirty_for_io+0x19f/0x1f0
kernel: [425684.111834]  write_cache_pages+0x207/0x4e0
kernel: [425684.111869]  ? xfs_vm_writepages+0xf0/0xf0 [xfs]
kernel: [425684.111875]  ? submit_bio+0x73/0x140
kernel: [425684.111878]  ? submit_bio+0x73/0x140
kernel: [425684.111911]  ? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs]
kernel: [425684.111944]  xfs_vm_writepages+0xbe/0xf0 [xfs]
kernel: [425684.111949]  do_writepages+0x4b/0xe0
kernel: [425684.111954]  ? fprop_fraction_percpu+0x2f/0x80
kernel: [425684.111958]  ? __wb_calc_thresh+0x3e/0x130
kernel: [425684.111963]  __writeback_single_inode+0x45/0x350
kernel: [425684.111966]  ? __writeback_single_inode+0x45/0x350
kernel: [425684.111970]  writeback_sb_inodes+0x1e1/0x510
kernel: [425684.111975]  __writeback_inodes_wb+0x67/0xb0
kernel: [425684.111979]  wb_writeback+0x271/0x300
kernel: [425684.111983]  wb_workfn+0x1bb/0x400
kernel: [425684.111986]  ? wb_workfn+0x1bb/0x400
kernel: [425684.111992]  process_one_work+0x1de/0x420
kernel: [425684.111996]  worker_thread+0x32/0x410
kernel: [425684.111999]  kthread+0x121/0x140
kernel: [425684.112003]  ? process_one_work+0x420/0x420
kernel: [425684.112005]  ? kthread_create_worker_on_cpu+0x70/0x70
kernel: [425684.112009]  ret_from_fork+0x35/0x40
kernel: [425684.112024] INFO: task kworker/74:0:9623 blocked for more
than 120 seconds.
kernel: [425684.112461]       Tainted: G           OE
4.15.0-64-generic #73-Ubuntu
kernel: [425684.112925] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [425684.113438] kworker/74:0    D    0  9623      2 0x80000080
kernel: [425684.113500] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs]
kernel: [425684.113502] Call Trace:
kernel: [425684.113508]  __schedule+0x24e/0x880
kernel: [425684.113559]  ? xlog_bdstrat+0x2b/0x60 [xfs]
kernel: [425684.113564]  schedule+0x2c/0x80
kernel: [425684.113609]  xlog_state_get_iclog_space+0x105/0x2d0 [xfs]
kernel: [425684.113614]  ? wake_up_q+0x80/0x80
kernel: [425684.113656]  xlog_write+0x163/0x6e0 [xfs]
kernel: [425684.113699]  xlog_cil_push+0x2a7/0x410 [xfs]
kernel: [425684.113740]  xlog_cil_push_work+0x15/0x20 [xfs]
kernel: [425684.113743]  process_one_work+0x1de/0x420
kernel: [425684.113747]  worker_thread+0x32/0x410
kernel: [425684.113750]  kthread+0x121/0x140
kernel: [425684.113753]  ? process_one_work+0x420/0x420
kernel: [425684.113756]  ? kthread_create_worker_on_cpu+0x70/0x70
kernel: [425684.113759]  ret_from_fork+0x35/0x40

Other directories on the same filesystem seem fine as do other XFS
filesystems on the same system.

-- 
Sitsofe | http://sucs.org/~sits/