From: Sitsofe Wheeler <sitsofe@gmail.com>
To: "Darrick J. Wong" <darrick.wong@oracle.com>
Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org
Subject: Tasks blocking forever with XFS stack traces
Date: Tue, 5 Nov 2019 07:27:16 +0000 [thread overview]
Message-ID: <CALjAwxiuTYAVvGGUXLx6Bo-zNuW5+WXL=A8DqR5oD6D5tsKwng@mail.gmail.com> (raw)
Hi,
We have a system that has been seeing tasks with XFS calls in their
stacks. Once these tasks start hanging with uninterruptible sleep any
write I/O to the directory they were doing I/O to will also hang
forever. The I/O they doing is being done to a bind mounted directory
atop an XFS filesystem on top an MD device (the MD device seems to be
still functional and isn't offline). The kernel is fairly old but I
thought I'd post a stack in case anyone can describe this or has seen
it before:
kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for
more than 120 seconds.
kernel: [425684.110800] Tainted: G OE
4.15.0-64-generic #73-Ubuntu
kernel: [425684.111164] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [425684.111568] kworker/u162:0 D 0 58843 2 0x80000080
kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126)
kernel: [425684.111585] Call Trace:
kernel: [425684.111595] __schedule+0x24e/0x880
kernel: [425684.111664] ? xfs_map_blocks+0x82/0x250 [xfs]
kernel: [425684.111668] schedule+0x2c/0x80
kernel: [425684.111671] rwsem_down_read_failed+0xf0/0x160
kernel: [425684.111675] ? bitmap_startwrite+0x9f/0x1f0
kernel: [425684.111679] call_rwsem_down_read_failed+0x18/0x30
kernel: [425684.111682] ? call_rwsem_down_read_failed+0x18/0x30
kernel: [425684.111685] down_read+0x20/0x40
kernel: [425684.111736] xfs_ilock+0xd5/0x100 [xfs]
kernel: [425684.111782] xfs_map_blocks+0x82/0x250 [xfs]
kernel: [425684.111823] xfs_do_writepage+0x167/0x6a0 [xfs]
kernel: [425684.111830] ? clear_page_dirty_for_io+0x19f/0x1f0
kernel: [425684.111834] write_cache_pages+0x207/0x4e0
kernel: [425684.111869] ? xfs_vm_writepages+0xf0/0xf0 [xfs]
kernel: [425684.111875] ? submit_bio+0x73/0x140
kernel: [425684.111878] ? submit_bio+0x73/0x140
kernel: [425684.111911] ? xfs_setfilesize_trans_alloc.isra.13+0x3e/0x90 [xfs]
kernel: [425684.111944] xfs_vm_writepages+0xbe/0xf0 [xfs]
kernel: [425684.111949] do_writepages+0x4b/0xe0
kernel: [425684.111954] ? fprop_fraction_percpu+0x2f/0x80
kernel: [425684.111958] ? __wb_calc_thresh+0x3e/0x130
kernel: [425684.111963] __writeback_single_inode+0x45/0x350
kernel: [425684.111966] ? __writeback_single_inode+0x45/0x350
kernel: [425684.111970] writeback_sb_inodes+0x1e1/0x510
kernel: [425684.111975] __writeback_inodes_wb+0x67/0xb0
kernel: [425684.111979] wb_writeback+0x271/0x300
kernel: [425684.111983] wb_workfn+0x1bb/0x400
kernel: [425684.111986] ? wb_workfn+0x1bb/0x400
kernel: [425684.111992] process_one_work+0x1de/0x420
kernel: [425684.111996] worker_thread+0x32/0x410
kernel: [425684.111999] kthread+0x121/0x140
kernel: [425684.112003] ? process_one_work+0x420/0x420
kernel: [425684.112005] ? kthread_create_worker_on_cpu+0x70/0x70
kernel: [425684.112009] ret_from_fork+0x35/0x40
kernel: [425684.112024] INFO: task kworker/74:0:9623 blocked for more
than 120 seconds.
kernel: [425684.112461] Tainted: G OE
4.15.0-64-generic #73-Ubuntu
kernel: [425684.112925] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: [425684.113438] kworker/74:0 D 0 9623 2 0x80000080
kernel: [425684.113500] Workqueue: xfs-cil/md126 xlog_cil_push_work [xfs]
kernel: [425684.113502] Call Trace:
kernel: [425684.113508] __schedule+0x24e/0x880
kernel: [425684.113559] ? xlog_bdstrat+0x2b/0x60 [xfs]
kernel: [425684.113564] schedule+0x2c/0x80
kernel: [425684.113609] xlog_state_get_iclog_space+0x105/0x2d0 [xfs]
kernel: [425684.113614] ? wake_up_q+0x80/0x80
kernel: [425684.113656] xlog_write+0x163/0x6e0 [xfs]
kernel: [425684.113699] xlog_cil_push+0x2a7/0x410 [xfs]
kernel: [425684.113740] xlog_cil_push_work+0x15/0x20 [xfs]
kernel: [425684.113743] process_one_work+0x1de/0x420
kernel: [425684.113747] worker_thread+0x32/0x410
kernel: [425684.113750] kthread+0x121/0x140
kernel: [425684.113753] ? process_one_work+0x420/0x420
kernel: [425684.113756] ? kthread_create_worker_on_cpu+0x70/0x70
kernel: [425684.113759] ret_from_fork+0x35/0x40
Other directories on the same filesystem seem fine as do other XFS
filesystems on the same system.
--
Sitsofe | http://sucs.org/~sits/
next reply other threads:[~2019-11-05 7:27 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-05 7:27 Sitsofe Wheeler [this message]
2019-11-05 8:54 ` Tasks blocking forever with XFS stack traces Carlos Maiolino
2019-11-05 9:32 ` Sitsofe Wheeler
2019-11-05 10:36 ` Carlos Maiolino
2019-11-05 11:58 ` Carlos Maiolino
2019-11-05 14:12 ` Sitsofe Wheeler
2019-11-05 16:09 ` Carlos Maiolino
2019-11-07 0:12 ` Chris Murphy
2019-11-13 10:04 ` Sitsofe Wheeler
2020-12-23 8:45 ` Sitsofe Wheeler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALjAwxiuTYAVvGGUXLx6Bo-zNuW5+WXL=A8DqR5oD6D5tsKwng@mail.gmail.com' \
--to=sitsofe@gmail.com \
--cc=darrick.wong@oracle.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.