XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi XFS developers,

We are seeing the following issue with XFS on kernel 3.18.19.

We have XFS mounted over a raw disk. Disk was pulled out manually. There
were async writes on files that were errored like this

Mar 16 16:03:21 host0 kernel: [ 4635.645613] XFS (dm-29): metadata I/O
error: block 0x7442c20 ("xfs_buf_iodone_callbacks") error 5 numblks 8
Mar 16 16:03:21 host0 kernel: [ 4635.945044] XFS (dm-29): Detected failing
async write on buffer block 0x7442c20. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.945048] XFS (dm-29): Detected failing
async write on buffer block 0x7442c28. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.945050] XFS (dm-29): Detected failing
async write on buffer block 0x7442c30. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965056] XFS (dm-29): Detected failing
async write on buffer block 0x7443008. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965070] XFS (dm-29): Detected failing
async write on buffer block 0xae64230. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965075] XFS (dm-29): Detected failing
async write on buffer block 0xae64210. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965080] XFS (dm-29): Detected failing
async write on buffer block 0xae64228. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965085] XFS (dm-29): Detected failing
async write on buffer block 0xae64220. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4635.965090] XFS (dm-29): Detected failing
async write on buffer block 0xae64208. Retrying async write.
Mar 16 16:03:21 host0 kernel: [ 4636.005024] XFS (dm-29): Detected failing
async write on buffer block 0xe885828. Retrying async write.

And XFS hit metadata & Log IO errors that it decides to shutdown:

Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
old_flags=0x0 new_flags=0x2
Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
Detected.  Shutting down filesystem
Mar 16 16:03:22 host0 kernel: [ 4637.353202] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:03:22 host0 kernel: [ 4637.354037] XFS (dm-29): Please umount
the filesystem and rectify the problem(s)
Mar 16 16:03:22 host0 kernel: [ 4637.354704] Buffer I/O error on dev
dm-29, logical block 122146379, lost async page write

After this happens, constantly XFS prints log force failure every 30-secs

Mar 16 16:03:51 host0 kernel: [ 4665.695023] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:04:21 host0 kernel: [ 4695.775022] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:04:51 host0 kernel: [ 4725.855021] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:05:21 host0 kernel: [ 4755.935033] XFS (dm-29): xfs_log_force:
error -5 returned.

Later the drive was re-inserted back. After the drive was re-inserted, XFS
was attempted to be unmounted

Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
: umount(/sdisk/vol5b0, xfs)

But nothing happens except for the 30-secs xfs_log_force errors that keeps
repeating

Mar 16 16:16:53 host0 kernel: [ 5448.122129] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:16:53 host0 kernel: [ 5448.135107] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:17:23 host0 kernel: [ 5477.855028] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:17:53 host0 kernel: [ 5507.935034] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:18:23 host0 kernel: [ 5538.015020] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:18:53 host0 kernel: [ 5568.095031] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:19:23 host0 kernel: [ 5598.175025] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:19:53 host0 kernel: [ 5628.255023] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:20:23 host0 kernel: [ 5658.335027] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:20:54 host0 kernel: [ 5688.415028] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:21:24 host0 kernel: [ 5718.495021] XFS (dm-29): xfs_log_force:
error -5 returned.
Mar 16 16:21:54 host0 kernel: [ 5748.575024] XFS (dm-29): xfs_log_force:
error -5 returned.


& we run into a hung-task timeout like this

Mar 16 16:22:07 host0 kernel: [ 5762.085031] INFO: task controld:2557
blocked for more than 180 seconds.
Mar 16 16:22:07 host0 kernel: [ 5762.086224] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 16 16:22:07 host0 kernel: [ 5762.086897] controld      D
ffff88023fc13540     0  2557      1 0x00000000
Mar 16 16:22:07 host0 kernel: [ 5762.086901]  ffff880223a07d58
0000000000000086 ffff88023332d100 0000000000013540
Mar 16 16:22:07 host0 kernel: [ 5762.086903]  ffff880223a07fd8
0000000000013540 ffffffff81c1d4c0 ffff88023332d100
Mar 16 16:22:07 host0 kernel: [ 5762.086906]  ffff880210f5f980
ffff88020fdb1600 ffff880210f5f980 ffff88020fdb1640
Mar 16 16:22:07 host0 kernel: [ 5762.086908] Call Trace:
Mar 16 16:22:07 host0 kernel: [ 5762.086914]  [<ffffffff817136c9>]
schedule+0x29/0x70
Mar 16 16:22:07 host0 kernel: [ 5762.086939]  [<ffffffffc087fb49>]
xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
Mar 16 16:22:07 host0 kernel: [ 5762.086942]  [<ffffffff810b26b0>] ?
prepare_to_wait_event+0x110/0x110
Mar 16 16:22:07 host0 kernel: [ 5762.086958]  [<ffffffffc0869fd1>]
xfs_unmountfs+0x61/0x1a0 [xfs]
Mar 16 16:22:07 host0 kernel: [ 5762.086970]  [<ffffffffc086a97b>] ?
xfs_mru_cache_destroy+0x6b/0x90 [xfs]
Mar 16 16:22:07 host0 kernel: [ 5762.086982]  [<ffffffffc086be4d>]
xfs_fs_put_super+0x2d/0x70 [xfs]
Mar 16 16:22:07 host0 kernel: [ 5762.086985]  [<ffffffff811e9e36>]
generic_shutdown_super+0x76/0x100
Mar 16 16:22:07 host0 kernel: [ 5762.086987]  [<ffffffff811ea207>]
kill_block_super+0x27/0x70
Mar 16 16:22:07 host0 kernel: [ 5762.086990]  [<ffffffff811ea519>]
deactivate_locked_super+0x49/0x60
Mar 16 16:22:07 host0 kernel: [ 5762.086992]  [<ffffffff811eaaee>]
deactivate_super+0x4e/0x70
Mar 16 16:22:07 host0 kernel: [ 5762.086995]  [<ffffffff81207593>]
cleanup_mnt+0x43/0x90
Mar 16 16:22:07 host0 kernel: [ 5762.086998]  [<ffffffff81207632>]
__cleanup_mnt+0x12/0x20
Mar 16 16:22:07 host0 kernel: [ 5762.087003]  [<ffffffff8108f8e7>]
task_work_run+0xa7/0xe0
Mar 16 16:22:07 host0 kernel: [ 5762.087016]  [<ffffffff81014ff7>]
do_notify_resume+0x97/0xb0
Mar 16 16:22:07 host0 kernel: [ 5762.087019]  [<ffffffff81717c6f>]
int_signal+0x12/0x17

Looks like unmount/xfs_ail_push_all_sync() decided to wake up the aild
task to flush, but is never woken up at all after this. These are the
other stack traces

[8506] xfsalloc
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8507] xfs_mru_cache
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8508] xfslogd
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8663] xfs-data/dm-29
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8665] xfs-conv/dm-29
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8670] xfs-cil/dm-29
[<ffffffff8108c679>] rescuer_thread+0x209/0x2a0
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

[8672] xfsaild/dm-29
[<ffffffffc087f8d7>] xfsaild+0x5a7/0x630 [xfs]
[<ffffffff810911b9>] kthread+0xc9/0xe0
[<ffffffff81717918>] ret_from_fork+0x58/0x90
[<ffffffffffffffff>] 0xffffffffffffffff

This problem doesn't happen consistently, but happens periodically with a
drive failure/recovery followed by XFS unmount. I couldn't find this issue
fixed in later kernels. Can you please suggest how I can debug this issue
further?

Thanks!

--Shyam

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Brian Foster]

On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> Hi XFS developers,
> 
> We are seeing the following issue with XFS on kernel 3.18.19.
> 
> We have XFS mounted over a raw disk. Disk was pulled out manually. There
> were async writes on files that were errored like this
> 
...
> 
> And XFS hit metadata & Log IO errors that it decides to shutdown:
> 
> Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> old_flags=0x0 new_flags=0x2
> Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> Detected.  Shutting down filesystem
...
> Later the drive was re-inserted back. After the drive was re-inserted, XFS
> was attempted to be unmounted
> 
> Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> : umount(/sdisk/vol5b0, xfs)
> 
> But nothing happens except for the 30-secs xfs_log_force errors that keeps
> repeating
> 
...
> 
> This problem doesn't happen consistently, but happens periodically with a
> drive failure/recovery followed by XFS unmount. I couldn't find this issue
> fixed in later kernels. Can you please suggest how I can debug this issue
> further?
> 

Similar problems have been reproduced due to racy/incorrect EFI/EFD
object tracking, which are internal data structures associated with
freeing extents.

What happens if you enable tracepoints while the fs is in this hung
unmount state? 

# trace-cmd start -e "xfs:*"
# cat /sys/kernel/debug/tracing/trace_pipe

Brian

> Thanks!
> 
> --Shyam
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Brian,

Thanks for your quick reply. I repeated the test & trace-pipe is
constantly filled with this:

   xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]


while regular activity seems to happen on other inodes/kworker threads

    kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0


looks like xfsaild is not able to take lock until hung-task timeout kicks
in

   xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL

Please let me know how to debug this further. Thanks.

--Shyam
-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 22 March 2016 17:49
To: Shyam Kaushik
Cc: xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> Hi XFS developers,
>
> We are seeing the following issue with XFS on kernel 3.18.19.
>
> We have XFS mounted over a raw disk. Disk was pulled out manually. There
> were async writes on files that were errored like this
>
...
>
> And XFS hit metadata & Log IO errors that it decides to shutdown:
>
> Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> old_flags=0x0 new_flags=0x2
> Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> Detected.  Shutting down filesystem
...
> Later the drive was re-inserted back. After the drive was re-inserted,
XFS
> was attempted to be unmounted
>
> Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> : umount(/sdisk/vol5b0, xfs)
>
> But nothing happens except for the 30-secs xfs_log_force errors that
keeps
> repeating
>
...
>
> This problem doesn't happen consistently, but happens periodically with
a
> drive failure/recovery followed by XFS unmount. I couldn't find this
issue
> fixed in later kernels. Can you please suggest how I can debug this
issue
> further?
>

Similar problems have been reproduced due to racy/incorrect EFI/EFD
object tracking, which are internal data structures associated with
freeing extents.

What happens if you enable tracepoints while the fs is in this hung
unmount state?

# trace-cmd start -e "xfs:*"
# cat /sys/kernel/debug/tracing/trace_pipe

Brian

> Thanks!
>
> --Shyam
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Brian Foster]

On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> Hi Brian,
> 
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
> 
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> 

So xfsaild is spinning on this inode. It was presumably modified, logged
and flushed to the log, hence it's sitting in the AIL waiting to be
flushed to disk. xfsaild wants to push it to get it flushed to disk and
off the AIL, but it sees it is already in the flushing state as the
flush lock is held.

It's not clear to me why the inode is not removed from the AIL, or
whether that I/O was actually submitted or aborted with an error. The
shutdown involved here most likely affects this one way or the other.
IIUC, the I/O completion should eventually release the flush lock and
remove the inode from the AIL. A complete trace log of the entire
reproducer might shed more light as to what's going on.

Also, it sounds like you have a reliable reproducer. Does this reproduce
on a recent kernel?

Brian

> 
> while regular activity seems to happen on other inodes/kworker threads
> 
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
> 
> 
> looks like xfsaild is not able to take lock until hung-task timeout kicks
> in
> 
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> 
> Please let me know how to debug this further. Thanks.
> 
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually. There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
> 
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
> 
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
> 
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
> 
> Brian
> 
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

Hi Brian,

These traces, and the stack trace presented, looks quite similar with the
one we were discussing a few days ago, using a dm-thin snapshot.

Looks like with the same bug I've been hunting and Shyam confirmed my hypothesis
of this bug be able to be reproduced with a regular device.

If it's the same bug, yes, I reproduced it using upstream kernel.

The difference between both (this bug and the one I've been working on) is how
xfs actually behaves when async metadata writes fail. Other than that, it pretty
much looks the same.

Trying to unmount the filesystem hungs in xfs_log_force(), well, basically the
reason I submitted the patch to include the caller into xfs_log_force trace. I'd
like to see ftrace traces from this system with that patch if possible.

I didn't have time to keep working on this for the past few days, but looks like
it's time to come back to it.

Shyam, after you reconnected the disks, the messages about failed async metadata
writes stops to be logged?

Any chance you can reliably reproduce it?

I'm not a xfs journal expert, but it looks like the writeback of items in AIL
got stuck due the IO errors, and were never completed, but I don't know what I
should expect after the disk is reconnected.

In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until I
tried to unmount the filesystem, which differs from this case, where xfs looks
to have shutdown the filesystem after a few tries to writeback the metadata.

Anyway, I can dig more into it this week, if nobody knows what is going on
before I do it :)


On Tue, Mar 22, 2016 at 10:03:46AM -0400, Brian Foster wrote:
> On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> > Hi Brian,
> > 
> > Thanks for your quick reply. I repeated the test & trace-pipe is
> > constantly filled with this:
> > 
> >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > 
> 
> So xfsaild is spinning on this inode. It was presumably modified, logged
> and flushed to the log, hence it's sitting in the AIL waiting to be
> flushed to disk. xfsaild wants to push it to get it flushed to disk and
> off the AIL, but it sees it is already in the flushing state as the
> flush lock is held.
> 
> It's not clear to me why the inode is not removed from the AIL, or
> whether that I/O was actually submitted or aborted with an error. The
> shutdown involved here most likely affects this one way or the other.
> IIUC, the I/O completion should eventually release the flush lock and
> remove the inode from the AIL. A complete trace log of the entire
> reproducer might shed more light as to what's going on.
> 
> Also, it sounds like you have a reliable reproducer. Does this reproduce
> on a recent kernel?
> 
> Brian
> 
> > 
> > while regular activity seems to happen on other inodes/kworker threads
> > 
> >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
> > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
> > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
> > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
> > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
> > 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
> > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> > 
> > 
> > looks like xfsaild is not able to take lock until hung-task timeout kicks
> > in
> > 
> >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > 
> > Please let me know how to debug this further. Thanks.
> > 
> > --Shyam
> > -----Original Message-----
> > From: Brian Foster [mailto:bfoster@redhat.com]
> > Sent: 22 March 2016 17:49
> > To: Shyam Kaushik
> > Cc: xfs@oss.sgi.com; Alex Lyakas
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> > 
> > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > Hi XFS developers,
> > >
> > > We are seeing the following issue with XFS on kernel 3.18.19.
> > >
> > > We have XFS mounted over a raw disk. Disk was pulled out manually. There
> > > were async writes on files that were errored like this
> > >
> > ...
> > >
> > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > >
> > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > old_flags=0x0 new_flags=0x2
> > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> > > Detected.  Shutting down filesystem
> > ...
> > > Later the drive was re-inserted back. After the drive was re-inserted,
> > XFS
> > > was attempted to be unmounted
> > >
> > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > : umount(/sdisk/vol5b0, xfs)
> > >
> > > But nothing happens except for the 30-secs xfs_log_force errors that
> > keeps
> > > repeating
> > >
> > ...
> > >
> > > This problem doesn't happen consistently, but happens periodically with
> > a
> > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > issue
> > > fixed in later kernels. Can you please suggest how I can debug this
> > issue
> > > further?
> > >
> > 
> > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > object tracking, which are internal data structures associated with
> > freeing extents.
> > 
> > What happens if you enable tracepoints while the fs is in this hung
> > unmount state?
> > 
> > # trace-cmd start -e "xfs:*"
> > # cat /sys/kernel/debug/tracing/trace_pipe
> > 
> > Brian
> > 
> > > Thanks!
> > >
> > > --Shyam
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

I think I know how to reproduce it, I'll give a try and let you know


On Tue, Mar 22, 2016 at 04:38:25PM +0100, Carlos Maiolino wrote:
> Hi Brian,
> 
> These traces, and the stack trace presented, looks quite similar with the
> one we were discussing a few days ago, using a dm-thin snapshot.
> 
> Looks like with the same bug I've been hunting and Shyam confirmed my hypothesis
> of this bug be able to be reproduced with a regular device.
> 
> If it's the same bug, yes, I reproduced it using upstream kernel.
> 
> The difference between both (this bug and the one I've been working on) is how
> xfs actually behaves when async metadata writes fail. Other than that, it pretty
> much looks the same.
> 
> Trying to unmount the filesystem hungs in xfs_log_force(), well, basically the
> reason I submitted the patch to include the caller into xfs_log_force trace. I'd
> like to see ftrace traces from this system with that patch if possible.
> 
> I didn't have time to keep working on this for the past few days, but looks like
> it's time to come back to it.
> 
> Shyam, after you reconnected the disks, the messages about failed async metadata
> writes stops to be logged?
> 
> Any chance you can reliably reproduce it?
> 
> I'm not a xfs journal expert, but it looks like the writeback of items in AIL
> got stuck due the IO errors, and were never completed, but I don't know what I
> should expect after the disk is reconnected.
> 
> In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until I
> tried to unmount the filesystem, which differs from this case, where xfs looks
> to have shutdown the filesystem after a few tries to writeback the metadata.
> 
> Anyway, I can dig more into it this week, if nobody knows what is going on
> before I do it :)
> 
> 
> On Tue, Mar 22, 2016 at 10:03:46AM -0400, Brian Foster wrote:
> > On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> > > Hi Brian,
> > > 
> > > Thanks for your quick reply. I repeated the test & trace-pipe is
> > > constantly filled with this:
> > > 
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > > 
> > 
> > So xfsaild is spinning on this inode. It was presumably modified, logged
> > and flushed to the log, hence it's sitting in the AIL waiting to be
> > flushed to disk. xfsaild wants to push it to get it flushed to disk and
> > off the AIL, but it sees it is already in the flushing state as the
> > flush lock is held.
> > 
> > It's not clear to me why the inode is not removed from the AIL, or
> > whether that I/O was actually submitted or aborted with an error. The
> > shutdown involved here most likely affects this one way or the other.
> > IIUC, the I/O completion should eventually release the flush lock and
> > remove the inode from the AIL. A complete trace log of the entire
> > reproducer might shed more light as to what's going on.
> > 
> > Also, it sounds like you have a reliable reproducer. Does this reproduce
> > on a recent kernel?
> > 
> > Brian
> > 
> > > 
> > > while regular activity seems to happen on other inodes/kworker threads
> > > 
> > >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
> > > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
> > > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
> > > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
> > > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
> > > 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
> > > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > > delalloc 1 unwritten 0
> > > 
> > > 
> > > looks like xfsaild is not able to take lock until hung-task timeout kicks
> > > in
> > > 
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > > 
> > > Please let me know how to debug this further. Thanks.
> > > 
> > > --Shyam
> > > -----Original Message-----
> > > From: Brian Foster [mailto:bfoster@redhat.com]
> > > Sent: 22 March 2016 17:49
> > > To: Shyam Kaushik
> > > Cc: xfs@oss.sgi.com; Alex Lyakas
> > > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > > after disk failure/recovery
> > > 
> > > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > > Hi XFS developers,
> > > >
> > > > We are seeing the following issue with XFS on kernel 3.18.19.
> > > >
> > > > We have XFS mounted over a raw disk. Disk was pulled out manually. There
> > > > were async writes on files that were errored like this
> > > >
> > > ...
> > > >
> > > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > > >
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > > old_flags=0x0 new_flags=0x2
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> > > > Detected.  Shutting down filesystem
> > > ...
> > > > Later the drive was re-inserted back. After the drive was re-inserted,
> > > XFS
> > > > was attempted to be unmounted
> > > >
> > > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > > : umount(/sdisk/vol5b0, xfs)
> > > >
> > > > But nothing happens except for the 30-secs xfs_log_force errors that
> > > keeps
> > > > repeating
> > > >
> > > ...
> > > >
> > > > This problem doesn't happen consistently, but happens periodically with
> > > a
> > > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > > issue
> > > > fixed in later kernels. Can you please suggest how I can debug this
> > > issue
> > > > further?
> > > >
> > > 
> > > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > > object tracking, which are internal data structures associated with
> > > freeing extents.
> > > 
> > > What happens if you enable tracepoints while the fs is in this hung
> > > unmount state?
> > > 
> > > # trace-cmd start -e "xfs:*"
> > > # cat /sys/kernel/debug/tracing/trace_pipe
> > > 
> > > Brian
> > > 
> > > > Thanks!
> > > >
> > > > --Shyam
> > > >
> > > > _______________________________________________
> > > > xfs mailing list
> > > > xfs@oss.sgi.com
> > > > http://oss.sgi.com/mailman/listinfo/xfs
> > > 
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> -- 
> Carlos
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Brian,

Here are two inodes on which the xfsaild is looping over & over during
unmount. This captures right from the cp that I started with copying some
files to xfs while the drive was pulled out, later drive was pushed back &
unmount was attempted. Does this give you better picture on the issue?
Please let me know if you prefer to do some other steps in the reproducer.

              cp-30344 [001] ...1  7474.284000: xfs_iget_miss: dev 253:11
ino 0x103c84a
              cp-30344 [001] ...2  7474.284000: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_EXCL caller xfs_iget [xfs]
              cp-30344 [001] ...1  7474.284004: xfs_dir2_block_addname:
dev 253:11 ino 0x103c839 name drop-append.gif namelen 15 hashval
0x23c66973 inumber 0x103c84a op_flags ADDNAME|OKNOENT
              cp-30344 [001] ...1  7474.284014: xfs_inode_pin: dev 253:11
ino 0x103c84a count 1 pincount 0 caller xfs_cil_prepare_item.isra.1 [xfs]
              cp-30344 [001] ...1  7474.284021: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_EXCL caller xfs_inode_item_unlock [xfs]
              cp-30344 [001] ...1  7474.284028: xfs_getattr: dev 253:11
ino 0x103c84a
              cp-30344 [001] ...1  7474.284033: xfs_ilock: dev 253:11 ino
0x103c84a flags IOLOCK_EXCL caller xfs_file_buffered_aio_write.isra.9
[xfs]
              cp-30344 [001] ...1  7474.284034: xfs_file_buffered_write:
dev 253:11 ino 0x103c84a size 0x0 offset 0x0 count 0x3e9 ioflags
              cp-30344 [001] ...1  7474.284038: xfs_ilock: dev 253:11 ino
0x103c84a flags ILOCK_EXCL caller __xfs_get_blocks [xfs]
              cp-30344 [001] ...1  7474.284040: xfs_iext_insert: dev
253:11 ino 0x103c84a state  idx 0 offset 0 block 4503599627239429 count 1
flag 0 caller xfs_bmap_add_extent_hole_delay [xfs]
              cp-30344 [001] ...1  7474.284040: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_EXCL caller __xfs_get_blocks [xfs]
              cp-30344 [001] ...1  7474.284041: xfs_get_blocks_alloc: dev
253:11 ino 0x103c84a size 0x0 offset 0x0 count 4096 type 0x0 startoff 0x0
startblock -1 blockcount 0x1
              cp-30344 [001] ...1  7474.284044: xfs_iunlock: dev 253:11
ino 0x103c84a flags IOLOCK_EXCL caller xfs_file_buffered_aio_write.isra.9
[xfs]
              cp-30344 [001] ...1  7474.284047: xfs_ilock: dev 253:11 ino
0x103c84a flags ILOCK_SHARED caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284048: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284048: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags IOLOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284051: xfs_ilock: dev 253:11 ino
0x103c84a flags ILOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284051:
xfs_itruncate_extents_start: dev 253:11 ino 0x103c84a size 0x0 new_size
0x3e9
              cp-30344 [001] ...1  7474.284052: xfs_bunmap: dev 253:11 ino
0x103c84a size 0x0 bno 0x1 len 0x8000000000000flags  caller
xfs_itruncate_extents [xfs]
              cp-30344 [001] ...1  7474.284055: xfs_itruncate_extents_end:
dev 253:11 ino 0x103c84a size 0x0 new_size 0x3e9
              cp-30344 [001] ...2  7474.284058:
xfs_inode_clear_eofblocks_tag: dev 253:11 ino 0x103c84a
              cp-30344 [001] ...1  7474.284059: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284060: xfs_iunlock: dev 253:11
ino 0x103c84a flags IOLOCK_EXCL caller xfs_free_eofblocks [xfs]
    kworker/0:1H-257   [000] ...1  7477.849984: xfs_inode_unpin: dev
253:11 ino 0x103c84a count 1 pincount 1 caller xfs_trans_committed_bulk
[xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.461991: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_iflush_cluster [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.461991: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_iflush_cluster [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.462174: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.462174: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.831384: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.831384: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.855242: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.855242: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.876488: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.876489: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.898474: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.898474: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.920398: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.920398: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.942246: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.942246: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.963293: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.963293: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.984229: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.984229: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.005554: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.005555: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.027267: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.027267: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.048246: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.048246: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.069222: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   ....
   xfsaild/dm-11-22098 [002] ...2  7616.981090: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.032312: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.032312: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.083162: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.083163: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.134140: xfs_ilock_nowait: dev
253:11 ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7617.134140: xfs_iunlock: dev 253:11
ino 0x103c84a flags ILOCK_SHARED caller xfs_inode_item_push [xfs]




              cp-30344 [001] ...1  7474.284111: xfs_iget_miss: dev 253:11
ino 0x103c84b
              cp-30344 [001] ...2  7474.284112: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_EXCL caller xfs_iget [xfs]
              cp-30344 [001] ...1  7474.284115: xfs_dir2_block_addname:
dev 253:11 ino 0x103c839 name s.gif namelen 5 hashval 0x35d9f4e1 inumber
0x103c84b op_flags ADDNAME|OKNOENT
              cp-30344 [001] ...1  7474.284126: xfs_inode_pin: dev 253:11
ino 0x103c84b count 1 pincount 0 caller xfs_cil_prepare_item.isra.1 [xfs]
              cp-30344 [001] ...1  7474.284133: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_EXCL caller xfs_inode_item_unlock [xfs]
              cp-30344 [001] ...1  7474.284139: xfs_getattr: dev 253:11
ino 0x103c84b
              cp-30344 [001] ...1  7474.284145: xfs_ilock: dev 253:11 ino
0x103c84b flags IOLOCK_EXCL caller xfs_file_buffered_aio_write.isra.9
[xfs]
              cp-30344 [001] ...1  7474.284146: xfs_file_buffered_write:
dev 253:11 ino 0x103c84b size 0x0 offset 0x0 count 0x2b ioflags
              cp-30344 [001] ...1  7474.284153: xfs_ilock: dev 253:11 ino
0x103c84b flags ILOCK_EXCL caller __xfs_get_blocks [xfs]
              cp-30344 [001] ...1  7474.284155: xfs_iext_insert: dev
253:11 ino 0x103c84b state  idx 0 offset 0 block 4503599627239429 count 1
flag 0 caller xfs_bmap_add_extent_hole_delay [xfs]
              cp-30344 [001] ...1  7474.284156: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_EXCL caller __xfs_get_blocks [xfs]
              cp-30344 [001] ...1  7474.284156: xfs_get_blocks_alloc: dev
253:11 ino 0x103c84b size 0x0 offset 0x0 count 4096 type 0x0 startoff 0x0
startblock -1 blockcount 0x1
              cp-30344 [001] ...1  7474.284159: xfs_iunlock: dev 253:11
ino 0x103c84b flags IOLOCK_EXCL caller xfs_file_buffered_aio_write.isra.9
[xfs]
              cp-30344 [001] ...1  7474.284162: xfs_ilock: dev 253:11 ino
0x103c84b flags ILOCK_SHARED caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284163: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284163: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags IOLOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284165: xfs_ilock: dev 253:11 ino
0x103c84b flags ILOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284165:
xfs_itruncate_extents_start: dev 253:11 ino 0x103c84b size 0x0 new_size
0x2b
              cp-30344 [001] ...1  7474.284166: xfs_bunmap: dev 253:11 ino
0x103c84b size 0x0 bno 0x1 len 0x8000000000000flags  caller
xfs_itruncate_extents [xfs]
              cp-30344 [001] ...1  7474.284169: xfs_itruncate_extents_end:
dev 253:11 ino 0x103c84b size 0x0 new_size 0x2b
              cp-30344 [001] ...2  7474.284172:
xfs_inode_clear_eofblocks_tag: dev 253:11 ino 0x103c84b
              cp-30344 [001] ...1  7474.284173: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_EXCL caller xfs_free_eofblocks [xfs]
              cp-30344 [001] ...1  7474.284174: xfs_iunlock: dev 253:11
ino 0x103c84b flags IOLOCK_EXCL caller xfs_free_eofblocks [xfs]
    kworker/0:1H-257   [000] ...1  7477.849984: xfs_inode_unpin: dev
253:11 ino 0x103c84b count 1 pincount 1 caller xfs_trans_committed_bulk
[xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.461992: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_iflush_cluster [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.461992: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_iflush_cluster [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.462173: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.462173: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.831382: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.831383: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.855241: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.855241: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.876487: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.876487: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.898472: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.898473: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.920396: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.920397: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.942245: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.942245: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.963292: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.963292: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.984228: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7514.984228: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.005553: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.005553: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.027266: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.027266: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.048245: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7515.048246: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.069221: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.069221: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.090315: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.090315: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.111351: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7515.111351: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   ....
   xfsaild/dm-11-22098 [002] ...2  7616.673429: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.673429: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.725283: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.725283: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.777260: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.777260: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.828237: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.828237: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7616.879318: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7616.879319: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]

xfsaild was stuck towards the end like. The inode number varies, some of
which trace-buffer doesn't capture as with too many events, trace-buffer
lost several events.

   xfsaild/dm-11-22098 [001] ...2  7596.447019: xfs_ilock_nowait: dev
253:11 ino 0x6842bb6 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447020: xfs_iunlock: dev 253:11
ino 0x6842bb6 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447020: xfs_ail_flushing: dev
253:11 lip 0xffff880095382130 lsn 4/8128 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-11-22098 [001] ...2  7596.447020: xfs_ilock_nowait: dev
253:11 ino 0x6842bb4 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447020: xfs_iunlock: dev 253:11
ino 0x6842bb4 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447020: xfs_ail_flushing: dev
253:11 lip 0xffff880095382000 lsn 4/8128 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-11-22098 [001] ...2  7596.447021: xfs_ilock_nowait: dev
253:11 ino 0x58302e1 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447021: xfs_iunlock: dev 253:11
ino 0x58302e1 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447021: xfs_ail_flushing: dev
253:11 lip 0xffff880093623ed8 lsn 4/8128 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-11-22098 [001] ...2  7596.447021: xfs_ilock_nowait: dev
253:11 ino 0x6842bb3 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447022: xfs_iunlock: dev 253:11
ino 0x6842bb3 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447022: xfs_ail_flushing: dev
253:11 lip 0xffff8800334fded8 lsn 4/8128 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-11-22098 [001] ...2  7596.447022: xfs_ilock_nowait: dev
253:11 ino 0x6842bd1 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [001] ...2  7596.447022: xfs_iunlock: dev 253:11
ino 0x6842bd1 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   ....
   xfsaild/dm-11-22098 [002] ...2  7616.673429: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.673429: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.725283: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.725283: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.777260: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.777260: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.828237: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7616.828237: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7616.879318: xfs_ilock_nowait: dev
253:11 ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] .N.2  7616.879319: xfs_iunlock: dev 253:11
ino 0x103c84b flags ILOCK_SHARED caller xfs_inode_item_push [xfs]

It will be hard for me to shift to a newer kernel. But if you say we are
left with no options to root cause this issue, I can attempt at this
direction. Pls let me know. Thanks.

--Shyam
-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 22 March 2016 19:34
To: Shyam Kaushik
Cc: Alex Lyakas; xfs@oss.sgi.com
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> Hi Brian,
>
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
>
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>

So xfsaild is spinning on this inode. It was presumably modified, logged
and flushed to the log, hence it's sitting in the AIL waiting to be
flushed to disk. xfsaild wants to push it to get it flushed to disk and
off the AIL, but it sees it is already in the flushing state as the
flush lock is held.

It's not clear to me why the inode is not removed from the AIL, or
whether that I/O was actually submitted or aborted with an error. The
shutdown involved here most likely affects this one way or the other.
IIUC, the I/O completion should eventually release the flush lock and
remove the inode from the AIL. A complete trace log of the entire
reproducer might shed more light as to what's going on.

Also, it sounds like you have a reliable reproducer. Does this reproduce
on a recent kernel?

Brian

>
> while regular activity seems to happen on other inodes/kworker threads
>
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>
>
> looks like xfsaild is not able to take lock until hung-task timeout
kicks
> in
>
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>
> Please let me know how to debug this further. Thanks.
>
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually.
There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically
with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
>
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
>
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
>
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
>
> Brian
>
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Brian Foster]

On Wed, Mar 23, 2016 at 03:13:57PM +0530, Shyam Kaushik wrote:
> Hi Brian,
> 
> Here are two inodes on which the xfsaild is looping over & over during
> unmount. This captures right from the cp that I started with copying some
> files to xfs while the drive was pulled out, later drive was pushed back &
> unmount was attempted. Does this give you better picture on the issue?
> Please let me know if you prefer to do some other steps in the reproducer.
> 
...
>     kworker/0:1H-257   [000] ...1  7477.849984: xfs_inode_unpin: dev
> 253:11 ino 0x103c84a count 1 pincount 1 caller xfs_trans_committed_bulk
> [xfs]

So it looks like the transaction is committed to the log. We don't
necessarily know whether the I/O completed or this was aborted, though I
suppose we can infer the former since the inode ends up on the AIL.
There's not much else that I can go on here, however. It looks like this
trace output is redacted and/or some events were lost. For example, I
don't see any xfs_ail_insert/move events at all, etc.

I'd suggest to use trace-record to collect and send (or post somewhere)
the full, raw trace dump. Something like 'trace-record -e xfs:* sleep
999' should dump everything to a file while you try and reproduce.

Alternatively, it sounds like Carlos is working towards a reproducer on
a recent kernel and might be closer to tracking this down. One random
thought/guess from skimming through the code: I wonder if there's some
kind of race going on between a failed metadata write retry and the fs
shutdown. It looks like we retry once in xfs_buf_iodone_callbacks(). If
the fs is shutdown, we invoke the callbacks which look like it should
ultimately release the flush lock (in xfs_iflush_done()), but I'm not
sure that happens if the shutdown occurs after a failed retry.

...
> It will be hard for me to shift to a newer kernel. But if you say we are
> left with no options to root cause this issue, I can attempt at this
> direction. Pls let me know. Thanks.
> 

I ask mainly for informational and debugging purposes. If the problem
doesn't occur on a recent kernel, we might have a fix that could be
identified via a bisect. If it does, then it's still preferable to debug
on something more recent. As mentioned above, it sounds like Carlos has
been working on tracking this down on recent kernels already.

Brian

> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 19:34
> To: Shyam Kaushik
> Cc: Alex Lyakas; xfs@oss.sgi.com
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> > Hi Brian,
> >
> > Thanks for your quick reply. I repeated the test & trace-pipe is
> > constantly filled with this:
> >
> >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> 
> So xfsaild is spinning on this inode. It was presumably modified, logged
> and flushed to the log, hence it's sitting in the AIL waiting to be
> flushed to disk. xfsaild wants to push it to get it flushed to disk and
> off the AIL, but it sees it is already in the flushing state as the
> flush lock is held.
> 
> It's not clear to me why the inode is not removed from the AIL, or
> whether that I/O was actually submitted or aborted with an error. The
> shutdown involved here most likely affects this one way or the other.
> IIUC, the I/O completion should eventually release the flush lock and
> remove the inode from the AIL. A complete trace log of the entire
> reproducer might shed more light as to what's going on.
> 
> Also, it sounds like you have a reliable reproducer. Does this reproduce
> on a recent kernel?
> 
> Brian
> 
> >
> > while regular activity seems to happen on other inodes/kworker threads
> >
> >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
> 253:10
> > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
> 253:10
> > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
> 253:10
> > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
> 253:10
> > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
> delalloc
> > 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
> 253:10
> > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >
> >
> > looks like xfsaild is not able to take lock until hung-task timeout
> kicks
> > in
> >
> >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >
> > Please let me know how to debug this further. Thanks.
> >
> > --Shyam
> > -----Original Message-----
> > From: Brian Foster [mailto:bfoster@redhat.com]
> > Sent: 22 March 2016 17:49
> > To: Shyam Kaushik
> > Cc: xfs@oss.sgi.com; Alex Lyakas
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > Hi XFS developers,
> > >
> > > We are seeing the following issue with XFS on kernel 3.18.19.
> > >
> > > We have XFS mounted over a raw disk. Disk was pulled out manually.
> There
> > > were async writes on files that were errored like this
> > >
> > ...
> > >
> > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > >
> > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > old_flags=0x0 new_flags=0x2
> > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
> Error
> > > Detected.  Shutting down filesystem
> > ...
> > > Later the drive was re-inserted back. After the drive was re-inserted,
> > XFS
> > > was attempted to be unmounted
> > >
> > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > : umount(/sdisk/vol5b0, xfs)
> > >
> > > But nothing happens except for the 30-secs xfs_log_force errors that
> > keeps
> > > repeating
> > >
> > ...
> > >
> > > This problem doesn't happen consistently, but happens periodically
> with
> > a
> > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > issue
> > > fixed in later kernels. Can you please suggest how I can debug this
> > issue
> > > further?
> > >
> >
> > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > object tracking, which are internal data structures associated with
> > freeing extents.
> >
> > What happens if you enable tracepoints while the fs is in this hung
> > unmount state?
> >
> > # trace-cmd start -e "xfs:*"
> > # cat /sys/kernel/debug/tracing/trace_pipe
> >
> > Brian
> >
> > > Thanks!
> > >
> > > --Shyam
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

I'm still trying to get a reliable reproducer, at least exactly with what I have
seen a few days ago.

Shyam, could you try to reproduce it with a recent/upstream kernel? That would
be great to make sure we have been seen the same issue.

AFAICT, it happens in the following situation:

1 - Something is written to the filesystem
2 - log checkpoint is done for the previous write
3 - Disk failure
4 - XFS tries to writeback metadata logged in [2]

When [4] happens, I can't trigger xfs_log_force messages all the time, most of
time I just get an infinite loop in these messages:

[12694.318109] XFS (dm-0): Failing async write on buffer block
0xffffffffffffffff. Retrying async write.

Sometimes I can trigger the xfs_log_force() loop.

I should have something new around tomorrow.


I'm building a kernel from xfs/for-next tree, to ensure I'm using the last code,
and then will try to reproduce it from there too.



On Wed, Mar 23, 2016 at 08:30:12AM -0400, Brian Foster wrote:
> On Wed, Mar 23, 2016 at 03:13:57PM +0530, Shyam Kaushik wrote:
> > Hi Brian,
> > 
> > Here are two inodes on which the xfsaild is looping over & over during
> > unmount. This captures right from the cp that I started with copying some
> > files to xfs while the drive was pulled out, later drive was pushed back &
> > unmount was attempted. Does this give you better picture on the issue?
> > Please let me know if you prefer to do some other steps in the reproducer.
> > 
> ...
> >     kworker/0:1H-257   [000] ...1  7477.849984: xfs_inode_unpin: dev
> > 253:11 ino 0x103c84a count 1 pincount 1 caller xfs_trans_committed_bulk
> > [xfs]
> 
> So it looks like the transaction is committed to the log. We don't
> necessarily know whether the I/O completed or this was aborted, though I
> suppose we can infer the former since the inode ends up on the AIL.
> There's not much else that I can go on here, however. It looks like this
> trace output is redacted and/or some events were lost. For example, I
> don't see any xfs_ail_insert/move events at all, etc.
> 
> I'd suggest to use trace-record to collect and send (or post somewhere)
> the full, raw trace dump. Something like 'trace-record -e xfs:* sleep
> 999' should dump everything to a file while you try and reproduce.
> 
> Alternatively, it sounds like Carlos is working towards a reproducer on
> a recent kernel and might be closer to tracking this down. One random
> thought/guess from skimming through the code: I wonder if there's some
> kind of race going on between a failed metadata write retry and the fs
> shutdown. It looks like we retry once in xfs_buf_iodone_callbacks(). If
> the fs is shutdown, we invoke the callbacks which look like it should
> ultimately release the flush lock (in xfs_iflush_done()), but I'm not
> sure that happens if the shutdown occurs after a failed retry.
> 
> ...
> > It will be hard for me to shift to a newer kernel. But if you say we are
> > left with no options to root cause this issue, I can attempt at this
> > direction. Pls let me know. Thanks.
> > 
> 
> I ask mainly for informational and debugging purposes. If the problem
> doesn't occur on a recent kernel, we might have a fix that could be
> identified via a bisect. If it does, then it's still preferable to debug
> on something more recent. As mentioned above, it sounds like Carlos has
> been working on tracking this down on recent kernels already.
> 
> Brian
> 
> > --Shyam
> > -----Original Message-----
> > From: Brian Foster [mailto:bfoster@redhat.com]
> > Sent: 22 March 2016 19:34
> > To: Shyam Kaushik
> > Cc: Alex Lyakas; xfs@oss.sgi.com
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> > 
> > On Tue, Mar 22, 2016 at 06:31:48PM +0530, Shyam Kaushik wrote:
> > > Hi Brian,
> > >
> > > Thanks for your quick reply. I repeated the test & trace-pipe is
> > > constantly filled with this:
> > >
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >
> > 
> > So xfsaild is spinning on this inode. It was presumably modified, logged
> > and flushed to the log, hence it's sitting in the AIL waiting to be
> > flushed to disk. xfsaild wants to push it to get it flushed to disk and
> > off the AIL, but it sees it is already in the flushing state as the
> > flush lock is held.
> > 
> > It's not clear to me why the inode is not removed from the AIL, or
> > whether that I/O was actually submitted or aborted with an error. The
> > shutdown involved here most likely affects this one way or the other.
> > IIUC, the I/O completion should eventually release the flush lock and
> > remove the inode from the AIL. A complete trace log of the entire
> > reproducer might shed more light as to what's going on.
> > 
> > Also, it sounds like you have a reliable reproducer. Does this reproduce
> > on a recent kernel?
> > 
> > Brian
> > 
> > >
> > > while regular activity seems to happen on other inodes/kworker threads
> > >
> > >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
> > 253:10
> > > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
> > 253:10
> > > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
> > 253:10
> > > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > > delalloc 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
> > 253:10
> > > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
> > delalloc
> > > 0 unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
> > 253:10
> > > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > > unwritten 0
> > >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > > delalloc 1 unwritten 0
> > >
> > >
> > > looks like xfsaild is not able to take lock until hung-task timeout
> > kicks
> > > in
> > >
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> > >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> > >
> > > Please let me know how to debug this further. Thanks.
> > >
> > > --Shyam
> > > -----Original Message-----
> > > From: Brian Foster [mailto:bfoster@redhat.com]
> > > Sent: 22 March 2016 17:49
> > > To: Shyam Kaushik
> > > Cc: xfs@oss.sgi.com; Alex Lyakas
> > > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > > after disk failure/recovery
> > >
> > > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > > Hi XFS developers,
> > > >
> > > > We are seeing the following issue with XFS on kernel 3.18.19.
> > > >
> > > > We have XFS mounted over a raw disk. Disk was pulled out manually.
> > There
> > > > were async writes on files that were errored like this
> > > >
> > > ...
> > > >
> > > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > > >
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > > old_flags=0x0 new_flags=0x2
> > > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
> > Error
> > > > Detected.  Shutting down filesystem
> > > ...
> > > > Later the drive was re-inserted back. After the drive was re-inserted,
> > > XFS
> > > > was attempted to be unmounted
> > > >
> > > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > > : umount(/sdisk/vol5b0, xfs)
> > > >
> > > > But nothing happens except for the 30-secs xfs_log_force errors that
> > > keeps
> > > > repeating
> > > >
> > > ...
> > > >
> > > > This problem doesn't happen consistently, but happens periodically
> > with
> > > a
> > > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > > issue
> > > > fixed in later kernels. Can you please suggest how I can debug this
> > > issue
> > > > further?
> > > >
> > >
> > > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > > object tracking, which are internal data structures associated with
> > > freeing extents.
> > >
> > > What happens if you enable tracepoints while the fs is in this hung
> > > unmount state?
> > >
> > > # trace-cmd start -e "xfs:*"
> > > # cat /sys/kernel/debug/tracing/trace_pipe
> > >
> > > Brian
> > >
> > > > Thanks!
> > > >
> > > > --Shyam
> > > >
> > > > _______________________________________________
> > > > xfs mailing list
> > > > xfs@oss.sgi.com
> > > > http://oss.sgi.com/mailman/listinfo/xfs
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> > 
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Wed, Mar 23, 2016 at 04:32:21PM +0100, Carlos Maiolino wrote:
> I'm still trying to get a reliable reproducer, at least exactly with what I have
> seen a few days ago.
> 
> Shyam, could you try to reproduce it with a recent/upstream kernel? That would
> be great to make sure we have been seen the same issue.
> 
> AFAICT, it happens in the following situation:
> 
> 1 - Something is written to the filesystem
> 2 - log checkpoint is done for the previous write
> 3 - Disk failure
> 4 - XFS tries to writeback metadata logged in [2]
> 
> When [4] happens, I can't trigger xfs_log_force messages all the time, most of
> time I just get an infinite loop in these messages:
> 
> [12694.318109] XFS (dm-0): Failing async write on buffer block
> 0xffffffffffffffff. Retrying async write.
>
> Sometimes I can trigger the xfs_log_force() loop.

This all smells like the filesystem is getting IO errors but it not
in a shutdown state. What happens when you run 'xfs_io -x -c
"shutdown" /mnt/pt' on a filesystem in this state? Can you then
unmount it?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

On Thu, Mar 24, 2016 at 09:37:47AM +1100, Dave Chinner wrote:
> On Wed, Mar 23, 2016 at 04:32:21PM +0100, Carlos Maiolino wrote:
> > I'm still trying to get a reliable reproducer, at least exactly with what I have
> > seen a few days ago.
> > 
> > Shyam, could you try to reproduce it with a recent/upstream kernel? That would
> > be great to make sure we have been seen the same issue.
> > 
> > AFAICT, it happens in the following situation:
> > 
> > 1 - Something is written to the filesystem
> > 2 - log checkpoint is done for the previous write
> > 3 - Disk failure
> > 4 - XFS tries to writeback metadata logged in [2]
> > 
> > When [4] happens, I can't trigger xfs_log_force messages all the time, most of
> > time I just get an infinite loop in these messages:
> > 
> > [12694.318109] XFS (dm-0): Failing async write on buffer block
> > 0xffffffffffffffff. Retrying async write.
> >
> > Sometimes I can trigger the xfs_log_force() loop.
> 
> This all smells like the filesystem is getting IO errors but it not
> in a shutdown state. What happens when you run 'xfs_io -x -c
> "shutdown" /mnt/pt' on a filesystem in this state? Can you then
> unmount it?
> 

I'll give it a try today, although, I can't do it while umount command is hung,
since, before the command get stuck, the mount point is removed from the
user namespace, so I have no access to the mountpoint from userspace while the
command is 'running'.


> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

I can now reproduce it, or at least part of the problem.

Regarding your question Dave, yes, it can be unmounted after I issue xfs_io shutdown
command. But, if a umount is issued before that, then we can't find the
mountpoint anymore.

I'm not sure if I'm correct, but, what it looks like to me, as you already
mentioned, is that we keep getting IO errors but we never actually shutdown
the filesystem while doing async metadata writes.

I believe I've found the problem. So, I will try to explain it, so you guys
can review and let me know if I'm right or not

I was looking the code, and for me, looks like async retries are designed to
keep retrying forever, and rely on some other part of the filesystem to actually
shutdown it.

I tested this hypothesis trying to issue some other IO after the messages below
starts to be logged with trace_xfs_buf_ioerror turned on, And before actually
umount the filesystem.

And yes, the filesystem has been shutdown after some extra IO (IO errors
returned of course).

But, if we try to unmount, we can't issue any other IO, since we loose the
mountpoint.

I apologize if my description is confusing, I'm not the best person organizing
ideas, but, feel free to ask me any question.




When I reproduce the problem, I can see these two messages being written to
kernel:

[   59.914938] XFS (dm-0): metadata I/O error: block 0x0
("xfs_buf_iodone_callbacks") error 5 numblks 1

[   59.964831] XFS (dm-0): Failing async write on buffer block
0xffffffffffffffff. Retrying async write.


Looking through the code, looks like inside xfs_buf_iodone_callbacks(), we
expect the filesystem to be put down somewhere else, and then stop retrying:

        /*
        ¦* If we've already decided to shutdown the filesystem because of
        ¦* I/O errors, there's no point in giving this a retry.
        ¦*/
        if (XFS_FORCED_SHUTDOWN(mp)) {


And so, since we are in ASYNC writes here, we fall in:

if (bp->b_flags & XBF_ASYNC) {

Where, we will clear bp->b_error and set XBF_WRITE_FAIL flag in the buffer,
and submit the buffer for IO.


xfs_buf_ioerror(bp, 0); /* errno of 0 unsets the flag */

if (!(bp->b_flags & (XBF_STALE|XBF_WRITE_FAIL))) {
                        bp->b_flags |= XBF_WRITE | XBF_ASYNC |
                                ¦      XBF_DONE | XBF_WRITE_FAIL;
                        xfs_buf_submit(bp);


So the buffer flag is set with XBF_WRITE_FAIL and IO submitted.

In xfs_buf_submit():

We clear the io_error, if any:

bp->b_io_error = 0;

Then, we send buffer for IO via _xfs_buf_ioapply():

atomic_set(&bp->b_io_remaining, 1);
_xfs_buf_ioapply(bp);

At _xfs_buf_ioapply, we clear buffer errors:

bp->b_error = 0;

Then call:  xfs_buf_ioapply_map()

Where we actually setup the bio and send it to block layer, where
such status is captured by xfs_buf_bio_end_io()

--- done, buffer has been sent for IO ---


Now, inside xfs_buf_bio_end_io():

Since we receive an IO error here, we set bp->b_io_error:

bp->b_io_error = bio->bi_error

I did no reads here, so we skip until:

if (atomic_dec_and_test(&bp->b_io_remaining) == 1)
        xfs_buf_ioend_async(bp);


Workqueue starts to handle the IO completion where we call xfS_buf_ioend():


At xfs_buf_ioend(), we catch the I/O error here, and set bp->berror accordingly:

If (!bp->b_error && bp->b_io_error)
        xfs_buf_ioerror(bp, bp->b_io_error);

and we have one of the traces being logged:

kworker/1:2-1294  [001] .... 11814.953710: xfs_buf_ioerror: dev 253:0 bno
0xffffffffffffffff len 0x200 hold 3 pincount 0 lock 0 error -5 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend_work

Since it's an inode, we have iodone set to xfs_buf_iodone_callbacks() and call
it here:

  if (bp->b_iodone)
                (*(bp->b_iodone))(bp);


At xfs_buf_iodone_callbacks:


bp->berror is set, so we do not skip until do_callbacks label.

        if (likely(!bp->b_error))
                goto do_callbacks;


we call xfs_buf_ioerror_alert() here, and then we have logged:

[15063.857999] XFS (dm-0): metadata I/O error: block 0x0
("xfs_buf_iodone_callbacks") error 5 numblks 1



So, we reach again:

if (bp->b_flags & XBF_ASYNC) {

and again, we clear the error here:

xfs_buf_ioerror(bp, 0); 

Issuing this trace:

     kworker/1:2-1294  [001] .... 11814.953711: xfs_buf_ioerror: dev 253:0 bno
0xffffffffffffffff len 0x200 hold 3 pincount 0 lock 0 error 0 flags
ASYNC|DONE|PAGES caller xfs_buf_iodone_callbacks

and submit the buffer again, restarting the process.




Looks like, somebody already noticed it:

        /*
        ¦* If the write was asynchronous then no one will be looking for the
        ¦* error.  Clear the error state and write the buffer out again.
        ¦*
        ¦* XXX: This helps against transient write errors, but we need to find
        ¦* a way to shut the filesystem down if the writes keep failing.
        ¦*
        ¦* In practice we'll shut the filesystem down soon as non-transient
        ¦* errors tend to affect the whole device and a failing log write
        ¦* will make us give up.  But we really ought to do better here.
        ¦*/




So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
setting limits to IO errors can be slightly modified to fix this issue too, but,
the problem is that the user must set it BEFORE he tries to unmount the
filesystem, otherwise it will get stuck here.

IMHO, we maybe need to fix it during the umount process, so, despite how the
error handler is set (using Dave's patchset), or in the current situation (try
forever), unmount process does not lockup because of this.

I wrote a "hacky" patch to notify buf_iodone_callback that the filesystem was
being unmounted, and stop retrying. I'm not sure if it's a good solution, but,
I'll send the patch to at least log it in the mailing list and keep the
discussion.


Sorry for the long e-mail, hope it was at least useful :)

Cheers


On Thu, Mar 24, 2016 at 09:37:47AM +1100, Dave Chinner wrote:
> On Wed, Mar 23, 2016 at 04:32:21PM +0100, Carlos Maiolino wrote:
> > I'm still trying to get a reliable reproducer, at least exactly with what I have
> > seen a few days ago.
> > 
> > Shyam, could you try to reproduce it with a recent/upstream kernel? That would
> > be great to make sure we have been seen the same issue.
> > 
> > AFAICT, it happens in the following situation:
> > 
> > 1 - Something is written to the filesystem
> > 2 - log checkpoint is done for the previous write
> > 3 - Disk failure
> > 4 - XFS tries to writeback metadata logged in [2]
> > 
> > When [4] happens, I can't trigger xfs_log_force messages all the time, most of
> > time I just get an infinite loop in these messages:
> > 
> > [12694.318109] XFS (dm-0): Failing async write on buffer block
> > 0xffffffffffffffff. Retrying async write.
> >
> > Sometimes I can trigger the xfs_log_force() loop.
> 
> This all smells like the filesystem is getting IO errors but it not
> in a shutdown state. What happens when you run 'xfs_io -x -c
> "shutdown" /mnt/pt' on a filesystem in this state? Can you then
> unmount it?
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Thu, Mar 24, 2016 at 05:52:44PM +0100, Carlos Maiolino wrote:
> I can now reproduce it, or at least part of the problem.
> 
> Regarding your question Dave, yes, it can be unmounted after I issue xfs_io shutdown
> command. But, if a umount is issued before that, then we can't find the
> mountpoint anymore.
> 
> I'm not sure if I'm correct, but, what it looks like to me, as you already
> mentioned, is that we keep getting IO errors but we never actually shutdown
> the filesystem while doing async metadata writes.

*nod*

> I believe I've found the problem. So, I will try to explain it, so you guys
> can review and let me know if I'm right or not
> 
> I was looking the code, and for me, looks like async retries are designed to
> keep retrying forever, and rely on some other part of the filesystem to actually
> shutdown it.

*nod*

[snip description of metadata IO error behaviour]

Yes, that is exactly how the code is expected to behave - in fact,
that's how it was originally designed to behave.

> Looks like, somebody already noticed it:
> 
>         /*
>         ¦* If the write was asynchronous then no one will be looking for the
>         ¦* error.  Clear the error state and write the buffer out again.
>         ¦*
>         ¦* XXX: This helps against transient write errors, but we need to find
>         ¦* a way to shut the filesystem down if the writes keep failing.
>         ¦*
>         ¦* In practice we'll shut the filesystem down soon as non-transient
>         ¦* errors tend to affect the whole device and a failing log write
>         ¦* will make us give up.  But we really ought to do better here.
>         ¦*/
> 
> 
> So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
> setting limits to IO errors can be slightly modified to fix this issue too, but,

The patchset I have doesn't need modification to fix this issue - it
has a patch specifically to address this, and it changes the default
behaviour to "fail async writes at unmount":

http://oss.sgi.com/archives/xfs/2015-08/msg00092.html

> the problem is that the user must set it BEFORE he tries to unmount the
> filesystem, otherwise it will get stuck here.

Yes, but that doesn't answer the big question: why don't the
periodic log forces that are failing with EIO cause a filesystem
shutdown? We issue a log force every 30s even during unmount, and a
failed log IO must cause the filesystem to shut down. So why aren't
these causing the filesystem to shutdown as we'd expect when the
device has been pulled?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

On Fri, Mar 25, 2016 at 08:56:03AM +1100, Dave Chinner wrote:
> On Thu, Mar 24, 2016 at 05:52:44PM +0100, Carlos Maiolino wrote:
> > I can now reproduce it, or at least part of the problem.
> > 
> > Regarding your question Dave, yes, it can be unmounted after I issue xfs_io shutdown
> > command. But, if a umount is issued before that, then we can't find the
> > mountpoint anymore.
> > 
> > I'm not sure if I'm correct, but, what it looks like to me, as you already
> > mentioned, is that we keep getting IO errors but we never actually shutdown
> > the filesystem while doing async metadata writes.
> 
> *nod*
> 
> > I believe I've found the problem. So, I will try to explain it, so you guys
> > can review and let me know if I'm right or not
> > 
> > I was looking the code, and for me, looks like async retries are designed to
> > keep retrying forever, and rely on some other part of the filesystem to actually
> > shutdown it.
> 
> *nod*
> 
> [snip description of metadata IO error behaviour]
> 
> Yes, that is exactly how the code is expected to behave - in fact,
> that's how it was originally designed to behave.
> 
> > Looks like, somebody already noticed it:
> > 
> >         /*
> >         ¦* If the write was asynchronous then no one will be looking for the
> >         ¦* error.  Clear the error state and write the buffer out again.
> >         ¦*
> >         ¦* XXX: This helps against transient write errors, but we need to find
> >         ¦* a way to shut the filesystem down if the writes keep failing.
> >         ¦*
> >         ¦* In practice we'll shut the filesystem down soon as non-transient
> >         ¦* errors tend to affect the whole device and a failing log write
> >         ¦* will make us give up.  But we really ought to do better here.
> >         ¦*/
> > 
> > 
> > So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
> > setting limits to IO errors can be slightly modified to fix this issue too, but,
> 
> The patchset I have doesn't need modification to fix this issue - it
> has a patch specifically to address this, and it changes the default
> behaviour to "fail async writes at unmount":
> 
> http://oss.sgi.com/archives/xfs/2015-08/msg00092.html
> 
> > the problem is that the user must set it BEFORE he tries to unmount the
> > filesystem, otherwise it will get stuck here.
> 
> Yes, but that doesn't answer the big question: why don't the
> periodic log forces that are failing with EIO cause a filesystem
> shutdown? We issue a log force every 30s even during unmount, and a
> failed log IO must cause the filesystem to shut down. So why aren't
> these causing the filesystem to shutdown as we'd expect when the
> device has been pulled?
> 

Right, good point, I'll take a look on it

> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Carlos,

w.r.t. your question below

>>> Shyam, after you reconnected the disks, the messages about failed
async metadata
>>> writes stops to be logged?

After I reconnect the disks, messages about failed async metadata writes
stops to be logged. But the key thing is messages like

XFS (dm-29): xfs_log_force: error -5 returned

Keeps repeating every 30-secs which indicates that there is some buf/io
error status that is being carried forward.

>>> Any chance you can reliably reproduce it?

Yes I have a way to reproduce it, but its not reliable. What I do is setup
a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
copy varying sized files into the FS. At this point pull out the drive
(with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
convert the dm-linear to dm-ioerror. Then I bring back the underlying
drive, convert back dm-ioerror to dm-linear & try to unmount XFS.

This problem somehow happens on a newly created XFS. If I copy several
files into XFS/delete them & then copy them again, repeat the drive
failure/recovery experiment it doesn't reproduce.

Thanks.

--Shyam

Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery
From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
Date: Tue, 22 Mar 2016 16:38:25 +0100
In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
Mail-followup-to: xfs@xxxxxxxxxxx
User-agent: Mutt/1.5.24 (2015-08-30)

Hi Brian,

These traces, and the stack trace presented, looks quite similar with the
one we were discussing a few days ago, using a dm-thin snapshot.

Looks like with the same bug I've been hunting and Shyam confirmed my
hypothesis
of this bug be able to be reproduced with a regular device.

If it's the same bug, yes, I reproduced it using upstream kernel.

The difference between both (this bug and the one I've been working on) is
how
xfs actually behaves when async metadata writes fail. Other than that, it
pretty
much looks the same.

Trying to unmount the filesystem hungs in xfs_log_force(), well, basically
the
reason I submitted the patch to include the caller into xfs_log_force
trace. I'd
like to see ftrace traces from this system with that patch if possible.

I didn't have time to keep working on this for the past few days, but
looks like
it's time to come back to it.

Shyam, after you reconnected the disks, the messages about failed async
metadata
writes stops to be logged?

Any chance you can reliably reproduce it?

I'm not a xfs journal expert, but it looks like the writeback of items in
AIL
got stuck due the IO errors, and were never completed, but I don't know
what I
should expect after the disk is reconnected.

In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until
I
tried to unmount the filesystem, which differs from this case, where xfs
looks
to have shutdown the filesystem after a few tries to writeback the
metadata.

Anyway, I can dig more into it this week, if nobody knows what is going on
before I do it :)

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 22 March 2016 18:32
To: 'Brian Foster'
Cc: 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Brian,

Thanks for your quick reply. I repeated the test & trace-pipe is
constantly filled with this:

   xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]


while regular activity seems to happen on other inodes/kworker threads

    kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0


looks like xfsaild is not able to take lock until hung-task timeout kicks
in

   xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL

Please let me know how to debug this further. Thanks.

--Shyam
-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 22 March 2016 17:49
To: Shyam Kaushik
Cc: xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> Hi XFS developers,
>
> We are seeing the following issue with XFS on kernel 3.18.19.
>
> We have XFS mounted over a raw disk. Disk was pulled out manually. There
> were async writes on files that were errored like this
>
...
>
> And XFS hit metadata & Log IO errors that it decides to shutdown:
>
> Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> old_flags=0x0 new_flags=0x2
> Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> Detected.  Shutting down filesystem
...
> Later the drive was re-inserted back. After the drive was re-inserted,
XFS
> was attempted to be unmounted
>
> Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> : umount(/sdisk/vol5b0, xfs)
>
> But nothing happens except for the 30-secs xfs_log_force errors that
keeps
> repeating
>
...
>
> This problem doesn't happen consistently, but happens periodically with
a
> drive failure/recovery followed by XFS unmount. I couldn't find this
issue
> fixed in later kernels. Can you please suggest how I can debug this
issue
> further?
>

Similar problems have been reproduced due to racy/incorrect EFI/EFD
object tracking, which are internal data structures associated with
freeing extents.

What happens if you enable tracepoints while the fs is in this hung
unmount state?

# trace-cmd start -e "xfs:*"
# cat /sys/kernel/debug/tracing/trace_pipe

Brian

> Thanks!
>
> --Shyam
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Brian, Carlos, Dave,

Not sure what changed since yesterday, but with several retries I couldn't
reproduce this issue today.

I will continue trying to reproduce this issue & check xfs_io/shutdown
setting + on latest kernel to see if I can help in getting better
insights. Thanks.

--Shyam

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 23 March 2016 15:23
To: 'Brian Foster'
Cc: 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Carlos,

w.r.t. your question below

>>> Shyam, after you reconnected the disks, the messages about failed
async metadata
>>> writes stops to be logged?

After I reconnect the disks, messages about failed async metadata writes
stops to be logged. But the key thing is messages like

XFS (dm-29): xfs_log_force: error -5 returned

Keeps repeating every 30-secs which indicates that there is some buf/io
error status that is being carried forward.

>>> Any chance you can reliably reproduce it?

Yes I have a way to reproduce it, but its not reliable. What I do is setup
a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
copy varying sized files into the FS. At this point pull out the drive
(with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
convert the dm-linear to dm-ioerror. Then I bring back the underlying
drive, convert back dm-ioerror to dm-linear & try to unmount XFS.

This problem somehow happens on a newly created XFS. If I copy several
files into XFS/delete them & then copy them again, repeat the drive
failure/recovery experiment it doesn't reproduce.

Thanks.

--Shyam

Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery
From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
Date: Tue, 22 Mar 2016 16:38:25 +0100
In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
Mail-followup-to: xfs@xxxxxxxxxxx
User-agent: Mutt/1.5.24 (2015-08-30)

Hi Brian,

These traces, and the stack trace presented, looks quite similar with the
one we were discussing a few days ago, using a dm-thin snapshot.

Looks like with the same bug I've been hunting and Shyam confirmed my
hypothesis
of this bug be able to be reproduced with a regular device.

If it's the same bug, yes, I reproduced it using upstream kernel.

The difference between both (this bug and the one I've been working on) is
how
xfs actually behaves when async metadata writes fail. Other than that, it
pretty
much looks the same.

Trying to unmount the filesystem hungs in xfs_log_force(), well, basically
the
reason I submitted the patch to include the caller into xfs_log_force
trace. I'd
like to see ftrace traces from this system with that patch if possible.

I didn't have time to keep working on this for the past few days, but
looks like
it's time to come back to it.

Shyam, after you reconnected the disks, the messages about failed async
metadata
writes stops to be logged?

Any chance you can reliably reproduce it?

I'm not a xfs journal expert, but it looks like the writeback of items in
AIL
got stuck due the IO errors, and were never completed, but I don't know
what I
should expect after the disk is reconnected.

In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until
I
tried to unmount the filesystem, which differs from this case, where xfs
looks
to have shutdown the filesystem after a few tries to writeback the
metadata.

Anyway, I can dig more into it this week, if nobody knows what is going on
before I do it :)

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 22 March 2016 18:32
To: 'Brian Foster'
Cc: 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Brian,

Thanks for your quick reply. I repeated the test & trace-pipe is
constantly filled with this:

   xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]


while regular activity seems to happen on other inodes/kworker threads

    kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0


looks like xfsaild is not able to take lock until hung-task timeout kicks
in

   xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL

Please let me know how to debug this further. Thanks.

--Shyam
-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 22 March 2016 17:49
To: Shyam Kaushik
Cc: xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> Hi XFS developers,
>
> We are seeing the following issue with XFS on kernel 3.18.19.
>
> We have XFS mounted over a raw disk. Disk was pulled out manually. There
> were async writes on files that were errored like this
>
...
>
> And XFS hit metadata & Log IO errors that it decides to shutdown:
>
> Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> old_flags=0x0 new_flags=0x2
> Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> Detected.  Shutting down filesystem
...
> Later the drive was re-inserted back. After the drive was re-inserted,
XFS
> was attempted to be unmounted
>
> Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> : umount(/sdisk/vol5b0, xfs)
>
> But nothing happens except for the 30-secs xfs_log_force errors that
keeps
> repeating
>
...
>
> This problem doesn't happen consistently, but happens periodically with
a
> drive failure/recovery followed by XFS unmount. I couldn't find this
issue
> fixed in later kernels. Can you please suggest how I can debug this
issue
> further?
>

Similar problems have been reproduced due to racy/incorrect EFI/EFD
object tracking, which are internal data structures associated with
freeing extents.

What happens if you enable tracepoints while the fs is in this hung
unmount state?

# trace-cmd start -e "xfs:*"
# cat /sys/kernel/debug/tracing/trace_pipe

Brian

> Thanks!
>
> --Shyam
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Dave, Brian, Carlos,

While trying to reproduce this issue I have been running into different
issues that are similar. Underlying issue remains the same when backend to
XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
with stack like

kernel: [14952.671131]  [<ffffffffc06a5f59>]
xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
kernel: [14952.671139]  [<ffffffff810b26b0>] ?
prepare_to_wait_event+0x110/0x110
kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
[xfs]

while running trace-events, XFS ail push keeps looping around

   xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL

observe ino==0x0 (its a freed inode) but the corresponding lip stays with
xfsaild push. I had run a systemtap when this problem happened & the
following keeps repeating.

xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
flags:0x1
xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
xfs_iflush_abort() - iip:0x0

xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
marked & does an xfs_iflush_abort().xfs_iflush_abort() sees i_itemp==NULL
on the lip & doesn't do anything. So we have this stale lip that is not
associated to inode anymore that keeps xfs_ail_push_all_sync() busy.

On few other occurrences, i have had NULL pointer/dereference or other
sorts of crashes at this line

	xfsaild_push()
		lock_result = lip->li_ops->iop_push(lip,
&ailp->xa_buf_list);

with debug prints, in one of the occurrence lip->li_ops was NULL & in
another lip->li_ops was pointing to a bad pointer that subsequent
dereference crashed it. This also indicates that a bad/freed lip was
inserted that xfsaild_push() is working.

I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" & realized
that this could explain the above issue where a lip that has been freed is
wrongly referenced & later we could even have the inode disconnected. What
do you think?

In any case I uploaded the full xfs trace events before this issue started
& till the end. It is at
https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0

Pls let me know. Thanks.

--Shyam

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 23 March 2016 15:23
To: 'Brian Foster'
Cc: 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Carlos,

w.r.t. your question below

>>> Shyam, after you reconnected the disks, the messages about failed
async metadata
>>> writes stops to be logged?

After I reconnect the disks, messages about failed async metadata writes
stops to be logged. But the key thing is messages like

XFS (dm-29): xfs_log_force: error -5 returned

Keeps repeating every 30-secs which indicates that there is some buf/io
error status that is being carried forward.

>>> Any chance you can reliably reproduce it?

Yes I have a way to reproduce it, but its not reliable. What I do is setup
a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
copy varying sized files into the FS. At this point pull out the drive
(with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
convert the dm-linear to dm-ioerror. Then I bring back the underlying
drive, convert back dm-ioerror to dm-linear & try to unmount XFS.

This problem somehow happens on a newly created XFS. If I copy several
files into XFS/delete them & then copy them again, repeat the drive
failure/recovery experiment it doesn't reproduce.

Thanks.

--Shyam

Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery
From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
Date: Tue, 22 Mar 2016 16:38:25 +0100
In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
Mail-followup-to: xfs@xxxxxxxxxxx
User-agent: Mutt/1.5.24 (2015-08-30)

Hi Brian,

These traces, and the stack trace presented, looks quite similar with the
one we were discussing a few days ago, using a dm-thin snapshot.

Looks like with the same bug I've been hunting and Shyam confirmed my
hypothesis
of this bug be able to be reproduced with a regular device.

If it's the same bug, yes, I reproduced it using upstream kernel.

The difference between both (this bug and the one I've been working on) is
how
xfs actually behaves when async metadata writes fail. Other than that, it
pretty
much looks the same.

Trying to unmount the filesystem hungs in xfs_log_force(), well, basically
the
reason I submitted the patch to include the caller into xfs_log_force
trace. I'd
like to see ftrace traces from this system with that patch if possible.

I didn't have time to keep working on this for the past few days, but
looks like
it's time to come back to it.

Shyam, after you reconnected the disks, the messages about failed async
metadata
writes stops to be logged?

Any chance you can reliably reproduce it?

I'm not a xfs journal expert, but it looks like the writeback of items in
AIL
got stuck due the IO errors, and were never completed, but I don't know
what I
should expect after the disk is reconnected.

In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until
I
tried to unmount the filesystem, which differs from this case, where xfs
looks
to have shutdown the filesystem after a few tries to writeback the
metadata.

Anyway, I can dig more into it this week, if nobody knows what is going on
before I do it :)

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 22 March 2016 18:32
To: 'Brian Foster'
Cc: 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Brian,

Thanks for your quick reply. I repeated the test & trace-pipe is
constantly filled with this:

   xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]


while regular activity seems to happen on other inodes/kworker threads

    kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
delalloc 0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
0 unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
unwritten 0
    kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
delalloc 1 unwritten 0


looks like xfsaild is not able to take lock until hung-task timeout kicks
in

   xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
   xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
   xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL

Please let me know how to debug this further. Thanks.

--Shyam
-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 22 March 2016 17:49
To: Shyam Kaushik
Cc: xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> Hi XFS developers,
>
> We are seeing the following issue with XFS on kernel 3.18.19.
>
> We have XFS mounted over a raw disk. Disk was pulled out manually. There
> were async writes on files that were errored like this
>
...
>
> And XFS hit metadata & Log IO errors that it decides to shutdown:
>
> Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> old_flags=0x0 new_flags=0x2
> Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> Detected.  Shutting down filesystem
...
> Later the drive was re-inserted back. After the drive was re-inserted,
XFS
> was attempted to be unmounted
>
> Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> : umount(/sdisk/vol5b0, xfs)
>
> But nothing happens except for the 30-secs xfs_log_force errors that
keeps
> repeating
>
...
>
> This problem doesn't happen consistently, but happens periodically with
a
> drive failure/recovery followed by XFS unmount. I couldn't find this
issue
> fixed in later kernels. Can you please suggest how I can debug this
issue
> further?
>

Similar problems have been reproduced due to racy/incorrect EFI/EFD
object tracking, which are internal data structures associated with
freeing extents.

What happens if you enable tracepoints while the fs is in this hung
unmount state?

# trace-cmd start -e "xfs:*"
# cat /sys/kernel/debug/tracing/trace_pipe

Brian

> Thanks!
>
> --Shyam
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Brian Foster]

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
> 
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
> 
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
> 
> while running trace-events, XFS ail push keeps looping around
> 
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
> 
> observe ino==0x0 (its a freed inode) but the corresponding lip stays with
> xfsaild push. I had run a systemtap when this problem happened & the
> following keeps repeating.
> 

That's interesting, and looks different from the original report in
terms of the inode number being cleared. The original report looks like
it has a valid inode and there's some problematic sequence where it's
not being removed from the AIL in the event of errors.

I think at this point we know that XFS attempts to retry these I/Os
indefinitely. Dave has outstanding patches to deal with this issue. The
question Dave raised above was whether the filesystem shut down, and if
not, why (as a failed log I/O should cause a shutdown)? Carlos was
looking into this it appears...

> xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> flags:0x1
> xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> xfs_iflush_abort() - iip:0x0
> 
> xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> marked & does an xfs_iflush_abort().xfs_iflush_abort() sees i_itemp==NULL
> on the lip & doesn't do anything. So we have this stale lip that is not
> associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
> 
> On few other occurrences, i have had NULL pointer/dereference or other
> sorts of crashes at this line
> 
> 	xfsaild_push()
> 		lock_result = lip->li_ops->iop_push(lip,
> &ailp->xa_buf_list);
> 
> with debug prints, in one of the occurrence lip->li_ops was NULL & in
> another lip->li_ops was pointing to a bad pointer that subsequent
> dereference crashed it. This also indicates that a bad/freed lip was
> inserted that xfsaild_push() is working.
> 
> I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" & realized
> that this could explain the above issue where a lip that has been freed is
> wrongly referenced & later we could even have the inode disconnected. What
> do you think?
> 
> In any case I uploaded the full xfs trace events before this issue started
> & till the end. It is at
> https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
> 

It seems like it could be related. I didn't catch anything obvious from
the trace, but there's a lot of data there. The RCU-unsafe issue was
difficult to track down without instrumentation.. I'm not sure that will
be evident from the trace. The best thing to do wrt to that might be to
try with Dave's patches, as that so far appears to address the problem.
(In fact, it might be worth it to try Dave's shutdown on umount patch
he referred to up-thread to address the original problem as well).

Brian

> Pls let me know. Thanks.
> 
> --Shyam
> 
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 23 March 2016 15:23
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> Hi Carlos,
> 
> w.r.t. your question below
> 
> >>> Shyam, after you reconnected the disks, the messages about failed
> async metadata
> >>> writes stops to be logged?
> 
> After I reconnect the disks, messages about failed async metadata writes
> stops to be logged. But the key thing is messages like
> 
> XFS (dm-29): xfs_log_force: error -5 returned
> 
> Keeps repeating every 30-secs which indicates that there is some buf/io
> error status that is being carried forward.
> 
> >>> Any chance you can reliably reproduce it?
> 
> Yes I have a way to reproduce it, but its not reliable. What I do is setup
> a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> copy varying sized files into the FS. At this point pull out the drive
> (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> convert the dm-linear to dm-ioerror. Then I bring back the underlying
> drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
> 
> This problem somehow happens on a newly created XFS. If I copy several
> files into XFS/delete them & then copy them again, repeat the drive
> failure/recovery experiment it doesn't reproduce.
> 
> Thanks.
> 
> --Shyam
> 
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> Date: Tue, 22 Mar 2016 16:38:25 +0100
> In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> Mail-followup-to: xfs@xxxxxxxxxxx
> User-agent: Mutt/1.5.24 (2015-08-30)
> 
> Hi Brian,
> 
> These traces, and the stack trace presented, looks quite similar with the
> one we were discussing a few days ago, using a dm-thin snapshot.
> 
> Looks like with the same bug I've been hunting and Shyam confirmed my
> hypothesis
> of this bug be able to be reproduced with a regular device.
> 
> If it's the same bug, yes, I reproduced it using upstream kernel.
> 
> The difference between both (this bug and the one I've been working on) is
> how
> xfs actually behaves when async metadata writes fail. Other than that, it
> pretty
> much looks the same.
> 
> Trying to unmount the filesystem hungs in xfs_log_force(), well, basically
> the
> reason I submitted the patch to include the caller into xfs_log_force
> trace. I'd
> like to see ftrace traces from this system with that patch if possible.
> 
> I didn't have time to keep working on this for the past few days, but
> looks like
> it's time to come back to it.
> 
> Shyam, after you reconnected the disks, the messages about failed async
> metadata
> writes stops to be logged?
> 
> Any chance you can reliably reproduce it?
> 
> I'm not a xfs journal expert, but it looks like the writeback of items in
> AIL
> got stuck due the IO errors, and were never completed, but I don't know
> what I
> should expect after the disk is reconnected.
> 
> In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN until
> I
> tried to unmount the filesystem, which differs from this case, where xfs
> looks
> to have shutdown the filesystem after a few tries to writeback the
> metadata.
> 
> Anyway, I can dig more into it this week, if nobody knows what is going on
> before I do it :)
> 
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 22 March 2016 18:32
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> Hi Brian,
> 
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
> 
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> 
> 
> while regular activity seems to happen on other inodes/kworker threads
> 
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev 253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev 253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev 253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev 253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev 253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
> 
> 
> looks like xfsaild is not able to take lock until hung-task timeout kicks
> in
> 
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> 
> Please let me know how to debug this further. Thanks.
> 
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually. There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
> 
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
> 
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
> 
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
> 
> Brian
> 
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Brian,

Yes the below report is a bit different than the original report. I wanted
to confirm if the new patches from Dave will see these odd
crashes/hung-task.

I applied Dave's patch & managed to recreate the original issue. With
systemtap I do see that FS has been marked for shutdown (xfs_aildpush() -
mp:0xffff880062d09800 m_flags:4325656), but still the infinite tries keep
happening with xfsaild_push() seeing XFS_IFLOCK on the inode. Pls note
that this issue happens only when I see

XFS (dm-10): Detected failing async write on buffer block 0x368d76a0.
Retrying async write.

in the logs. In all other cases, this problem doesn't happen. So there is
something with this async-write retries that is leaving the state of the
inode with IFLOCK that rest of aildpush is not able to handle.

--Shyam

-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 08 April 2016 18:47
To: Shyam Kaushik
Cc: david@fromorbit.com; xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
>
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend
to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
>
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
>
> while running trace-events, XFS ail push keeps looping around
>
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>
> observe ino==0x0 (its a freed inode) but the corresponding lip stays
with
> xfsaild push. I had run a systemtap when this problem happened & the
> following keeps repeating.
>

That's interesting, and looks different from the original report in
terms of the inode number being cleared. The original report looks like
it has a valid inode and there's some problematic sequence where it's
not being removed from the AIL in the event of errors.

I think at this point we know that XFS attempts to retry these I/Os
indefinitely. Dave has outstanding patches to deal with this issue. The
question Dave raised above was whether the filesystem shut down, and if
not, why (as a failed log I/O should cause a shutdown)? Carlos was
looking into this it appears...

> xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> flags:0x1
> xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> xfs_iflush_abort() - iip:0x0
>
> xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> marked & does an xfs_iflush_abort().xfs_iflush_abort() sees
i_itemp==NULL
> on the lip & doesn't do anything. So we have this stale lip that is not
> associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
>
> On few other occurrences, i have had NULL pointer/dereference or other
> sorts of crashes at this line
>
> 	xfsaild_push()
> 		lock_result = lip->li_ops->iop_push(lip,
> &ailp->xa_buf_list);
>
> with debug prints, in one of the occurrence lip->li_ops was NULL & in
> another lip->li_ops was pointing to a bad pointer that subsequent
> dereference crashed it. This also indicates that a bad/freed lip was
> inserted that xfsaild_push() is working.
>
> I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" &
realized
> that this could explain the above issue where a lip that has been freed
is
> wrongly referenced & later we could even have the inode disconnected.
What
> do you think?
>
> In any case I uploaded the full xfs trace events before this issue
started
> & till the end. It is at
> https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
>

It seems like it could be related. I didn't catch anything obvious from
the trace, but there's a lot of data there. The RCU-unsafe issue was
difficult to track down without instrumentation.. I'm not sure that will
be evident from the trace. The best thing to do wrt to that might be to
try with Dave's patches, as that so far appears to address the problem.
(In fact, it might be worth it to try Dave's shutdown on umount patch
he referred to up-thread to address the original problem as well).

Brian

> Pls let me know. Thanks.
>
> --Shyam
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 23 March 2016 15:23
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Carlos,
>
> w.r.t. your question below
>
> >>> Shyam, after you reconnected the disks, the messages about failed
> async metadata
> >>> writes stops to be logged?
>
> After I reconnect the disks, messages about failed async metadata writes
> stops to be logged. But the key thing is messages like
>
> XFS (dm-29): xfs_log_force: error -5 returned
>
> Keeps repeating every 30-secs which indicates that there is some buf/io
> error status that is being carried forward.
>
> >>> Any chance you can reliably reproduce it?
>
> Yes I have a way to reproduce it, but its not reliable. What I do is
setup
> a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> copy varying sized files into the FS. At this point pull out the drive
> (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> convert the dm-linear to dm-ioerror. Then I bring back the underlying
> drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
>
> This problem somehow happens on a newly created XFS. If I copy several
> files into XFS/delete them & then copy them again, repeat the drive
> failure/recovery experiment it doesn't reproduce.
>
> Thanks.
>
> --Shyam
>
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> Date: Tue, 22 Mar 2016 16:38:25 +0100
> In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> Mail-followup-to: xfs@xxxxxxxxxxx
> User-agent: Mutt/1.5.24 (2015-08-30)
>
> Hi Brian,
>
> These traces, and the stack trace presented, looks quite similar with
the
> one we were discussing a few days ago, using a dm-thin snapshot.
>
> Looks like with the same bug I've been hunting and Shyam confirmed my
> hypothesis
> of this bug be able to be reproduced with a regular device.
>
> If it's the same bug, yes, I reproduced it using upstream kernel.
>
> The difference between both (this bug and the one I've been working on)
is
> how
> xfs actually behaves when async metadata writes fail. Other than that,
it
> pretty
> much looks the same.
>
> Trying to unmount the filesystem hungs in xfs_log_force(), well,
basically
> the
> reason I submitted the patch to include the caller into xfs_log_force
> trace. I'd
> like to see ftrace traces from this system with that patch if possible.
>
> I didn't have time to keep working on this for the past few days, but
> looks like
> it's time to come back to it.
>
> Shyam, after you reconnected the disks, the messages about failed async
> metadata
> writes stops to be logged?
>
> Any chance you can reliably reproduce it?
>
> I'm not a xfs journal expert, but it looks like the writeback of items
in
> AIL
> got stuck due the IO errors, and were never completed, but I don't know
> what I
> should expect after the disk is reconnected.
>
> In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN
until
> I
> tried to unmount the filesystem, which differs from this case, where xfs
> looks
> to have shutdown the filesystem after a few tries to writeback the
> metadata.
>
> Anyway, I can dig more into it this week, if nobody knows what is going
on
> before I do it :)
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 22 March 2016 18:32
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Brian,
>
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
>
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>
>
> while regular activity seems to happen on other inodes/kworker threads
>
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>
>
> looks like xfsaild is not able to take lock until hung-task timeout
kicks
> in
>
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>
> Please let me know how to debug this further. Thanks.
>
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually.
There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically
with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
>
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
>
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
>
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
>
> Brian
>
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

Hi Shyam,

do you mind to share your systemtap script with us?

I'd like to take a look on it, and probably Brian will be interested to.


On Fri, Apr 08, 2016 at 07:05:24PM +0530, Shyam Kaushik wrote:
> Hi Brian,
> 
> Yes the below report is a bit different than the original report. I wanted
> to confirm if the new patches from Dave will see these odd
> crashes/hung-task.
> 
> I applied Dave's patch & managed to recreate the original issue. With
> systemtap I do see that FS has been marked for shutdown (xfs_aildpush() -
> mp:0xffff880062d09800 m_flags:4325656), but still the infinite tries keep
> happening with xfsaild_push() seeing XFS_IFLOCK on the inode. Pls note
> that this issue happens only when I see
> 
> XFS (dm-10): Detected failing async write on buffer block 0x368d76a0.
> Retrying async write.
> 
> in the logs. In all other cases, this problem doesn't happen. So there is
> something with this async-write retries that is leaving the state of the
> inode with IFLOCK that rest of aildpush is not able to handle.
> 
> --Shyam
> 
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 08 April 2016 18:47
> To: Shyam Kaushik
> Cc: david@fromorbit.com; xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> > Hi Dave, Brian, Carlos,
> >
> > While trying to reproduce this issue I have been running into different
> > issues that are similar. Underlying issue remains the same when backend
> to
> > XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> > with stack like
> >
> > kernel: [14952.671131]  [<ffffffffc06a5f59>]
> > xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> > kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> > prepare_to_wait_event+0x110/0x110
> > kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> > [xfs]
> >
> > while running trace-events, XFS ail push keeps looping around
> >
> >    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> > 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> > ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> > lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> > 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> > ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> > lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
> >
> > observe ino==0x0 (its a freed inode) but the corresponding lip stays
> with
> > xfsaild push. I had run a systemtap when this problem happened & the
> > following keeps repeating.
> >
> 
> That's interesting, and looks different from the original report in
> terms of the inode number being cleared. The original report looks like
> it has a valid inode and there's some problematic sequence where it's
> not being removed from the AIL in the event of errors.
> 
> I think at this point we know that XFS attempts to retry these I/Os
> indefinitely. Dave has outstanding patches to deal with this issue. The
> question Dave raised above was whether the filesystem shut down, and if
> not, why (as a failed log I/O should cause a shutdown)? Carlos was
> looking into this it appears...
> 
> > xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> > flags:0x1
> > xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> > xfs_iflush_abort() - iip:0x0
> >
> > xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> > marked & does an xfs_iflush_abort().xfs_iflush_abort() sees
> i_itemp==NULL
> > on the lip & doesn't do anything. So we have this stale lip that is not
> > associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
> >
> > On few other occurrences, i have had NULL pointer/dereference or other
> > sorts of crashes at this line
> >
> > 	xfsaild_push()
> > 		lock_result = lip->li_ops->iop_push(lip,
> > &ailp->xa_buf_list);
> >
> > with debug prints, in one of the occurrence lip->li_ops was NULL & in
> > another lip->li_ops was pointing to a bad pointer that subsequent
> > dereference crashed it. This also indicates that a bad/freed lip was
> > inserted that xfsaild_push() is working.
> >
> > I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" &
> realized
> > that this could explain the above issue where a lip that has been freed
> is
> > wrongly referenced & later we could even have the inode disconnected.
> What
> > do you think?
> >
> > In any case I uploaded the full xfs trace events before this issue
> started
> > & till the end. It is at
> > https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
> >
> 
> It seems like it could be related. I didn't catch anything obvious from
> the trace, but there's a lot of data there. The RCU-unsafe issue was
> difficult to track down without instrumentation.. I'm not sure that will
> be evident from the trace. The best thing to do wrt to that might be to
> try with Dave's patches, as that so far appears to address the problem.
> (In fact, it might be worth it to try Dave's shutdown on umount patch
> he referred to up-thread to address the original problem as well).
> 
> Brian
> 
> > Pls let me know. Thanks.
> >
> > --Shyam
> >
> > -----Original Message-----
> > From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> > Sent: 23 March 2016 15:23
> > To: 'Brian Foster'
> > Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> > Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > Hi Carlos,
> >
> > w.r.t. your question below
> >
> > >>> Shyam, after you reconnected the disks, the messages about failed
> > async metadata
> > >>> writes stops to be logged?
> >
> > After I reconnect the disks, messages about failed async metadata writes
> > stops to be logged. But the key thing is messages like
> >
> > XFS (dm-29): xfs_log_force: error -5 returned
> >
> > Keeps repeating every 30-secs which indicates that there is some buf/io
> > error status that is being carried forward.
> >
> > >>> Any chance you can reliably reproduce it?
> >
> > Yes I have a way to reproduce it, but its not reliable. What I do is
> setup
> > a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> > copy varying sized files into the FS. At this point pull out the drive
> > (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> > convert the dm-linear to dm-ioerror. Then I bring back the underlying
> > drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
> >
> > This problem somehow happens on a newly created XFS. If I copy several
> > files into XFS/delete them & then copy them again, repeat the drive
> > failure/recovery experiment it doesn't reproduce.
> >
> > Thanks.
> >
> > --Shyam
> >
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> > From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> > Date: Tue, 22 Mar 2016 16:38:25 +0100
> > In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> > Mail-followup-to: xfs@xxxxxxxxxxx
> > User-agent: Mutt/1.5.24 (2015-08-30)
> >
> > Hi Brian,
> >
> > These traces, and the stack trace presented, looks quite similar with
> the
> > one we were discussing a few days ago, using a dm-thin snapshot.
> >
> > Looks like with the same bug I've been hunting and Shyam confirmed my
> > hypothesis
> > of this bug be able to be reproduced with a regular device.
> >
> > If it's the same bug, yes, I reproduced it using upstream kernel.
> >
> > The difference between both (this bug and the one I've been working on)
> is
> > how
> > xfs actually behaves when async metadata writes fail. Other than that,
> it
> > pretty
> > much looks the same.
> >
> > Trying to unmount the filesystem hungs in xfs_log_force(), well,
> basically
> > the
> > reason I submitted the patch to include the caller into xfs_log_force
> > trace. I'd
> > like to see ftrace traces from this system with that patch if possible.
> >
> > I didn't have time to keep working on this for the past few days, but
> > looks like
> > it's time to come back to it.
> >
> > Shyam, after you reconnected the disks, the messages about failed async
> > metadata
> > writes stops to be logged?
> >
> > Any chance you can reliably reproduce it?
> >
> > I'm not a xfs journal expert, but it looks like the writeback of items
> in
> > AIL
> > got stuck due the IO errors, and were never completed, but I don't know
> > what I
> > should expect after the disk is reconnected.
> >
> > In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN
> until
> > I
> > tried to unmount the filesystem, which differs from this case, where xfs
> > looks
> > to have shutdown the filesystem after a few tries to writeback the
> > metadata.
> >
> > Anyway, I can dig more into it this week, if nobody knows what is going
> on
> > before I do it :)
> >
> > -----Original Message-----
> > From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> > Sent: 22 March 2016 18:32
> > To: 'Brian Foster'
> > Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> > Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > Hi Brian,
> >
> > Thanks for your quick reply. I repeated the test & trace-pipe is
> > constantly filled with this:
> >
> >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> >
> > while regular activity seems to happen on other inodes/kworker threads
> >
> >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
> 253:10
> > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
> 253:10
> > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
> 253:10
> > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
> 253:10
> > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
> delalloc
> > 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
> 253:10
> > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >
> >
> > looks like xfsaild is not able to take lock until hung-task timeout
> kicks
> > in
> >
> >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >
> > Please let me know how to debug this further. Thanks.
> >
> > --Shyam
> > -----Original Message-----
> > From: Brian Foster [mailto:bfoster@redhat.com]
> > Sent: 22 March 2016 17:49
> > To: Shyam Kaushik
> > Cc: xfs@oss.sgi.com; Alex Lyakas
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > Hi XFS developers,
> > >
> > > We are seeing the following issue with XFS on kernel 3.18.19.
> > >
> > > We have XFS mounted over a raw disk. Disk was pulled out manually.
> There
> > > were async writes on files that were errored like this
> > >
> > ...
> > >
> > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > >
> > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > old_flags=0x0 new_flags=0x2
> > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
> Error
> > > Detected.  Shutting down filesystem
> > ...
> > > Later the drive was re-inserted back. After the drive was re-inserted,
> > XFS
> > > was attempted to be unmounted
> > >
> > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > : umount(/sdisk/vol5b0, xfs)
> > >
> > > But nothing happens except for the 30-secs xfs_log_force errors that
> > keeps
> > > repeating
> > >
> > ...
> > >
> > > This problem doesn't happen consistently, but happens periodically
> with
> > a
> > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > issue
> > > fixed in later kernels. Can you please suggest how I can debug this
> > issue
> > > further?
> > >
> >
> > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > object tracking, which are internal data structures associated with
> > freeing extents.
> >
> > What happens if you enable tracepoints while the fs is in this hung
> > unmount state?
> >
> > # trace-cmd start -e "xfs:*"
> > # cat /sys/kernel/debug/tracing/trace_pipe
> >
> > Brian
> >
> > > Thanks!
> > >
> > > --Shyam
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Dave, Brian, Carlos,

I think I see the problem. The problem is when the device just starts to
return IO errors & not shutdown yet that buf could get returned with IO
error. We mark it with XBF_WRITE_FAIL & re-submit the IO. But this IO
could fail again & at this point since we already have XBF_WRITE_FAIL we
dont submit the buf, but leave it in a bad state.

w.r.t. trace-buffer

we have xfs_buf_iodone_callbacks() called with bp->b_error & its an async
IO. So we mark XBF_WRITE_FAIL on the buf & do xfs_buf_submit()

    kworker/2:1H-163   [002] ...1  7513.493658: xfs_buf_item_iodone_async:
dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]

but then this again fails & comes back to the same call, but this time
with XBF_WRITE_FAIL set & in this case unfortunately we just do
xfs_buf_relse() & there is no callback flow that is going through.

    kworker/2:1H-163   [002] ...1  7513.494385: xfs_buf_item_iodone_async:
dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]

Do you think this is the problem?

There are several bufs that got hit like this. Here is complete trace for
one such buf

   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
caller xfs_buf_item_push
 [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_item_push: dev
253:11 bno 0x25718d0 len 0x2000 hold 1 pincount 0 lock 0 flags DONE|PAGES
recur 0 refcount 0 bli
flags DIRTY|INODE_ALLOC lidesc 0x          (null) liflags IN_AIL
   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_delwri_queue: dev
253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
caller xfs_buf_item
_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.466617: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_buf_i
tem_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller _xfs_buf
_find [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_find: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_buf_g
et_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_get: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_buf_re
ad_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_read: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_trans
_read_buf_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_delwri_queued: dev
253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller xf
s_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_inode
_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_inode_i
tem_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller __xfs_bu
f_delwri_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_delwri_split: dev
253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller xfs
_buf_delwri_submit_nowait [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_submit: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller __xfs_buf_delwri_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_hold: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469868: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_iodone: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_item_iodone_async:
dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
ASYNC|DONE|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_submit: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_hold: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.493766: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_iodone: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_item_iodone_async:
dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]


as seen above the first time buf hits ioerror we retry, but the 2nd time
we give it up without finishing the buf. Its in a stuck state & this keeps
xfsaild thread busy when unmounting trying to flush it out.

Thanks.

--Shyam
-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 08 April 2016 19:05
To: 'Brian Foster'
Cc: 'david@fromorbit.com'; 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Brian,

Yes the below report is a bit different than the original report. I wanted
to confirm if the new patches from Dave will see these odd
crashes/hung-task.

I applied Dave's patch & managed to recreate the original issue. With
systemtap I do see that FS has been marked for shutdown (xfs_aildpush() -
mp:0xffff880062d09800 m_flags:4325656), but still the infinite tries keep
happening with xfsaild_push() seeing XFS_IFLOCK on the inode. Pls note
that this issue happens only when I see

XFS (dm-10): Detected failing async write on buffer block 0x368d76a0.
Retrying async write.

in the logs. In all other cases, this problem doesn't happen. So there is
something with this async-write retries that is leaving the state of the
inode with IFLOCK that rest of aildpush is not able to handle.

--Shyam

-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 08 April 2016 18:47
To: Shyam Kaushik
Cc: david@fromorbit.com; xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
>
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend
to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
>
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
>
> while running trace-events, XFS ail push keeps looping around
>
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>
> observe ino==0x0 (its a freed inode) but the corresponding lip stays
with
> xfsaild push. I had run a systemtap when this problem happened & the
> following keeps repeating.
>

That's interesting, and looks different from the original report in
terms of the inode number being cleared. The original report looks like
it has a valid inode and there's some problematic sequence where it's
not being removed from the AIL in the event of errors.

I think at this point we know that XFS attempts to retry these I/Os
indefinitely. Dave has outstanding patches to deal with this issue. The
question Dave raised above was whether the filesystem shut down, and if
not, why (as a failed log I/O should cause a shutdown)? Carlos was
looking into this it appears...

> xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> flags:0x1
> xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> xfs_iflush_abort() - iip:0x0
>
> xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> marked & does an xfs_iflush_abort().xfs_iflush_abort() sees
i_itemp==NULL
> on the lip & doesn't do anything. So we have this stale lip that is not
> associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
>
> On few other occurrences, i have had NULL pointer/dereference or other
> sorts of crashes at this line
>
> 	xfsaild_push()
> 		lock_result = lip->li_ops->iop_push(lip,
> &ailp->xa_buf_list);
>
> with debug prints, in one of the occurrence lip->li_ops was NULL & in
> another lip->li_ops was pointing to a bad pointer that subsequent
> dereference crashed it. This also indicates that a bad/freed lip was
> inserted that xfsaild_push() is working.
>
> I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" &
realized
> that this could explain the above issue where a lip that has been freed
is
> wrongly referenced & later we could even have the inode disconnected.
What
> do you think?
>
> In any case I uploaded the full xfs trace events before this issue
started
> & till the end. It is at
> https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
>

It seems like it could be related. I didn't catch anything obvious from
the trace, but there's a lot of data there. The RCU-unsafe issue was
difficult to track down without instrumentation.. I'm not sure that will
be evident from the trace. The best thing to do wrt to that might be to
try with Dave's patches, as that so far appears to address the problem.
(In fact, it might be worth it to try Dave's shutdown on umount patch
he referred to up-thread to address the original problem as well).

Brian

> Pls let me know. Thanks.
>
> --Shyam
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 23 March 2016 15:23
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Carlos,
>
> w.r.t. your question below
>
> >>> Shyam, after you reconnected the disks, the messages about failed
> async metadata
> >>> writes stops to be logged?
>
> After I reconnect the disks, messages about failed async metadata writes
> stops to be logged. But the key thing is messages like
>
> XFS (dm-29): xfs_log_force: error -5 returned
>
> Keeps repeating every 30-secs which indicates that there is some buf/io
> error status that is being carried forward.
>
> >>> Any chance you can reliably reproduce it?
>
> Yes I have a way to reproduce it, but its not reliable. What I do is
setup
> a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> copy varying sized files into the FS. At this point pull out the drive
> (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> convert the dm-linear to dm-ioerror. Then I bring back the underlying
> drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
>
> This problem somehow happens on a newly created XFS. If I copy several
> files into XFS/delete them & then copy them again, repeat the drive
> failure/recovery experiment it doesn't reproduce.
>
> Thanks.
>
> --Shyam
>
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> Date: Tue, 22 Mar 2016 16:38:25 +0100
> In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> Mail-followup-to: xfs@xxxxxxxxxxx
> User-agent: Mutt/1.5.24 (2015-08-30)
>
> Hi Brian,
>
> These traces, and the stack trace presented, looks quite similar with
the
> one we were discussing a few days ago, using a dm-thin snapshot.
>
> Looks like with the same bug I've been hunting and Shyam confirmed my
> hypothesis
> of this bug be able to be reproduced with a regular device.
>
> If it's the same bug, yes, I reproduced it using upstream kernel.
>
> The difference between both (this bug and the one I've been working on)
is
> how
> xfs actually behaves when async metadata writes fail. Other than that,
it
> pretty
> much looks the same.
>
> Trying to unmount the filesystem hungs in xfs_log_force(), well,
basically
> the
> reason I submitted the patch to include the caller into xfs_log_force
> trace. I'd
> like to see ftrace traces from this system with that patch if possible.
>
> I didn't have time to keep working on this for the past few days, but
> looks like
> it's time to come back to it.
>
> Shyam, after you reconnected the disks, the messages about failed async
> metadata
> writes stops to be logged?
>
> Any chance you can reliably reproduce it?
>
> I'm not a xfs journal expert, but it looks like the writeback of items
in
> AIL
> got stuck due the IO errors, and were never completed, but I don't know
> what I
> should expect after the disk is reconnected.
>
> In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN
until
> I
> tried to unmount the filesystem, which differs from this case, where xfs
> looks
> to have shutdown the filesystem after a few tries to writeback the
> metadata.
>
> Anyway, I can dig more into it this week, if nobody knows what is going
on
> before I do it :)
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 22 March 2016 18:32
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Brian,
>
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
>
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>
>
> while regular activity seems to happen on other inodes/kworker threads
>
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>
>
> looks like xfsaild is not able to take lock until hung-task timeout
kicks
> in
>
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>
> Please let me know how to debug this further. Thanks.
>
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually.
There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically
with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
>
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
>
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
>
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
>
> Brian
>
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Brian Foster]

On Fri, Apr 08, 2016 at 11:18:42PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
> 
> I think I see the problem. The problem is when the device just starts to
> return IO errors & not shutdown yet that buf could get returned with IO
> error. We mark it with XBF_WRITE_FAIL & re-submit the IO. But this IO
> could fail again & at this point since we already have XBF_WRITE_FAIL we
> dont submit the buf, but leave it in a bad state.
> 
> w.r.t. trace-buffer
> 
> we have xfs_buf_iodone_callbacks() called with bp->b_error & its an async
> IO. So we mark XBF_WRITE_FAIL on the buf & do xfs_buf_submit()
> 
>     kworker/2:1H-163   [002] ...1  7513.493658: xfs_buf_item_iodone_async:
> dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
> ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]
> 
> but then this again fails & comes back to the same call, but this time
> with XBF_WRITE_FAIL set & in this case unfortunately we just do
> xfs_buf_relse() & there is no callback flow that is going through.
> 
>     kworker/2:1H-163   [002] ...1  7513.494385: xfs_buf_item_iodone_async:
> dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]
> 
> Do you think this is the problem?
> 

No, there are two separate things going on here. The WRITE_FAIL thing
stuck out to me as well when Carlos and I were reading through some of
the code. IIRC, the write fail bit is intended to deal with transient
storage errors. So if the I/O returns with an error, we resubmit it once
before actually completing the I/O (and thus propagating the error up
the stack).

If the retry attempt still returns an error, we simply release the
buffer and leave it on the AIL. The next go around sees the failed state
on the buffer, warns about the repeatedly failing async write (see
xfs_buf_item_push()) and starts the whole sequence over again.

The question I have is why does this buffer remain on the AIL if the
filesystem has indeed shutdown. IIRC, Dave's original patch addressed
the umount hang due to failing I/O problem by forcing a shutdown in that
context. I don't see how that helps us if the fs is already shutdown,
however. :/

Brian

> There are several bufs that got hit like this. Here is complete trace for
> one such buf
> 
>    xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_trylock: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
> caller xfs_buf_item_push
>  [xfs]
>    xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_item_push: dev
> 253:11 bno 0x25718d0 len 0x2000 hold 1 pincount 0 lock 0 flags DONE|PAGES
> recur 0 refcount 0 bli
> flags DIRTY|INODE_ALLOC lidesc 0x          (null) liflags IN_AIL
>    xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_delwri_queue: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
> caller xfs_buf_item
> _push [xfs]
>    xfsaild/dm-11-22098 [002] ...2  7484.466617: xfs_buf_unlock: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
> DONE|PAGES|DELWRI_Q caller xfs_buf_i
> tem_push [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_trylock: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
> DONE|PAGES|DELWRI_Q caller _xfs_buf
> _find [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_find: dev 253:11
> bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
> READ|TRYLOCK|UNMAPPED caller xfs_buf_g
> et_map [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_get: dev 253:11
> bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
> READ|TRYLOCK|UNMAPPED caller xfs_buf_re
> ad_map [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_read: dev 253:11
> bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
> READ|TRYLOCK|UNMAPPED caller xfs_trans
> _read_buf_map [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_delwri_queued: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
> DONE|PAGES|DELWRI_Q caller xf
> s_inode_item_push [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_unlock: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
> DONE|PAGES|DELWRI_Q caller xfs_inode
> _item_push [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_rele: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
> DONE|PAGES|DELWRI_Q caller xfs_inode_i
> tem_push [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_trylock: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> DONE|PAGES|DELWRI_Q caller __xfs_bu
> f_delwri_submit [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_delwri_split: dev
> 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> DONE|PAGES|DELWRI_Q caller xfs
> _buf_delwri_submit_nowait [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_submit: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|PAGES caller __xfs_buf_delwri_submit [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_hold: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
>    xfsaild/dm-11-22098 [002] ...1  7484.469868: xfs_buf_rele: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_iodone: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
> 253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
> ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_item_iodone_async:
> dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
> 253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
> ASYNC|DONE|PAGES caller xfs_buf_iodone_callbacks [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_submit: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_hold: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
>     kworker/2:1H-163   [002] ...1  7513.493766: xfs_buf_rele: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_iodone: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
> 253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_item_iodone_async:
> dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
> 253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_unlock: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
>     kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_rele: dev 253:11
> bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
> ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
> 
> 
> as seen above the first time buf hits ioerror we retry, but the 2nd time
> we give it up without finishing the buf. Its in a stuck state & this keeps
> xfsaild thread busy when unmounting trying to flush it out.
> 
> Thanks.
> 
> --Shyam
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 08 April 2016 19:05
> To: 'Brian Foster'
> Cc: 'david@fromorbit.com'; 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> Hi Brian,
> 
> Yes the below report is a bit different than the original report. I wanted
> to confirm if the new patches from Dave will see these odd
> crashes/hung-task.
> 
> I applied Dave's patch & managed to recreate the original issue. With
> systemtap I do see that FS has been marked for shutdown (xfs_aildpush() -
> mp:0xffff880062d09800 m_flags:4325656), but still the infinite tries keep
> happening with xfsaild_push() seeing XFS_IFLOCK on the inode. Pls note
> that this issue happens only when I see
> 
> XFS (dm-10): Detected failing async write on buffer block 0x368d76a0.
> Retrying async write.
> 
> in the logs. In all other cases, this problem doesn't happen. So there is
> something with this async-write retries that is leaving the state of the
> inode with IFLOCK that rest of aildpush is not able to handle.
> 
> --Shyam
> 
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 08 April 2016 18:47
> To: Shyam Kaushik
> Cc: david@fromorbit.com; xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> 
> On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> > Hi Dave, Brian, Carlos,
> >
> > While trying to reproduce this issue I have been running into different
> > issues that are similar. Underlying issue remains the same when backend
> to
> > XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> > with stack like
> >
> > kernel: [14952.671131]  [<ffffffffc06a5f59>]
> > xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> > kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> > prepare_to_wait_event+0x110/0x110
> > kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> > [xfs]
> >
> > while running trace-events, XFS ail push keeps looping around
> >
> >    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> > 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> > ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> > lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> > 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> > ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> > lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
> >
> > observe ino==0x0 (its a freed inode) but the corresponding lip stays
> with
> > xfsaild push. I had run a systemtap when this problem happened & the
> > following keeps repeating.
> >
> 
> That's interesting, and looks different from the original report in
> terms of the inode number being cleared. The original report looks like
> it has a valid inode and there's some problematic sequence where it's
> not being removed from the AIL in the event of errors.
> 
> I think at this point we know that XFS attempts to retry these I/Os
> indefinitely. Dave has outstanding patches to deal with this issue. The
> question Dave raised above was whether the filesystem shut down, and if
> not, why (as a failed log I/O should cause a shutdown)? Carlos was
> looking into this it appears...
> 
> > xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> > flags:0x1
> > xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> > xfs_iflush_abort() - iip:0x0
> >
> > xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> > marked & does an xfs_iflush_abort().xfs_iflush_abort() sees
> i_itemp==NULL
> > on the lip & doesn't do anything. So we have this stale lip that is not
> > associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
> >
> > On few other occurrences, i have had NULL pointer/dereference or other
> > sorts of crashes at this line
> >
> > 	xfsaild_push()
> > 		lock_result = lip->li_ops->iop_push(lip,
> > &ailp->xa_buf_list);
> >
> > with debug prints, in one of the occurrence lip->li_ops was NULL & in
> > another lip->li_ops was pointing to a bad pointer that subsequent
> > dereference crashed it. This also indicates that a bad/freed lip was
> > inserted that xfsaild_push() is working.
> >
> > I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" &
> realized
> > that this could explain the above issue where a lip that has been freed
> is
> > wrongly referenced & later we could even have the inode disconnected.
> What
> > do you think?
> >
> > In any case I uploaded the full xfs trace events before this issue
> started
> > & till the end. It is at
> > https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
> >
> 
> It seems like it could be related. I didn't catch anything obvious from
> the trace, but there's a lot of data there. The RCU-unsafe issue was
> difficult to track down without instrumentation.. I'm not sure that will
> be evident from the trace. The best thing to do wrt to that might be to
> try with Dave's patches, as that so far appears to address the problem.
> (In fact, it might be worth it to try Dave's shutdown on umount patch
> he referred to up-thread to address the original problem as well).
> 
> Brian
> 
> > Pls let me know. Thanks.
> >
> > --Shyam
> >
> > -----Original Message-----
> > From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> > Sent: 23 March 2016 15:23
> > To: 'Brian Foster'
> > Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> > Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > Hi Carlos,
> >
> > w.r.t. your question below
> >
> > >>> Shyam, after you reconnected the disks, the messages about failed
> > async metadata
> > >>> writes stops to be logged?
> >
> > After I reconnect the disks, messages about failed async metadata writes
> > stops to be logged. But the key thing is messages like
> >
> > XFS (dm-29): xfs_log_force: error -5 returned
> >
> > Keeps repeating every 30-secs which indicates that there is some buf/io
> > error status that is being carried forward.
> >
> > >>> Any chance you can reliably reproduce it?
> >
> > Yes I have a way to reproduce it, but its not reliable. What I do is
> setup
> > a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> > copy varying sized files into the FS. At this point pull out the drive
> > (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> > convert the dm-linear to dm-ioerror. Then I bring back the underlying
> > drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
> >
> > This problem somehow happens on a newly created XFS. If I copy several
> > files into XFS/delete them & then copy them again, repeat the drive
> > failure/recovery experiment it doesn't reproduce.
> >
> > Thanks.
> >
> > --Shyam
> >
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> > From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> > Date: Tue, 22 Mar 2016 16:38:25 +0100
> > In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> > Mail-followup-to: xfs@xxxxxxxxxxx
> > User-agent: Mutt/1.5.24 (2015-08-30)
> >
> > Hi Brian,
> >
> > These traces, and the stack trace presented, looks quite similar with
> the
> > one we were discussing a few days ago, using a dm-thin snapshot.
> >
> > Looks like with the same bug I've been hunting and Shyam confirmed my
> > hypothesis
> > of this bug be able to be reproduced with a regular device.
> >
> > If it's the same bug, yes, I reproduced it using upstream kernel.
> >
> > The difference between both (this bug and the one I've been working on)
> is
> > how
> > xfs actually behaves when async metadata writes fail. Other than that,
> it
> > pretty
> > much looks the same.
> >
> > Trying to unmount the filesystem hungs in xfs_log_force(), well,
> basically
> > the
> > reason I submitted the patch to include the caller into xfs_log_force
> > trace. I'd
> > like to see ftrace traces from this system with that patch if possible.
> >
> > I didn't have time to keep working on this for the past few days, but
> > looks like
> > it's time to come back to it.
> >
> > Shyam, after you reconnected the disks, the messages about failed async
> > metadata
> > writes stops to be logged?
> >
> > Any chance you can reliably reproduce it?
> >
> > I'm not a xfs journal expert, but it looks like the writeback of items
> in
> > AIL
> > got stuck due the IO errors, and were never completed, but I don't know
> > what I
> > should expect after the disk is reconnected.
> >
> > In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN
> until
> > I
> > tried to unmount the filesystem, which differs from this case, where xfs
> > looks
> > to have shutdown the filesystem after a few tries to writeback the
> > metadata.
> >
> > Anyway, I can dig more into it this week, if nobody knows what is going
> on
> > before I do it :)
> >
> > -----Original Message-----
> > From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> > Sent: 22 March 2016 18:32
> > To: 'Brian Foster'
> > Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> > Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > Hi Brian,
> >
> > Thanks for your quick reply. I repeated the test & trace-pipe is
> > constantly filled with this:
> >
> >    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> >
> > while regular activity seems to happen on other inodes/kworker threads
> >
> >     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
> 253:10
> > ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> > 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
> 253:10
> > ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> > 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
> 253:10
> > ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> > 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> > delalloc 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
> 253:10
> > ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> > 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
> delalloc
> > 0 unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
> 253:10
> > ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> > unwritten 0
> >     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> > 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> > delalloc 1 unwritten 0
> >
> >
> > looks like xfsaild is not able to take lock until hung-task timeout
> kicks
> > in
> >
> >    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> > 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> > ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> > 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
> >
> > Please let me know how to debug this further. Thanks.
> >
> > --Shyam
> > -----Original Message-----
> > From: Brian Foster [mailto:bfoster@redhat.com]
> > Sent: 22 March 2016 17:49
> > To: Shyam Kaushik
> > Cc: xfs@oss.sgi.com; Alex Lyakas
> > Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> > after disk failure/recovery
> >
> > On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > > Hi XFS developers,
> > >
> > > We are seeing the following issue with XFS on kernel 3.18.19.
> > >
> > > We have XFS mounted over a raw disk. Disk was pulled out manually.
> There
> > > were async writes on files that were errored like this
> > >
> > ...
> > >
> > > And XFS hit metadata & Log IO errors that it decides to shutdown:
> > >
> > > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > > old_flags=0x0 new_flags=0x2
> > > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
> Error
> > > Detected.  Shutting down filesystem
> > ...
> > > Later the drive was re-inserted back. After the drive was re-inserted,
> > XFS
> > > was attempted to be unmounted
> > >
> > > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > > : umount(/sdisk/vol5b0, xfs)
> > >
> > > But nothing happens except for the 30-secs xfs_log_force errors that
> > keeps
> > > repeating
> > >
> > ...
> > >
> > > This problem doesn't happen consistently, but happens periodically
> with
> > a
> > > drive failure/recovery followed by XFS unmount. I couldn't find this
> > issue
> > > fixed in later kernels. Can you please suggest how I can debug this
> > issue
> > > further?
> > >
> >
> > Similar problems have been reproduced due to racy/incorrect EFI/EFD
> > object tracking, which are internal data structures associated with
> > freeing extents.
> >
> > What happens if you enable tracepoints while the fs is in this hung
> > unmount state?
> >
> > # trace-cmd start -e "xfs:*"
> > # cat /sys/kernel/debug/tracing/trace_pipe
> >
> > Brian
> >
> > > Thanks!
> > >
> > > --Shyam
> > >
> > > _______________________________________________
> > > xfs mailing list
> > > xfs@oss.sgi.com
> > > http://oss.sgi.com/mailman/listinfo/xfs
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Specific problematic flow in xfs_buf_iodone_callbacks() is this

        if (XFS_BUF_ISASYNC(bp)) {
                xfs_buf_ioerror(bp, 0); /* errno of 0 unsets the flag */

                if (!(bp->b_flags & (XBF_STALE|XBF_WRITE_FAIL))) {
                        bp->b_flags |= XBF_WRITE | XBF_ASYNC |
                                       XBF_DONE | XBF_WRITE_FAIL;
                        xfs_buf_submit(bp);
                } else {
                        xfs_buf_relse(bp);
                }

That in the else part we just do xfs_buf_relse(). I am not sure if this is
alright. Pls let me know. Thanks.

--Shyam

-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 08 April 2016 23:19
To: 'Brian Foster'
Cc: 'david@fromorbit.com'; 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Dave, Brian, Carlos,

I think I see the problem. The problem is when the device just starts to
return IO errors & not shutdown yet that buf could get returned with IO
error. We mark it with XBF_WRITE_FAIL & re-submit the IO. But this IO
could fail again & at this point since we already have XBF_WRITE_FAIL we
dont submit the buf, but leave it in a bad state.

w.r.t. trace-buffer

we have xfs_buf_iodone_callbacks() called with bp->b_error & its an async
IO. So we mark XBF_WRITE_FAIL on the buf & do xfs_buf_submit()

    kworker/2:1H-163   [002] ...1  7513.493658: xfs_buf_item_iodone_async:
dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]

but then this again fails & comes back to the same call, but this time
with XBF_WRITE_FAIL set & in this case unfortunately we just do
xfs_buf_relse() & there is no callback flow that is going through.

    kworker/2:1H-163   [002] ...1  7513.494385: xfs_buf_item_iodone_async:
dev 253:11 bno 0x2571850 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]

Do you think this is the problem?

There are several bufs that got hit like this. Here is complete trace for
one such buf

   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
caller xfs_buf_item_push
 [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_item_push: dev
253:11 bno 0x25718d0 len 0x2000 hold 1 pincount 0 lock 0 flags DONE|PAGES
recur 0 refcount 0 bli
flags DIRTY|INODE_ALLOC lidesc 0x          (null) liflags IN_AIL
   xfsaild/dm-11-22098 [002] ...2  7484.466616: xfs_buf_delwri_queue: dev
253:11 bno 0x25718d0 nblks 0x10 hold 1 pincount 0 lock 0 flags DONE|PAGES
caller xfs_buf_item
_push [xfs]
   xfsaild/dm-11-22098 [002] ...2  7484.466617: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_buf_i
tem_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller _xfs_buf
_find [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466765: xfs_buf_find: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_buf_g
et_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_get: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_buf_re
ad_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466766: xfs_buf_read: dev 253:11
bno 0x25718d0 len 0x2000 hold 3 pincount 0 lock 0 flags
READ|TRYLOCK|UNMAPPED caller xfs_trans
_read_buf_map [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_delwri_queued: dev
253:11 bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller xf
s_inode_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_inode
_item_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.466792: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 1 flags
DONE|PAGES|DELWRI_Q caller xfs_inode_i
tem_push [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_trylock: dev
253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller __xfs_bu
f_delwri_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.467962: xfs_buf_delwri_split: dev
253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
DONE|PAGES|DELWRI_Q caller xfs
_buf_delwri_submit_nowait [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_submit: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller __xfs_buf_delwri_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469864: xfs_buf_hold: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
   xfsaild/dm-11-22098 [002] ...1  7484.469868: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_iodone: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_item_iodone_async:
dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|PAGES caller xfs_buf_ioend [xfs]
    kworker/2:1H-163   [002] ...1  7513.493754: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
ASYNC|DONE|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_submit: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.493755: xfs_buf_hold: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.493766: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 3 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_submit [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_iodone: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
WRITE|ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error -5 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend_work [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_item_iodone_async:
dev 253:11 bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_ioend [xfs]
    kworker/2:1H-163   [002] ...1  7513.494398: xfs_buf_ioerror: dev
253:11 bno 0x25718d0 len 0x2000 hold 2 pincount 0 lock 0 error 0 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_unlock: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]
    kworker/2:1H-163   [002] ...1  7513.494399: xfs_buf_rele: dev 253:11
bno 0x25718d0 nblks 0x10 hold 2 pincount 0 lock 1 flags
ASYNC|DONE|WRITE_FAIL|PAGES caller xfs_buf_iodone_callbacks [xfs]


as seen above the first time buf hits ioerror we retry, but the 2nd time
we give it up without finishing the buf. Its in a stuck state & this keeps
xfsaild thread busy when unmounting trying to flush it out.

Thanks.

--Shyam
-----Original Message-----
From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
Sent: 08 April 2016 19:05
To: 'Brian Foster'
Cc: 'david@fromorbit.com'; 'xfs@oss.sgi.com'; Alex Lyakas
Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

Hi Brian,

Yes the below report is a bit different than the original report. I wanted
to confirm if the new patches from Dave will see these odd
crashes/hung-task.

I applied Dave's patch & managed to recreate the original issue. With
systemtap I do see that FS has been marked for shutdown (xfs_aildpush() -
mp:0xffff880062d09800 m_flags:4325656), but still the infinite tries keep
happening with xfsaild_push() seeing XFS_IFLOCK on the inode. Pls note
that this issue happens only when I see

XFS (dm-10): Detected failing async write on buffer block 0x368d76a0.
Retrying async write.

in the logs. In all other cases, this problem doesn't happen. So there is
something with this async-write retries that is leaving the state of the
inode with IFLOCK that rest of aildpush is not able to handle.

--Shyam

-----Original Message-----
From: Brian Foster [mailto:bfoster@redhat.com]
Sent: 08 April 2016 18:47
To: Shyam Kaushik
Cc: david@fromorbit.com; xfs@oss.sgi.com; Alex Lyakas
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
>
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend
to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
>
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
>
> while running trace-events, XFS ail push keeps looping around
>
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.555138: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-21143 [001] ...2 17878.605136: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605141: xfs_iunlock: dev 253:10
> ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-21143 [001] ...2 17878.605142: xfs_ail_push: dev 253:10
> lip 0xffff880031f6ae40 lsn 1/73448 type XFS_LI_INODE flags IN_AIL
>
> observe ino==0x0 (its a freed inode) but the corresponding lip stays
with
> xfsaild push. I had run a systemtap when this problem happened & the
> following keeps repeating.
>

That's interesting, and looks different from the original report in
terms of the inode number being cleared. The original report looks like
it has a valid inode and there's some problematic sequence where it's
not being removed from the AIL in the event of errors.

I think at this point we know that XFS attempts to retry these I/Os
indefinitely. Dave has outstanding patches to deal with this issue. The
question Dave raised above was whether the filesystem shut down, and if
not, why (as a failed log I/O should cause a shutdown)? Carlos was
looking into this it appears...

> xfs_inode_item_push() - inode:0xffff88003d8ca000 m_flags:0x420118
> flags:0x1
> xfs_iflush() - inode:ffff88003d8ca000 aborting for forced shutdown
> xfs_iflush_abort() - iip:0x0
>
> xfs_inode_item_push() is doing xfs_iflush() but it sees FS_SHUTDOWN flag
> marked & does an xfs_iflush_abort().xfs_iflush_abort() sees
i_itemp==NULL
> on the lip & doesn't do anything. So we have this stale lip that is not
> associated to inode anymore that keeps xfs_ail_push_all_sync() busy.
>
> On few other occurrences, i have had NULL pointer/dereference or other
> sorts of crashes at this line
>
> 	xfsaild_push()
> 		lock_result = lip->li_ops->iop_push(lip,
> &ailp->xa_buf_list);
>
> with debug prints, in one of the occurrence lip->li_ops was NULL & in
> another lip->li_ops was pointing to a bad pointer that subsequent
> dereference crashed it. This also indicates that a bad/freed lip was
> inserted that xfsaild_push() is working.
>
> I hit upon Dave's patch "xfs: xfs_inode_free() isn't RCU safe" &
realized
> that this could explain the above issue where a lip that has been freed
is
> wrongly referenced & later we could even have the inode disconnected.
What
> do you think?
>
> In any case I uploaded the full xfs trace events before this issue
started
> & till the end. It is at
> https://www.dropbox.com/s/21qstz4ld1gn8yi/xfs_trace_pipe.gz?dl=0
>

It seems like it could be related. I didn't catch anything obvious from
the trace, but there's a lot of data there. The RCU-unsafe issue was
difficult to track down without instrumentation.. I'm not sure that will
be evident from the trace. The best thing to do wrt to that might be to
try with Dave's patches, as that so far appears to address the problem.
(In fact, it might be worth it to try Dave's shutdown on umount patch
he referred to up-thread to address the original problem as well).

Brian

> Pls let me know. Thanks.
>
> --Shyam
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 23 March 2016 15:23
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Carlos,
>
> w.r.t. your question below
>
> >>> Shyam, after you reconnected the disks, the messages about failed
> async metadata
> >>> writes stops to be logged?
>
> After I reconnect the disks, messages about failed async metadata writes
> stops to be logged. But the key thing is messages like
>
> XFS (dm-29): xfs_log_force: error -5 returned
>
> Keeps repeating every 30-secs which indicates that there is some buf/io
> error status that is being carried forward.
>
> >>> Any chance you can reliably reproduce it?
>
> Yes I have a way to reproduce it, but its not reliable. What I do is
setup
> a dm-linear over a disk. Create XFS, mount & trigger few copy threads to
> copy varying sized files into the FS. At this point pull out the drive
> (with scsi-remove-single-device in /proc/scsi/scsi) & in a short-while
> convert the dm-linear to dm-ioerror. Then I bring back the underlying
> drive, convert back dm-ioerror to dm-linear & try to unmount XFS.
>
> This problem somehow happens on a newly created XFS. If I copy several
> files into XFS/delete them & then copy them again, repeat the drive
> failure/recovery experiment it doesn't reproduce.
>
> Thanks.
>
> --Shyam
>
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
> From: Carlos Maiolino <cmaiolino@xxxxxxxxxx>
> Date: Tue, 22 Mar 2016 16:38:25 +0100
> In-reply-to: <20160322140345.GA54245@bfoster.bfoster>
> Mail-followup-to: xfs@xxxxxxxxxxx
> User-agent: Mutt/1.5.24 (2015-08-30)
>
> Hi Brian,
>
> These traces, and the stack trace presented, looks quite similar with
the
> one we were discussing a few days ago, using a dm-thin snapshot.
>
> Looks like with the same bug I've been hunting and Shyam confirmed my
> hypothesis
> of this bug be able to be reproduced with a regular device.
>
> If it's the same bug, yes, I reproduced it using upstream kernel.
>
> The difference between both (this bug and the one I've been working on)
is
> how
> xfs actually behaves when async metadata writes fail. Other than that,
it
> pretty
> much looks the same.
>
> Trying to unmount the filesystem hungs in xfs_log_force(), well,
basically
> the
> reason I submitted the patch to include the caller into xfs_log_force
> trace. I'd
> like to see ftrace traces from this system with that patch if possible.
>
> I didn't have time to keep working on this for the past few days, but
> looks like
> it's time to come back to it.
>
> Shyam, after you reconnected the disks, the messages about failed async
> metadata
> writes stops to be logged?
>
> Any chance you can reliably reproduce it?
>
> I'm not a xfs journal expert, but it looks like the writeback of items
in
> AIL
> got stuck due the IO errors, and were never completed, but I don't know
> what I
> should expect after the disk is reconnected.
>
> In my case though, with upstream kernel, I didn't get a XFS_SHUTDOWN
until
> I
> tried to unmount the filesystem, which differs from this case, where xfs
> looks
> to have shutdown the filesystem after a few tries to writeback the
> metadata.
>
> Anyway, I can dig more into it this week, if nobody knows what is going
on
> before I do it :)
>
> -----Original Message-----
> From: Shyam Kaushik [mailto:shyam@zadarastorage.com]
> Sent: 22 March 2016 18:32
> To: 'Brian Foster'
> Cc: 'xfs@oss.sgi.com'; Alex Lyakas
> Subject: RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> Hi Brian,
>
> Thanks for your quick reply. I repeated the test & trace-pipe is
> constantly filled with this:
>
>    xfsaild/dm-10-3335  [003] ...2 24890.546491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.546493: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.596491: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596492: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.596494: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.646497: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646498: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.646500: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.696467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.696468: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.746548: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.746550: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.796479: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.796480: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 24890.846467: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 24890.846468: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>
>
> while regular activity seems to happen on other inodes/kworker threads
>
>     kworker/u8:4-27691 [001] ...1 24895.811474: xfs_writepage: dev
253:10
> ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811477: xfs_invalidatepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811478: xfs_releasepage: dev
> 253:10 ino 0x1801061 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_writepage: dev
253:10
> ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811482: xfs_invalidatepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811483: xfs_releasepage: dev
> 253:10 ino 0x4017bdf pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811485: xfs_writepage: dev
253:10
> ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_invalidatepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.811486: xfs_releasepage: dev
> 253:10 ino 0x68048c3 pgoff 0x29000 size 0x1aebbc offset 0 length 0
> delalloc 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812381: xfs_writepage: dev
253:10
> ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_invalidatepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812382: xfs_releasepage: dev
> 253:10 ino 0x1805e37 pgoff 0x29000 size 0x68470 offset 0 length 0
delalloc
> 0 unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_writepage: dev
253:10
> ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 0 delalloc 1
> unwritten 0
>     kworker/u8:4-27691 [001] ...1 24895.812385: xfs_invalidatepage: dev
> 253:10 ino 0x4019c95 pgoff 0x29000 size 0x68470 offset 0 length 1000
> delalloc 1 unwritten 0
>
>
> looks like xfsaild is not able to take lock until hung-task timeout
kicks
> in
>
>    xfsaild/dm-10-3335  [003] ...2 25247.649468: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.649469: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.699478: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699516: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.699517: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>    xfsaild/dm-10-3335  [003] ...2 25247.749471: xfs_ilock_nowait: dev
> 253:10 ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749478: xfs_iunlock: dev 253:10
> ino 0xc0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>    xfsaild/dm-10-3335  [003] ...2 25247.749479: xfs_ail_flushing: dev
> 253:10 lip 0xffff8800a9f437b8 lsn 1/38624 type XFS_LI_INODE flags IN_AIL
>
> Please let me know how to debug this further. Thanks.
>
> --Shyam
> -----Original Message-----
> From: Brian Foster [mailto:bfoster@redhat.com]
> Sent: 22 March 2016 17:49
> To: Shyam Kaushik
> Cc: xfs@oss.sgi.com; Alex Lyakas
> Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
> after disk failure/recovery
>
> On Tue, Mar 22, 2016 at 04:51:39PM +0530, Shyam Kaushik wrote:
> > Hi XFS developers,
> >
> > We are seeing the following issue with XFS on kernel 3.18.19.
> >
> > We have XFS mounted over a raw disk. Disk was pulled out manually.
There
> > were async writes on files that were errored like this
> >
> ...
> >
> > And XFS hit metadata & Log IO errors that it decides to shutdown:
> >
> > Mar 16 16:03:22 host0 kernel: [ 4637.351841] XFS (dm-29): metadata I/O
> > error: block 0x3a27fbd0 ("xlog_iodone") error 5 numblks 64
> > Mar 16 16:03:22 host0 kernel: [ 4637.352820] XFS(dm-29): SHUTDOWN!!!
> > old_flags=0x0 new_flags=0x2
> > Mar 16 16:03:22 host0 kernel: [ 4637.353187] XFS (dm-29): Log I/O
Error
> > Detected.  Shutting down filesystem
> ...
> > Later the drive was re-inserted back. After the drive was re-inserted,
> XFS
> > was attempted to be unmounted
> >
> > Mar 16 16:16:53 host0 controld: [2557] [     ] umount[202]
> > : umount(/sdisk/vol5b0, xfs)
> >
> > But nothing happens except for the 30-secs xfs_log_force errors that
> keeps
> > repeating
> >
> ...
> >
> > This problem doesn't happen consistently, but happens periodically
with
> a
> > drive failure/recovery followed by XFS unmount. I couldn't find this
> issue
> > fixed in later kernels. Can you please suggest how I can debug this
> issue
> > further?
> >
>
> Similar problems have been reproduced due to racy/incorrect EFI/EFD
> object tracking, which are internal data structures associated with
> freeing extents.
>
> What happens if you enable tracepoints while the fs is in this hung
> unmount state?
>
> # trace-cmd start -e "xfs:*"
> # cat /sys/kernel/debug/tracing/trace_pipe
>
> Brian
>
> > Thanks!
> >
> > --Shyam
> >
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> Hi Dave, Brian, Carlos,
> 
> While trying to reproduce this issue I have been running into different
> issues that are similar. Underlying issue remains the same when backend to
> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> with stack like
> 
> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> prepare_to_wait_event+0x110/0x110
> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> [xfs]
> 
> while running trace-events, XFS ail push keeps looping around
> 
>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]

Looks like either a stale inode (which should never reach the AIL)
or it's an inode that's been reclaimed and this is a use after free
situation. Given that we are failing IOs here, I'd suggest it's more
likely to be an IO failure that's caused a writeback problem, not an
interaction with stale inodes.

So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
inode by calling xfs_iflush_abort(), which will also remove it from
the AIL. This can also happen on reclaim of a dirty inode, and if so
we'll still reclaim the inode because reclaim assumes xfs_iflush()
cleans up properly.

Which, apparently, it doesn't:

        /*
         * Get the buffer containing the on-disk inode.
         */
        error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp, XBF_TRYLOCK, 0);
        if (error || !bp) {
                xfs_ifunlock(ip);
                return error;
        }

This looks like a bug - xfs_iflush hasn't aborted the inode
writeback on failure - it's just unlocked the flush lock. Hence it
has left the inode dirty in the AIL, and then the inode has probably
then been reclaimed, setting the inode number to zero.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Alex Lyakas]

Hello Dave,

On Sat, Apr 9, 2016 at 1:46 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
>> Hi Dave, Brian, Carlos,
>>
>> While trying to reproduce this issue I have been running into different
>> issues that are similar. Underlying issue remains the same when backend to
>> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
>> with stack like
>>
>> kernel: [14952.671131]  [<ffffffffc06a5f59>]
>> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
>> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
>> prepare_to_wait_event+0x110/0x110
>> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
>> [xfs]
>>
>> while running trace-events, XFS ail push keeps looping around
>>
>>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
>> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
>
> Looks like either a stale inode (which should never reach the AIL)
> or it's an inode that's been reclaimed and this is a use after free
> situation. Given that we are failing IOs here, I'd suggest it's more
> likely to be an IO failure that's caused a writeback problem, not an
> interaction with stale inodes.
>
> So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
> inode by calling xfs_iflush_abort(), which will also remove it from
> the AIL. This can also happen on reclaim of a dirty inode, and if so
> we'll still reclaim the inode because reclaim assumes xfs_iflush()
> cleans up properly.
>
> Which, apparently, it doesn't:
>
>         /*
>          * Get the buffer containing the on-disk inode.
>          */
>         error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp, XBF_TRYLOCK, 0);
>         if (error || !bp) {
>                 xfs_ifunlock(ip);
>                 return error;
>         }
>
> This looks like a bug - xfs_iflush hasn't aborted the inode
> writeback on failure - it's just unlocked the flush lock. Hence it
> has left the inode dirty in the AIL, and then the inode has probably
> then been reclaimed, setting the inode number to zero.
In our case, we do not reach this call, because XFS is already marked
as "shutdown", so in our case we do:
    /*
     * This may have been unpinned because the filesystem is shutting
     * down forcibly. If that's the case we must not write this inode
     * to disk, because the log record didn't make it to disk.
     *
     * We also have to remove the log item from the AIL in this case,
     * as we wait for an empty AIL as part of the unmount process.
     */
    if (XFS_FORCED_SHUTDOWN(mp)) {
        error = -EIO;
        goto abort_out;
    }

So we call xfs_iflush_abort, but due to "iip" being NULL (as Shyam
mentioned earlier in this thread), we proceed directly to
xfs_ifunlock(ip), which now becomes the same situation as you
described above.

The comment clearly says "We also have to remove the log item from the
AIL in this case, as we wait for an empty AIL as part of the unmount
process." Could you perhaps point us at the code that is supposed to
remove the log item from the AIL? Apparently this is not happening.

Thanks,
Alex.



>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Sun, Apr 10, 2016 at 09:40:29PM +0300, Alex Lyakas wrote:
> Hello Dave,
> 
> On Sat, Apr 9, 2016 at 1:46 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> >> Hi Dave, Brian, Carlos,
> >>
> >> While trying to reproduce this issue I have been running into different
> >> issues that are similar. Underlying issue remains the same when backend to
> >> XFS is failed & we unmount XFS, we run into hung-task timeout (180-secs)
> >> with stack like
> >>
> >> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> >> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> >> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> >> prepare_to_wait_event+0x110/0x110
> >> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> >> [xfs]
> >>
> >> while running trace-events, XFS ail push keeps looping around
> >>
> >>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> >> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> > Looks like either a stale inode (which should never reach the AIL)
> > or it's an inode that's been reclaimed and this is a use after free
> > situation. Given that we are failing IOs here, I'd suggest it's more
> > likely to be an IO failure that's caused a writeback problem, not an
> > interaction with stale inodes.
> >
> > So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
> > inode by calling xfs_iflush_abort(), which will also remove it from
> > the AIL. This can also happen on reclaim of a dirty inode, and if so
> > we'll still reclaim the inode because reclaim assumes xfs_iflush()
> > cleans up properly.
> >
> > Which, apparently, it doesn't:
> >
> >         /*
> >          * Get the buffer containing the on-disk inode.
> >          */
> >         error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp, XBF_TRYLOCK, 0);
> >         if (error || !bp) {
> >                 xfs_ifunlock(ip);
> >                 return error;
> >         }
> >
> > This looks like a bug - xfs_iflush hasn't aborted the inode
> > writeback on failure - it's just unlocked the flush lock. Hence it
> > has left the inode dirty in the AIL, and then the inode has probably
> > then been reclaimed, setting the inode number to zero.
> In our case, we do not reach this call, because XFS is already marked
> as "shutdown", so in our case we do:
>     /*
>      * This may have been unpinned because the filesystem is shutting
>      * down forcibly. If that's the case we must not write this inode
>      * to disk, because the log record didn't make it to disk.
>      *
>      * We also have to remove the log item from the AIL in this case,
>      * as we wait for an empty AIL as part of the unmount process.
>      */
>     if (XFS_FORCED_SHUTDOWN(mp)) {
>         error = -EIO;
>         goto abort_out;
>     }
> 
> So we call xfs_iflush_abort, but due to "iip" being NULL (as Shyam
> mentioned earlier in this thread), we proceed directly to
> xfs_ifunlock(ip), which now becomes the same situation as you
> described above.

If you are getting this occuring, something else has already gone
wrong as you can't have a dirty inode without a log item attached to
it. So it appears to me that you are reporting a symptom of a
problem after it has occured, not the root cause. Maybe it is the
same root cause, maybe not. Either way, it doesn't help us solve any
problem.

> The comment clearly says "We also have to remove the log item from the
> AIL in this case, as we wait for an empty AIL as part of the unmount
> process." Could you perhaps point us at the code that is supposed to
> remove the log item from the AIL? Apparently this is not happening.

xfs_iflush_abort or xfs_iflush_done does that work. 

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Dave,

Do you plan to post a patch for the bug you discovered in xfs_iflush() to
goto abort_out when xfs_imap_to_bp() fails?

We can include this patch & see we can hit a recreate of this issue.

Thanks.

--Shyam

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com]
Sent: 11 April 2016 06:51
To: Alex Lyakas
Cc: Shyam Kaushik; Brian Foster; xfs@oss.sgi.com
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Sun, Apr 10, 2016 at 09:40:29PM +0300, Alex Lyakas wrote:
> Hello Dave,
>
> On Sat, Apr 9, 2016 at 1:46 AM, Dave Chinner <david@fromorbit.com>
wrote:
> > On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> >> Hi Dave, Brian, Carlos,
> >>
> >> While trying to reproduce this issue I have been running into
different
> >> issues that are similar. Underlying issue remains the same when
backend to
> >> XFS is failed & we unmount XFS, we run into hung-task timeout
(180-secs)
> >> with stack like
> >>
> >> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> >> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> >> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> >> prepare_to_wait_event+0x110/0x110
> >> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> >> [xfs]
> >>
> >> while running trace-events, XFS ail push keeps looping around
> >>
> >>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> >> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> > Looks like either a stale inode (which should never reach the AIL)
> > or it's an inode that's been reclaimed and this is a use after free
> > situation. Given that we are failing IOs here, I'd suggest it's more
> > likely to be an IO failure that's caused a writeback problem, not an
> > interaction with stale inodes.
> >
> > So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
> > inode by calling xfs_iflush_abort(), which will also remove it from
> > the AIL. This can also happen on reclaim of a dirty inode, and if so
> > we'll still reclaim the inode because reclaim assumes xfs_iflush()
> > cleans up properly.
> >
> > Which, apparently, it doesn't:
> >
> >         /*
> >          * Get the buffer containing the on-disk inode.
> >          */
> >         error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp,
XBF_TRYLOCK, 0);
> >         if (error || !bp) {
> >                 xfs_ifunlock(ip);
> >                 return error;
> >         }
> >
> > This looks like a bug - xfs_iflush hasn't aborted the inode
> > writeback on failure - it's just unlocked the flush lock. Hence it
> > has left the inode dirty in the AIL, and then the inode has probably
> > then been reclaimed, setting the inode number to zero.
> In our case, we do not reach this call, because XFS is already marked
> as "shutdown", so in our case we do:
>     /*
>      * This may have been unpinned because the filesystem is shutting
>      * down forcibly. If that's the case we must not write this inode
>      * to disk, because the log record didn't make it to disk.
>      *
>      * We also have to remove the log item from the AIL in this case,
>      * as we wait for an empty AIL as part of the unmount process.
>      */
>     if (XFS_FORCED_SHUTDOWN(mp)) {
>         error = -EIO;
>         goto abort_out;
>     }
>
> So we call xfs_iflush_abort, but due to "iip" being NULL (as Shyam
> mentioned earlier in this thread), we proceed directly to
> xfs_ifunlock(ip), which now becomes the same situation as you
> described above.

If you are getting this occuring, something else has already gone
wrong as you can't have a dirty inode without a log item attached to
it. So it appears to me that you are reporting a symptom of a
problem after it has occured, not the root cause. Maybe it is the
same root cause, maybe not. Either way, it doesn't help us solve any
problem.

> The comment clearly says "We also have to remove the log item from the
> AIL in this case, as we wait for an empty AIL as part of the unmount
> process." Could you perhaps point us at the code that is supposed to
> remove the log item from the AIL? Apparently this is not happening.

xfs_iflush_abort or xfs_iflush_done does that work.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Mon, Apr 11, 2016 at 08:22:14PM +0530, Shyam Kaushik wrote:
> Hi Dave,
> 
> Do you plan to post a patch for the bug you discovered in xfs_iflush() to
> goto abort_out when xfs_imap_to_bp() fails?

I'll get to it, but I can't do everything for everyone at once.

There is enough information in this thread that it shoul dbe
straight forward to write the two line patch to jump to the correct
error handling code, so the fact I haven't written the trivial
change yet should not be holding anyone up...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Carlos,

xfs.stap that I have been using is here
(https://www.dropbox.com/s/dluse4s3a1c7dj3/xfs.stap?dl=0). It has
line-numbers matching xfs with printks I added. You will have to tweak it
a bit.

Thanks.

--Shyam

-----Original Message-----
From: Carlos Maiolino
Sent: Fri Apr 8 09:31:31 CDT 2016

> Hi Shyam,
>
> do you mind to share your systemtap script with us?
>
> I'd like to take a look on it, and probably Brian will be interested to.
-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com]
Sent: 11 April 2016 06:51
To: Alex Lyakas
Cc: Shyam Kaushik; Brian Foster; xfs@oss.sgi.com
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Sun, Apr 10, 2016 at 09:40:29PM +0300, Alex Lyakas wrote:
> Hello Dave,
>
> On Sat, Apr 9, 2016 at 1:46 AM, Dave Chinner <david@fromorbit.com>
wrote:
> > On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> >> Hi Dave, Brian, Carlos,
> >>
> >> While trying to reproduce this issue I have been running into
different
> >> issues that are similar. Underlying issue remains the same when
backend to
> >> XFS is failed & we unmount XFS, we run into hung-task timeout
(180-secs)
> >> with stack like
> >>
> >> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> >> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> >> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> >> prepare_to_wait_event+0x110/0x110
> >> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> >> [xfs]
> >>
> >> while running trace-events, XFS ail push keeps looping around
> >>
> >>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> >> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> > Looks like either a stale inode (which should never reach the AIL)
> > or it's an inode that's been reclaimed and this is a use after free
> > situation. Given that we are failing IOs here, I'd suggest it's more
> > likely to be an IO failure that's caused a writeback problem, not an
> > interaction with stale inodes.
> >
> > So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
> > inode by calling xfs_iflush_abort(), which will also remove it from
> > the AIL. This can also happen on reclaim of a dirty inode, and if so
> > we'll still reclaim the inode because reclaim assumes xfs_iflush()
> > cleans up properly.
> >
> > Which, apparently, it doesn't:
> >
> >         /*
> >          * Get the buffer containing the on-disk inode.
> >          */
> >         error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp,
XBF_TRYLOCK, 0);
> >         if (error || !bp) {
> >                 xfs_ifunlock(ip);
> >                 return error;
> >         }
> >
> > This looks like a bug - xfs_iflush hasn't aborted the inode
> > writeback on failure - it's just unlocked the flush lock. Hence it
> > has left the inode dirty in the AIL, and then the inode has probably
> > then been reclaimed, setting the inode number to zero.
> In our case, we do not reach this call, because XFS is already marked
> as "shutdown", so in our case we do:
>     /*
>      * This may have been unpinned because the filesystem is shutting
>      * down forcibly. If that's the case we must not write this inode
>      * to disk, because the log record didn't make it to disk.
>      *
>      * We also have to remove the log item from the AIL in this case,
>      * as we wait for an empty AIL as part of the unmount process.
>      */
>     if (XFS_FORCED_SHUTDOWN(mp)) {
>         error = -EIO;
>         goto abort_out;
>     }
>
> So we call xfs_iflush_abort, but due to "iip" being NULL (as Shyam
> mentioned earlier in this thread), we proceed directly to
> xfs_ifunlock(ip), which now becomes the same situation as you
> described above.

If you are getting this occuring, something else has already gone
wrong as you can't have a dirty inode without a log item attached to
it. So it appears to me that you are reporting a symptom of a
problem after it has occured, not the root cause. Maybe it is the
same root cause, maybe not. Either way, it doesn't help us solve any
problem.

> The comment clearly says "We also have to remove the log item from the
> AIL in this case, as we wait for an empty AIL as part of the unmount
> process." Could you perhaps point us at the code that is supposed to
> remove the log item from the AIL? Apparently this is not happening.

xfs_iflush_abort or xfs_iflush_done does that work.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Shyam Kaushik]

Hi Dave,

I posted the patch.

After applying this patch & your use-after-free patch,
xfs_ail_push_all_sync() processing intent-logs that are no longer relevant
upon disk failure/recovery doesn't happen anymore. I tried several times
to recreate and the issue doesn't show up.

Thanks.

--Shyam

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com]
Sent: 11 April 2016 06:51
To: Alex Lyakas
Cc: Shyam Kaushik; Brian Foster; xfs@oss.sgi.com
Subject: Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS
after disk failure/recovery

On Sun, Apr 10, 2016 at 09:40:29PM +0300, Alex Lyakas wrote:
> Hello Dave,
>
> On Sat, Apr 9, 2016 at 1:46 AM, Dave Chinner <david@fromorbit.com>
wrote:
> > On Fri, Apr 08, 2016 at 04:21:02PM +0530, Shyam Kaushik wrote:
> >> Hi Dave, Brian, Carlos,
> >>
> >> While trying to reproduce this issue I have been running into
different
> >> issues that are similar. Underlying issue remains the same when
backend to
> >> XFS is failed & we unmount XFS, we run into hung-task timeout
(180-secs)
> >> with stack like
> >>
> >> kernel: [14952.671131]  [<ffffffffc06a5f59>]
> >> xfs_ail_push_all_sync+0xa9/0xe0 [xfs]
> >> kernel: [14952.671139]  [<ffffffff810b26b0>] ?
> >> prepare_to_wait_event+0x110/0x110
> >> kernel: [14952.671181]  [<ffffffffc0690111>] xfs_unmountfs+0x61/0x1a0
> >> [xfs]
> >>
> >> while running trace-events, XFS ail push keeps looping around
> >>
> >>    xfsaild/dm-10-21143 [001] ...2 17878.555133: xfs_ilock_nowait: dev
> >> 253:10 ino 0x0 flags ILOCK_SHARED caller xfs_inode_item_push [xfs]
> >
> > Looks like either a stale inode (which should never reach the AIL)
> > or it's an inode that's been reclaimed and this is a use after free
> > situation. Given that we are failing IOs here, I'd suggest it's more
> > likely to be an IO failure that's caused a writeback problem, not an
> > interaction with stale inodes.
> >
> > So, look at xfs_iflush. If an IO fails, it is supposed to unlock the
> > inode by calling xfs_iflush_abort(), which will also remove it from
> > the AIL. This can also happen on reclaim of a dirty inode, and if so
> > we'll still reclaim the inode because reclaim assumes xfs_iflush()
> > cleans up properly.
> >
> > Which, apparently, it doesn't:
> >
> >         /*
> >          * Get the buffer containing the on-disk inode.
> >          */
> >         error = xfs_imap_to_bp(mp, NULL, &ip->i_imap, &dip, &bp,
XBF_TRYLOCK, 0);
> >         if (error || !bp) {
> >                 xfs_ifunlock(ip);
> >                 return error;
> >         }
> >
> > This looks like a bug - xfs_iflush hasn't aborted the inode
> > writeback on failure - it's just unlocked the flush lock. Hence it
> > has left the inode dirty in the AIL, and then the inode has probably
> > then been reclaimed, setting the inode number to zero.
> In our case, we do not reach this call, because XFS is already marked
> as "shutdown", so in our case we do:
>     /*
>      * This may have been unpinned because the filesystem is shutting
>      * down forcibly. If that's the case we must not write this inode
>      * to disk, because the log record didn't make it to disk.
>      *
>      * We also have to remove the log item from the AIL in this case,
>      * as we wait for an empty AIL as part of the unmount process.
>      */
>     if (XFS_FORCED_SHUTDOWN(mp)) {
>         error = -EIO;
>         goto abort_out;
>     }
>
> So we call xfs_iflush_abort, but due to "iip" being NULL (as Shyam
> mentioned earlier in this thread), we proceed directly to
> xfs_ifunlock(ip), which now becomes the same situation as you
> described above.

If you are getting this occuring, something else has already gone
wrong as you can't have a dirty inode without a log item attached to
it. So it appears to me that you are reporting a symptom of a
problem after it has occured, not the root cause. Maybe it is the
same root cause, maybe not. Either way, it doesn't help us solve any
problem.

> The comment clearly says "We also have to remove the log item from the
> AIL in this case, as we wait for an empty AIL as part of the unmount
> process." Could you perhaps point us at the code that is supposed to
> remove the log item from the AIL? Apparently this is not happening.

xfs_iflush_abort or xfs_iflush_done does that work.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Dave Chinner]

On Tue, Apr 12, 2016 at 12:29:48PM +0530, Shyam Kaushik wrote:
> Hi Dave,
> 
> I posted the patch.
> 
> After applying this patch & your use-after-free patch,
> xfs_ail_push_all_sync() processing intent-logs that are no longer relevant
> upon disk failure/recovery doesn't happen anymore. I tried several times
> to recreate and the issue doesn't show up.

OK, so that confirms we are leaking inodes at that spot. I'll have a
look at the patch....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs