Possible deadlock when deleting from realtime section

Possible deadlock when deleting from realtime section

[Denny Priebe]

Hi,

I'm experiencing a deadlock situation when deleting directories placed in
the realtime section. This is reproducable with kernel versions 2.6.35.7 and
2.6.36-rc8. I haven't tried final 2.6.36 yet. The same setup is working
perfectly without using the realtime section. The file system has been
created with

 mkfs.xfs -f -l logdev=/dev/sdb1 -r rtdev=/dev/sdb3,extsize=256k /dev/sdb2

and mounted with

 mount -t xfs -o logdev=/dev/sdb1,rtdev=/dev/sdb3,sunit=512,swidth=2048 \
 /dev/sdb2 /there

This is where rm blocks:

SysRq : Show Blocked State
 task                        PC stack   pid father
rm            D 0000000000000005     0  1705   1658 0x00000080
ffff88018c40d858 0000000000000086 ffff880100000000 ffff88018c40c010
ffff88018c40dfd8 00000000000148c0 ffff88018b1c2e20 ffff88018b1c31d8
ffff88018b1c31d0 00000000000148c0 00000000000148c0 ffff88018c40dfd8
Call Trace:
[<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
[<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
[<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff81436e3d>] ? down_write+0x40/0x44
[<ffffffffa033be9b>] xfs_ilock+0x4a/0x9a [xfs]
[<ffffffffa033c413>] xfs_iget+0x34c/0x5cd [xfs]
[<ffffffffa0351fa9>] xfs_trans_iget+0x1b/0x56 [xfs]
[<ffffffffa0313035>] xfs_rtfree_extent+0x37/0xdc [xfs]
[<ffffffffa033fce2>] ? xfs_iext_remove+0xc0/0xd2 [xfs]
[<ffffffffa0324ffe>] ? xfs_bmap_del_extent+0x2e8/0x93d [xfs]
[<ffffffffa0324e87>] xfs_bmap_del_extent+0x171/0x93d [xfs]
[<ffffffffa03506eb>] ? xfs_trans_commit_iclog+0x2ba/0x2d3 [xfs]
[<ffffffffa0325dc3>] xfs_bunmapi+0x770/0xa28 [xfs]
[<ffffffffa033e44e>] xfs_itruncate_finish+0x185/0x2b8 [xfs]
[<ffffffffa0354437>] xfs_inactive+0x1c8/0x3d5 [xfs]
[<ffffffffa035f38c>] xfs_fs_evict_inode+0xd5/0xdd [xfs]
[<ffffffff8111282b>] evict+0x22/0x92
[<ffffffff81112c60>] iput+0x1bc/0x225
[<ffffffff8110b079>] do_unlinkat+0x103/0x156
[<ffffffff811086c5>] ? path_put+0x1d/0x22
[<ffffffff81093e2c>] ? audit_syscall_entry+0x119/0x145
[<ffffffff8110b205>] sys_unlinkat+0x24/0x26
[<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b

Thanks in advance for any feedback,
Denny

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Possible deadlock when deleting from realtime section

[Denny Priebe]

After that, recovery hangs as well:

SGI XFS with ACLs, security attributes, realtime, large block/inode
numbers, debug enabled
SGI XFS Quota Management subsystem
Filesystem "sdb2": Disabling barriers, not supported with external log device
XFS mounting filesystem sdb2
Starting XFS recovery on filesystem: sdb2 (logdev: /dev/sdb1)
SysRq : Show Blocked State
  task                        PC stack   pid father
mount         D 0000000000000000     0  1481   1464 0x00000080
 ffff880186c17558 0000000000000086 ffff880100000000 ffff880186c16010
 ffff880186c17fd8 00000000000148c0 ffff880187364530 ffff8801873648e8
 ffff8801873648e0 00000000000148c0 00000000000148c0 ffff880186c17fd8
Call Trace:
 [<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
 [<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
 [<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
 [<ffffffff81436e3d>] ? down_write+0x40/0x44
 [<ffffffffa032b7d0>] xfs_ilock+0xb7/0x114 [xfs]
 [<ffffffffa032bead>] xfs_iget+0x34c/0x61b [xfs]
 [<ffffffffa03479b7>] xfs_trans_iget+0x1b/0x54 [xfs]
 [<ffffffffa02f588e>] xfs_rtfree_extent+0x37/0x117 [xfs]
 [<ffffffffa03311f2>] ? xfs_iext_remove+0xdd/0xef [xfs]
 [<ffffffffa030b52f>] ? xfs_bmap_del_extent+0x40c/0xab5 [xfs]
 [<ffffffffa030b3ad>] xfs_bmap_del_extent+0x28a/0xab5 [xfs]
 [<ffffffffa030dce8>] xfs_bunmapi+0xa0c/0xdd2 [xfs]
 [<ffffffff810f17ef>] ? new_slab+0x199/0x1c0
 [<ffffffffa032e26a>] xfs_itruncate_finish+0x32e/0x4f1 [xfs]
 [<ffffffffa034a5d1>] xfs_inactive+0x224/0x47a [xfs]
 [<ffffffffa035615d>] xfs_fs_evict_inode+0x101/0x109 [xfs]
 [<ffffffff8111282b>] evict+0x22/0x92
 [<ffffffff81112c60>] iput+0x1bc/0x225
 [<ffffffffa033b84a>] xlog_recover_process_one_iunlink+0x118/0x17e [xfs]
 [<ffffffffa033b930>] xlog_recover_process_iunlinks+0x80/0xf7 [xfs]
 [<ffffffffa033bc5b>] xlog_recover_finish+0x54/0xbc [xfs]
 [<ffffffffa0339e46>] xfs_log_mount_finish+0x1c/0x44 [xfs]
 [<ffffffffa0342832>] xfs_mountfs+0x559/0x683 [xfs]
 [<ffffffffa034c569>] ? kmem_zalloc+0x11/0x2c [xfs]
 [<ffffffffa0342bfb>] ? xfs_mru_cache_create+0x117/0x146 [xfs]
 [<ffffffffa0357a03>] xfs_fs_fill_super+0x1d9/0x316 [xfs]
 [<ffffffff81102c48>] get_sb_bdev+0x14b/0x1ae
 [<ffffffffa035782a>] ? xfs_fs_fill_super+0x0/0x316 [xfs]
 [<ffffffffa0355c54>] xfs_fs_get_sb+0x13/0x15 [xfs]
 [<ffffffff8110229f>] vfs_kern_mount+0xb8/0x1a2
 [<ffffffff811023e7>] do_kern_mount+0x48/0xe8
 [<ffffffff811175ef>] do_mount+0x74f/0x7c6
 [<ffffffff811176e9>] sys_mount+0x83/0xbd
 [<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b

Regards,
Denny

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Possible deadlock when deleting from realtime section

[Dave Chinner]

On Mon, Oct 25, 2010 at 03:59:22PM +0000, Denny Priebe wrote:
> Hi,
> 
> I'm experiencing a deadlock situation when deleting directories placed in
> the realtime section. This is reproducable with kernel versions 2.6.35.7 and
> 2.6.36-rc8. I haven't tried final 2.6.36 yet. The same setup is working
> perfectly without using the realtime section. The file system has been
> created with
> 
>  mkfs.xfs -f -l logdev=/dev/sdb1 -r rtdev=/dev/sdb3,extsize=256k /dev/sdb2

FYI, using an external log on the same device as the main device will be
slower than using the default internal log....

> and mounted with
> 
>  mount -t xfs -o logdev=/dev/sdb1,rtdev=/dev/sdb3,sunit=512,swidth=2048 \
>  /dev/sdb2 /there
> 
> This is where rm blocks:
> 
> SysRq : Show Blocked State
>  task                        PC stack   pid father
> rm            D 0000000000000005     0  1705   1658 0x00000080
> ffff88018c40d858 0000000000000086 ffff880100000000 ffff88018c40c010
> ffff88018c40dfd8 00000000000148c0 ffff88018b1c2e20 ffff88018b1c31d8
> ffff88018b1c31d0 00000000000148c0 00000000000148c0 ffff88018c40dfd8
> Call Trace:
> [<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
> [<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
> [<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
> [<ffffffff81436e3d>] ? down_write+0x40/0x44
> [<ffffffffa033be9b>] xfs_ilock+0x4a/0x9a [xfs]
> [<ffffffffa033c413>] xfs_iget+0x34c/0x5cd [xfs]
> [<ffffffffa0351fa9>] xfs_trans_iget+0x1b/0x56 [xfs]
> [<ffffffffa0313035>] xfs_rtfree_extent+0x37/0xdc [xfs]
> [<ffffffffa033fce2>] ? xfs_iext_remove+0xc0/0xd2 [xfs]
> [<ffffffffa0324ffe>] ? xfs_bmap_del_extent+0x2e8/0x93d [xfs]
> [<ffffffffa0324e87>] xfs_bmap_del_extent+0x171/0x93d [xfs]
> [<ffffffffa03506eb>] ? xfs_trans_commit_iclog+0x2ba/0x2d3 [xfs]
> [<ffffffffa0325dc3>] xfs_bunmapi+0x770/0xa28 [xfs]
> [<ffffffffa033e44e>] xfs_itruncate_finish+0x185/0x2b8 [xfs]
> [<ffffffffa0354437>] xfs_inactive+0x1c8/0x3d5 [xfs]
> [<ffffffffa035f38c>] xfs_fs_evict_inode+0xd5/0xdd [xfs]
> [<ffffffff8111282b>] evict+0x22/0x92
> [<ffffffff81112c60>] iput+0x1bc/0x225
> [<ffffffff8110b079>] do_unlinkat+0x103/0x156
> [<ffffffff811086c5>] ? path_put+0x1d/0x22
> [<ffffffff81093e2c>] ? audit_syscall_entry+0x119/0x145
> [<ffffffff8110b205>] sys_unlinkat+0x24/0x26
> [<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b

So it is blocked trying to lock the allocation bitmap inode.
Hmmm, I suspect that XFS_ITRUNC_MAX_EXTENTS is the start of
the problem here.

i.e. what I think might be the problem is that xfs_bunmapi() is
trying to free two extents in the one transaction, and what we see
above is the second extent being freed via xfs_rtfree_extent().
The bitmap inode won't be unlocked until the transaction commits,
so the second call to xfs_trans_iget() in the same transaction will
hang like this.

Hmmm. Looks like we broke recursive inode locking in
xfs_trans_iget() in commit aa72a5cf00001d0b952c7c755be404b9118ceb2e
("xfs: simplify xfs_trans_iget"). The changelog says:

"....
    A quick audit of the callers of xfs_trans_iget shows that no
    caller really relies on this behaviour fortunately - xfs_ialloc
    allows this inode from disk so it must not be there before, and
    all the RT allocator routines only every add each RT bitmap
    inode once.
...."

What we didn't take into account is multiple RT allocator calls in
the same transaction context. Let me think about the best way to fix
this.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Possible deadlock when deleting from realtime section

[Denny Priebe]

On Tue, Oct 26, 2010 at 10:58 PM, Dave Chinner <david@fromorbit.com> wrote:

> Hmmm. Looks like we broke recursive inode locking in
> xfs_trans_iget() in commit aa72a5cf00001d0b952c7c755be404b9118ceb2e
> ("xfs: simplify xfs_trans_iget").

> What we didn't take into account is multiple RT allocator calls in
> the same transaction context. Let me think about the best way to fix
> this.

Is there any news on this? Maybe some patch I could try out?

Regards,
Denny

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs