All of lore.kernel.org
 help / color / mirror / Atom feed
* 4.6-rc7 xfs circular locking dependency
@ 2016-05-11 17:52 Bart Van Assche
  2016-05-11 23:07 ` Dave Chinner
  0 siblings, 1 reply; 2+ messages in thread
From: Bart Van Assche @ 2016-05-11 17:52 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Hi Dave,

While retesting the SRP initiator with xfstests on top of an XFS filesystem I
hit the below call trace once. I do not expect that this is related to the SRP
initiator changes I made. Please let me know if you need more information.

Thanks,

Bart.

======================================================
[ INFO: possible circular locking dependency detected ]
4.6.0-rc7-dbg+ #1 Not tainted
-------------------------------------------------------
fsstress/17356 is trying to acquire lock:
 (sb_internal#2){++++.+}, at: [<ffffffff81193172>] __sb_start_write+0xb2/0xf0

but task is already holding lock:
 (&s->s_sync_lock){+.+...}, at: [<ffffffff811bba2d>] sync_inodes_sb+0xbd/0x1d0

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

nc_lock){+.+...}:
       [<ffffffff810a4e70>] lock_acquire+0x60/0x80
       [<ffffffff81582d2f>] mutex_lock_nested+0x5f/0x360
       [<ffffffff811bba2d>] sync_inodes_sb+0xbd/0x1d0
       [<ffffffffa0616963>] xfs_flush_inodes+0x23/0x30 [xfs]
       [<ffffffffa060ea9f>] xfs_create+0x46f/0x5f0 [xfs]
       [<ffffffffa0609f39>] xfs_generic_create+0x1b9/0x290 [xfs]
       [<ffffffffa060a03f>] xfs_vn_mknod+0xf/0x20 [xfs]
       [<ffffffffa060a07e>] xfs_vn_create+0xe/0x10 [xfs]
       [<ffffffff8119af96>] vfs_create+0x76/0xd0
       [<ffffffff8119f13e>] path_openat+0xc1e/0x10d0
       [<ffffffff811a04d9>] do_filp_open+0x79/0xd0
       [<ffffffff8118f636>] do_sys_open+0x116/0x1f0
       [<ffffffff8118f769>] SyS_creat+0x19/0x20
       [<ffffffff81585fe5>] entry_SYSCALL_64_fastpath+0x18/0xa8

nal#2){++++.+}:
       [<ffffffff810a461f>] __lock_acquire+0x1b0f/0x1b20
       [<ffffffff810a4e70>] lock_acquire+0x60/0x80
       [<ffffffff8109edd5>] percpu_down_read+0x45/0x90
       [<ffffffff81193172>] __sb_start_write+0xb2/0xf0
       [<ffffffffa061838f>] xfs_trans_alloc+0x1f/0x40 [xfs]
       [<ffffffffa060f4d0>] xfs_inactive_truncate+0x20/0x130 [xfs]
       [<ffffffffa060fc9e>] xfs_inactive+0x1ae/0x1e0 [xfs]
       [<ffffffffa0614e88>] xfs_fs_evict_inode+0xb8/0xc0 [xfs]
       [<ffffffff811acc83>] evict+0xb3/0x180
       [<ffffffff811acefc>] iput+0x14c/0x1e0
       [<ffffffff811bbab5>] sync_inodes_sb+0x145/0x1d0
       [<ffffffff811c23e0>] sync_inodes_one_sb+0x10/0x20
       [<ffffffff81193bda>] iterate_supers+0xaa/0x100
       [<ffffffff811c26d0>] sys_sync+0x30/0x90
       [<ffffffff81585fe5>] entry_SYSCALL_64_fastpath+0x18/0xa8

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&s->s_sync_lock);
                               lock(sb_internal#2);
                               lock(&s->s_sync_lock);
  lock(sb_internal#2);

 *** DEADLOCK ***

2 locks held by fsstress/17356:
 #0:  (&type->s_umount_key#34){++++++}, at: [<ffffffff81193bc4>] iterate_supers+0x94/0x100
 #1:  (&s->s_sync_lock){+.+...}, at: [<ffffffff811bba2d>] sync_inodes_sb+0xbd/0x1d0

stack backtrace:
CPU: 2 PID: 17356 Comm: fsstress Not tainted 4.6.0-rc7-dbg+ #1
Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.0.2 11/17/2014
 0000000000000000 ffff880442b3bbe0 ffffffff812ac6b5 ffffffff8238e0c0
 ffffffff8238e0c0 ffff880442b3bc20 ffffffff810a1233 ffff880442b3bc70
 ffff8804153635c0 ffff880415363598 ffff880415362d80 ffff880415363570
Call Trace:
 [<ffffffff812ac6b5>] dump_stack+0x67/0x92
 [<ffffffff810a1233>] print_circular_bug+0x1e3/0x250
 [<ffffffff810a461f>] __lock_acquire+0x1b0f/0x1b20
 [<ffffffff8112382f>] ? truncate_inode_pages_range+0x2af/0x790
 [<ffffffff810a4e70>] lock_acquire+0x60/0x80
 [<ffffffff81193172>] ? __sb_start_write+0xb2/0xf0
 [<ffffffff8109edd5>] percpu_down_read+0x45/0x90
 [<ffffffff81193172>] ? __sb_start_write+0xb2/0xf0
 [<ffffffff81193172>] __sb_start_write+0xb2/0xf0
 [<ffffffff810a266f>] ? mark_held_locks+0x6f/0xa0
 [<ffffffffa061838f>] xfs_trans_alloc+0x1f/0x40 [xfs]
 [<ffffffffa060f4d0>] xfs_inactive_truncate+0x20/0x130 [xfs]
 [<ffffffffa060fc9e>] xfs_inactive+0x1ae/0x1e0 [xfs]
 [<ffffffffa0614e88>] xfs_fs_evict_inode+0xb8/0xc0 [xfs]
 [<ffffffff811acc83>] evict+0xb3/0x180
 [<ffffffff811acefc>] iput+0x14c/0x1e0
 [<ffffffff811bbab5>] sync_inodes_sb+0x145/0x1d0
 [<ffffffff811c23d0>] ? SyS_tee+0x400/0x400
 [<ffffffff811c23e0>] sync_inodes_one_sb+0x10/0x20
 [<ffffffff81193bda>] iterate_supers+0xaa/0x100
 [<ffffffff811c26d0>] sys_sync+0x30/0x90
 [<ffffffff81585fe5>] entry_SYSCALL_64_fastpath+0x18/0xa8

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 2+ messages in thread
* Re: 4.6-rc7 xfs circular locking dependency
  2016-05-11 17:52 4.6-rc7 xfs circular locking dependency Bart Van Assche
@ 2016-05-11 23:07 ` Dave Chinner
  0 siblings, 0 replies; 2+ messages in thread
From: Dave Chinner @ 2016-05-11 23:07 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: xfs

On Wed, May 11, 2016 at 10:52:00AM -0700, Bart Van Assche wrote:
> Hi Dave,
> 
> While retesting the SRP initiator with xfstests on top of an XFS filesystem I
> hit the below call trace once. I do not expect that this is related to the SRP
> initiator changes I made. Please let me know if you need more information.

YALFP.(*)

It's a "problem" with the filesystem freeze protection when
interacting with emergency ENOSPC paths in the filesystem.
sb_start_intwrite() can block when a freeze is being progressed, and
hence on the surface this looks like a trivial ABBA deadlock
condition.

However, this *cannot deadlock* because the trigger for
sb_start_intwrite() to block is a change of state to
SB_FREEZE_COMPLETE, and that happens under a
down_write(sb->s_umount) context. i.e.:

int freeze_super(struct super_block *sb)
{
        int ret;

        atomic_inc(&sb->s_active);
        down_write(&sb->s_umount)
	.....

The XFS code that takes s_sync_lock here:

> nc_lock){+.+...}:
>        [<ffffffff810a4e70>] lock_acquire+0x60/0x80
>        [<ffffffff81582d2f>] mutex_lock_nested+0x5f/0x360
>        [<ffffffff811bba2d>] sync_inodes_sb+0xbd/0x1d0
>        [<ffffffffa0616963>] xfs_flush_inodes+0x23/0x30 [xfs]

Does:

        if (down_read_trylock(&sb->s_umount)) {
                sync_inodes_sb(sb);
                up_read(&sb->s_umount);
        }

So any attempt to freeze the filesystem is prevented /before/ XFS
tries to flush data in this ENOSPC path. And if a freeze is
alreadybeing processed, XFS will skip this path, thereby avoiding
the possibility of a ABBA deadlock between freeze and s_sync_lock.

>        [<ffffffffa060ea9f>] xfs_create+0x46f/0x5f0 [xfs]
>        [<ffffffffa0609f39>] xfs_generic_create+0x1b9/0x290 [xfs]
>        [<ffffffffa060a03f>] xfs_vn_mknod+0xf/0x20 [xfs]
>        [<ffffffffa060a07e>] xfs_vn_create+0xe/0x10 [xfs]
>        [<ffffffff8119af96>] vfs_create+0x76/0xd0
>        [<ffffffff8119f13e>] path_openat+0xc1e/0x10d0
>        [<ffffffff811a04d9>] do_filp_open+0x79/0xd0
>        [<ffffffff8118f636>] do_sys_open+0x116/0x1f0
>        [<ffffffff8118f769>] SyS_creat+0x19/0x20
>        [<ffffffff81585fe5>] entry_SYSCALL_64_fastpath+0x18/0xa8
> 
> nal#2){++++.+}:
>        [<ffffffff810a461f>] __lock_acquire+0x1b0f/0x1b20
>        [<ffffffff810a4e70>] lock_acquire+0x60/0x80
>        [<ffffffff8109edd5>] percpu_down_read+0x45/0x90
>        [<ffffffff81193172>] __sb_start_write+0xb2/0xf0
>        [<ffffffffa061838f>] xfs_trans_alloc+0x1f/0x40 [xfs]
>        [<ffffffffa060f4d0>] xfs_inactive_truncate+0x20/0x130 [xfs]
>        [<ffffffffa060fc9e>] xfs_inactive+0x1ae/0x1e0 [xfs]
>        [<ffffffffa0614e88>] xfs_fs_evict_inode+0xb8/0xc0 [xfs]
>        [<ffffffff811acc83>] evict+0xb3/0x180
>        [<ffffffff811acefc>] iput+0x14c/0x1e0
>        [<ffffffff811bbab5>] sync_inodes_sb+0x145/0x1d0
>        [<ffffffff811c23e0>] sync_inodes_one_sb+0x10/0x20
>        [<ffffffff81193bda>] iterate_supers+0xaa/0x100

And iterate_supers() does the same thing:

                down_read(&sb->s_umount);
                if (sb->s_root && (sb->s_flags & MS_BORN))
                        f(sb, arg);
                up_read(&sb->s_umount);

IOWs, freeze_super() cannot start/progress freeze state to the point
that it would cause a deadlock in this code path because we hold the
lock that freeze_super() requires to change the freeze state.

Cheers,

Dave.

(*) YALFP: Yet Another Lockdep False Positive
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 2+ messages in thread
end of thread, other threads:[~2016-05-11 23:07 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-11 17:52 4.6-rc7 xfs circular locking dependency Bart Van Assche
2016-05-11 23:07 ` Dave Chinner

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.