[GIT PULL] Btrfs deadlock fix

[GIT PULL] Btrfs deadlock fix

[Chris Mason]

Hi Linus,

My for-linus branch:

git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus

Has a fix for a long standing deadlock that we've been trying to nail
down for a while.  It ended up being a bad interaction with the fair
reader/writer locks and the order btrfs reacquires locks in the btree.

Chris Mason (1) commits (+25/-15):
    btrfs: fix lockups from btrfs_clear_path_blocking

Total: (1) commits (+25/-15)

 fs/btrfs/ctree.c   | 14 ++------------
 fs/btrfs/locking.c | 24 +++++++++++++++++++++---
 fs/btrfs/locking.h |  2 ++
 3 files changed, 25 insertions(+), 15 deletions(-)

Re: [GIT PULL] Btrfs deadlock fix

[Benjamin Herrenschmidt]

On Sun, 2014-11-23 at 10:14 -0500, Chris Mason wrote:
> Hi Linus,
> 
> My for-linus branch:

Not sure if this is related, but with -rc4, on this ppc64el box, when
grub updates (it starts trying to mount everything under the sun with
every filesystem), the box hangs with:

[  177.344522] INFO: rcu_sched detected stalls on CPUs/tasks: { 8}
(detected by 9, t=25883 jiffies, g=3642, c=3641, q=5206)

With xmon I can catch it with this backtrace:

[link register   ] d00000000ecec3f8 test_check_exists+0x1f8/0x280 [btrfs]
[c000000f0cc63930] d00000000ecec2b0 test_check_exists+0xb0/0x280 [btrfs] (unreliable)
[c000000f0cc63990] d00000000ed2a4b4 btrfs_test_free_space_cache+0x774/0x1150 [btrfs]
[c000000f0cc63a20] d00000000ed30f04 init_btrfs_fs+0x11c/0x210 [btrfs]
[c000000f0cc63a90] c00000000000b4dc do_one_initcall+0x12c/0x280
[c000000f0cc63b60] c0000000001628b8 load_module+0x2088/0x2970
[c000000f0cc63d50] c0000000001633d0 SyS_finit_module+0xc0/0x120
[c000000f0cc63e30] c00000000000927c syscall_exit+0x0/0x7c
--- Exception: c01 (System Call) at 00003fff88697184

Note that afaik there isn't any btrfs partition on any disk on that
machine... well not that I know of :)

Cheers,
Ben.

> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus
> 
> Has a fix for a long standing deadlock that we've been trying to nail
> down for a while.  It ended up being a bad interaction with the fair
> reader/writer locks and the order btrfs reacquires locks in the btree.
> 
> Chris Mason (1) commits (+25/-15):
>     btrfs: fix lockups from btrfs_clear_path_blocking
> 
> Total: (1) commits (+25/-15)
> 
>  fs/btrfs/ctree.c   | 14 ++------------
>  fs/btrfs/locking.c | 24 +++++++++++++++++++++---
>  fs/btrfs/locking.h |  2 ++
>  3 files changed, 25 insertions(+), 15 deletions(-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] Btrfs deadlock fix

[Chris Mason]

On Sun, Nov 23, 2014 at 7:23 PM, Benjamin Herrenschmidt 
<benh@kernel.crashing.org> wrote:
> On Sun, 2014-11-23 at 10:14 -0500, Chris Mason wrote:
>>  Hi Linus,
>> 
>>  My for-linus branch:
> 
> Not sure if this is related, but with -rc4, on this ppc64el box, when
> grub updates (it starts trying to mount everything under the sun with
> every filesystem), the box hangs with:
> 
> [  177.344522] INFO: rcu_sched detected stalls on CPUs/tasks: { 8}
> (detected by 9, t=25883 jiffies, g=3642, c=3641, q=5206)
> 
> With xmon I can catch it with this backtrace:
> 
> [link register   ] d00000000ecec3f8 test_check_exists+0x1f8/0x280 
> [btrfs]
> [c000000f0cc63930] d00000000ecec2b0 test_check_exists+0xb0/0x280 
> [btrfs] (unreliable)
> [c000000f0cc63990] d00000000ed2a4b4 
> btrfs_test_free_space_cache+0x774/0x1150 [btrfs]
> [c000000f0cc63a20] d00000000ed30f04 init_btrfs_fs+0x11c/0x210 [btrfs]
> [c000000f0cc63a90] c00000000000b4dc do_one_initcall+0x12c/0x280
> [c000000f0cc63b60] c0000000001628b8 load_module+0x2088/0x2970
> [c000000f0cc63d50] c0000000001633d0 SyS_finit_module+0xc0/0x120
> [c000000f0cc63e30] c00000000000927c syscall_exit+0x0/0x7c
> --- Exception: c01 (System Call) at 00003fff88697184
> 
> Note that afaik there isn't any btrfs partition on any disk on that
> machine... well not that I know of :)

This should be from CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y.  I do test with 
this on here, but I'll look it over for ppc related fun.

The deadlock fix from this pull shouldn't be related.  Just to make 
sure I read your email right, this stall isn't new with rc6 is it?

-chris

Re: [GIT PULL] Btrfs deadlock fix

[Benjamin Herrenschmidt]

On Sun, 2014-11-23 at 19:27 -0500, Chris Mason wrote:
> This should be from CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y.  I do test
> with this on here, but I'll look it over for ppc related fun.
> 
> The deadlock fix from this pull shouldn't be related.  Just to make 
> sure I read your email right, this stall isn't new with rc6 is it?

No, I observed it with -rc4 which is the first 3.18 I tried on this
specific machine. I can try to bisect but it will take a while, it
didn't happen afaik with 3.17

Cheers,
Ben.

Re: [GIT PULL] Btrfs deadlock fix

[David Sterba]

Hi Chris,

I thought the fix for the scrub/replace deadlock would be included in
this pull, I can reproduce it on each run of xfstests with 3.18-rc.

btrfs: fix dead lock while running replace and defrag concurrently
https://patchwork.kernel.org/patch/5264531/

I've retested it again including this pull and still deadlocks reliably
at btrfs/070.

Re: [GIT PULL] Btrfs deadlock fix

[Chris Mason]

On Mon, Nov 24, 2014 at 12:39 PM, David Sterba <dsterba@suse.cz> wrote:
> Hi Chris,
> 
> I thought the fix for the scrub/replace deadlock would be included in
> this pull, I can reproduce it on each run of xfstests with 3.18-rc.
> 
> btrfs: fix dead lock while running replace and defrag concurrently

> https://patchwork.kernel.org/patch/5264531/
> 
> I've retested it again including this pull and still deadlocks 
> reliably
> at btrfs/070.

This wasn't a new problem, so I had it queued for the merge window.

-chris

Re: [GIT PULL] Btrfs deadlock fix

[David Sterba]

On Mon, Nov 24, 2014 at 01:01:10PM -0500, Chris Mason wrote:
> > I've retested it again including this pull and still deadlocks 
> > reliably
> > at btrfs/070.
> 
> This wasn't a new problem, so I had it queued for the merge window.

Well, I don't remember seeing this problem with anything 3.17 based but
I can check. I know this does not mean that it was introduced in 3.18,
only that other changes exposed the problem, but this still IMHO
qualifies as a bug to be fixed within 3.18. Besides, it makes testing
3.18-rc slightly more annoying (either skip the test or manually pick
the patch after each pull).