[PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Daeho Jeong]

From: Daeho Jeong <daehojeong@google.com>

We detected the below circular locking dependency between sb_internal
and fs_reclaim. So, removed it by calling dquot_initialize() before
sb_start_intwrite().

 ======================================================
 WARNING: possible circular locking dependency detected
 ------------------------------------------------------
 kswapd0/133 is trying to acquire lock:
ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8

but task is already holding lock:
ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
__fs_reclaim_acquire+0x4/0x50

which lock already depends on the new lock.
...
other info that might help us debug this:

 Chain exists of:

sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim

  Possible unsafe locking scenario:

        CPU0                    CPU1
        ----                    ----
   lock(fs_reclaim);
                                lock(&s->s_dquot.dqio_sem);
                                lock(fs_reclaim);
   lock(sb_internal#2);

Signed-off-by: Daeho Jeong <daehojeong@google.com>
---
 fs/f2fs/super.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 86eeb019cc52..a133932333c5 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
 			/* should remain fi->extent_tree for writepage */
 			f2fs_destroy_extent_node(inode);
 
+			dquot_initialize(inode);
+
 			sb_start_intwrite(inode->i_sb);
 			f2fs_i_size_write(inode, 0);
 
-- 
2.33.0.1079.g6e70778dc9-goog

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Chao Yu]

On 2021/10/15 3:05, Daeho Jeong wrote:
> From: Daeho Jeong <daehojeong@google.com>
> 
> We detected the below circular locking dependency between sb_internal
> and fs_reclaim. So, removed it by calling dquot_initialize() before
> sb_start_intwrite().
> 
>   ======================================================
>   WARNING: possible circular locking dependency detected
>   ------------------------------------------------------
>   kswapd0/133 is trying to acquire lock:
> ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
> 
> but task is already holding lock:
> ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
> __fs_reclaim_acquire+0x4/0x50
> 
> which lock already depends on the new lock.
> ...
> other info that might help us debug this:
> 
>   Chain exists of:
> 
> sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
> 
>    Possible unsafe locking scenario:
> 
>          CPU0                    CPU1
>          ----                    ----
>     lock(fs_reclaim);
>                                  lock(&s->s_dquot.dqio_sem);
>                                  lock(fs_reclaim);
>     lock(sb_internal#2);

Sorry, I still didn't get the root cause of this deadlock issue, could
you please explain more about this?

And why calling dquot_initialize() in drop_inode() could break the
circular locking dependency?

Thanks,

> 
> Signed-off-by: Daeho Jeong <daehojeong@google.com>
> ---
>   fs/f2fs/super.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> index 86eeb019cc52..a133932333c5 100644
> --- a/fs/f2fs/super.c
> +++ b/fs/f2fs/super.c
> @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
>   			/* should remain fi->extent_tree for writepage */
>   			f2fs_destroy_extent_node(inode);
>   
> +			dquot_initialize(inode);
> +
>   			sb_start_intwrite(inode->i_sb);
>   			f2fs_i_size_write(inode, 0);
>   
> 

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Daeho Jeong]

There is a deadlock between sb_internal lock (sb_start_intwrite()) and
dquot related lock.
It's because we call f2fs_truncate(), which eventually calls
dquot_initialize(), while holding sb_internal lock.
So, I called dquot_initialize() in advance to make the 2nd calling of
it in f2fs_truncate() ineffective.
This is similar with the thing in f2fs_evict_inode() in inode.c

Thanks,

On Thu, Oct 21, 2021 at 5:11 AM Chao Yu <chao@kernel.org> wrote:
>
> On 2021/10/15 3:05, Daeho Jeong wrote:
> > From: Daeho Jeong <daehojeong@google.com>
> >
> > We detected the below circular locking dependency between sb_internal
> > and fs_reclaim. So, removed it by calling dquot_initialize() before
> > sb_start_intwrite().
> >
> >   ======================================================
> >   WARNING: possible circular locking dependency detected
> >   ------------------------------------------------------
> >   kswapd0/133 is trying to acquire lock:
> > ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
> >
> > but task is already holding lock:
> > ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
> > __fs_reclaim_acquire+0x4/0x50
> >
> > which lock already depends on the new lock.
> > ...
> > other info that might help us debug this:
> >
> >   Chain exists of:
> >
> > sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
> >
> >    Possible unsafe locking scenario:
> >
> >          CPU0                    CPU1
> >          ----                    ----
> >     lock(fs_reclaim);
> >                                  lock(&s->s_dquot.dqio_sem);
> >                                  lock(fs_reclaim);
> >     lock(sb_internal#2);
>
> Sorry, I still didn't get the root cause of this deadlock issue, could
> you please explain more about this?
>
> And why calling dquot_initialize() in drop_inode() could break the
> circular locking dependency?
>
> Thanks,
>
> >
> > Signed-off-by: Daeho Jeong <daehojeong@google.com>
> > ---
> >   fs/f2fs/super.c | 2 ++
> >   1 file changed, 2 insertions(+)
> >
> > diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> > index 86eeb019cc52..a133932333c5 100644
> > --- a/fs/f2fs/super.c
> > +++ b/fs/f2fs/super.c
> > @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
> >                       /* should remain fi->extent_tree for writepage */
> >                       f2fs_destroy_extent_node(inode);
> >
> > +                     dquot_initialize(inode);
> > +
> >                       sb_start_intwrite(inode->i_sb);
> >                       f2fs_i_size_write(inode, 0);
> >
> >

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Chao Yu]

On 2021/10/22 0:44, Daeho Jeong wrote:
> There is a deadlock between sb_internal lock (sb_start_intwrite()) and
> dquot related lock.
> It's because we call f2fs_truncate(), which eventually calls
> dquot_initialize(), while holding sb_internal lock.
> So, I called dquot_initialize() in advance to make the 2nd calling of
> it in f2fs_truncate() ineffective.
> This is similar with the thing in f2fs_evict_inode() in inode.c

Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
into deadlock?

Thanks,

> 
> Thanks,
> 
> On Thu, Oct 21, 2021 at 5:11 AM Chao Yu <chao@kernel.org> wrote:
>>
>> On 2021/10/15 3:05, Daeho Jeong wrote:
>>> From: Daeho Jeong <daehojeong@google.com>
>>>
>>> We detected the below circular locking dependency between sb_internal
>>> and fs_reclaim. So, removed it by calling dquot_initialize() before
>>> sb_start_intwrite().
>>>
>>>    ======================================================
>>>    WARNING: possible circular locking dependency detected
>>>    ------------------------------------------------------
>>>    kswapd0/133 is trying to acquire lock:
>>> ffffff80d5fb9680 (sb_internal#2){.+.+}-{0:0}, at: evict+0xd4/0x2f8
>>>
>>> but task is already holding lock:
>>> ffffffda597c93a8 (fs_reclaim){+.+.}-{0:0}, at:
>>> __fs_reclaim_acquire+0x4/0x50
>>>
>>> which lock already depends on the new lock.
>>> ...
>>> other info that might help us debug this:
>>>
>>>    Chain exists of:
>>>
>>> sb_internal#2 --> &s->s_dquot.dqio_sem --> fs_reclaim
>>>
>>>     Possible unsafe locking scenario:
>>>
>>>           CPU0                    CPU1
>>>           ----                    ----
>>>      lock(fs_reclaim);
>>>                                   lock(&s->s_dquot.dqio_sem);
>>>                                   lock(fs_reclaim);
>>>      lock(sb_internal#2);
>>
>> Sorry, I still didn't get the root cause of this deadlock issue, could
>> you please explain more about this?
>>
>> And why calling dquot_initialize() in drop_inode() could break the
>> circular locking dependency?
>>
>> Thanks,
>>
>>>
>>> Signed-off-by: Daeho Jeong <daehojeong@google.com>
>>> ---
>>>    fs/f2fs/super.c | 2 ++
>>>    1 file changed, 2 insertions(+)
>>>
>>> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
>>> index 86eeb019cc52..a133932333c5 100644
>>> --- a/fs/f2fs/super.c
>>> +++ b/fs/f2fs/super.c
>>> @@ -1370,6 +1370,8 @@ static int f2fs_drop_inode(struct inode *inode)
>>>                        /* should remain fi->extent_tree for writepage */
>>>                        f2fs_destroy_extent_node(inode);
>>>
>>> +                     dquot_initialize(inode);
>>> +
>>>                        sb_start_intwrite(inode->i_sb);
>>>                        f2fs_i_size_write(inode, 0);
>>>
>>>

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Daeho Jeong]

On Fri, Oct 22, 2021 at 8:32 AM Chao Yu <chao@kernel.org> wrote:
>
> On 2021/10/22 0:44, Daeho Jeong wrote:
> > There is a deadlock between sb_internal lock (sb_start_intwrite()) and
> > dquot related lock.
> > It's because we call f2fs_truncate(), which eventually calls
> > dquot_initialize(), while holding sb_internal lock.
> > So, I called dquot_initialize() in advance to make the 2nd calling of
> > it in f2fs_truncate() ineffective.
> > This is similar with the thing in f2fs_evict_inode() in inode.c
>
> Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
> into deadlock?
>

Do you think the same issue is in f2fs_evict_inode() in inode.c?
In fact, I picked up the idea from here.

        err = dquot_initialize(inode);
        if (err) {
                err = 0;
                set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
        }

        f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
        f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
        f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);

        sb_start_intwrite(inode->i_sb);
        set_inode_flag(inode, FI_NO_ALLOC);
        i_size_write(inode, 0);
retry:
        if (F2FS_HAS_BLOCKS(inode))
                err = f2fs_truncate(inode);

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Chao Yu]

On 2021/10/26 0:22, Daeho Jeong wrote:
> On Fri, Oct 22, 2021 at 8:32 AM Chao Yu <chao@kernel.org> wrote:
>>
>> On 2021/10/22 0:44, Daeho Jeong wrote:
>>> There is a deadlock between sb_internal lock (sb_start_intwrite()) and
>>> dquot related lock.
>>> It's because we call f2fs_truncate(), which eventually calls
>>> dquot_initialize(), while holding sb_internal lock.
>>> So, I called dquot_initialize() in advance to make the 2nd calling of
>>> it in f2fs_truncate() ineffective.
>>> This is similar with the thing in f2fs_evict_inode() in inode.c
>>
>> Well, if dquot_initialize() fails in f2fs_drop_inode(), will we still run
>> into deadlock?
>>
> 
> Do you think the same issue is in f2fs_evict_inode() in inode.c?

Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
callpath:

- evict_inode
  - dquot_initialize failed
  - sb_start_intwrite
  - f2fs_truncate
   - dquot_initialize lock dqio_sem

How about this?

diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
old mode 100644
new mode 100755
index b24b9bc..0e49593
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -728,6 +728,7 @@ enum {
      FI_ENABLE_COMPRESS,    /* enable compression in "user" compression mode */
      FI_COMPRESS_RELEASED,    /* compressed blocks were released */
      FI_ALIGNED_WRITE,    /* enable aligned write */
+    FI_QUOTA_INIT_FAIL,    /* inidicate failed to initialize quota in drop_inode()/evict_inode() */
      FI_MAX,            /* max flag, never be used */
  };

diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
old mode 100644
new mode 100755
index 13deae0..2fb53f54
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -789,9 +789,11 @@ int f2fs_truncate(struct inode *inode)
          return -EIO;
      }

-    err = dquot_initialize(inode);
-    if (err)
-        return err;
+    if (!is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL)) {
+        err = dquot_initialize(inode);
+        if (err)
+            return err;
+    }

      /* we should check inline_data size */
      if (!f2fs_may_inline_data(inode)) {
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
old mode 100644
new mode 100755
index 1213f15..16cf50c
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -758,6 +758,7 @@ void f2fs_evict_inode(struct inode *inode)
      if (err) {
          err = 0;
          set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+        set_inode_flag(inode, FI_QUOTA_INIT_FAIL);
      }

      f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
@@ -770,6 +771,8 @@ void f2fs_evict_inode(struct inode *inode)
  retry:
      if (F2FS_HAS_BLOCKS(inode))
          err = f2fs_truncate(inode);
+    if (is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL))
+        clear_inode_flag(inode, FI_QUOTA_INIT_FAIL);

      if (time_to_inject(sbi, FAULT_EVICT_INODE)) {
          f2fs_show_injection_info(sbi, FAULT_EVICT_INODE);

Thanks,


> In fact, I picked up the idea from here.
> 
>          err = dquot_initialize(inode);
>          if (err) {
>                  err = 0;
>                  set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
>          }
> 
>          f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
>          f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
>          f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
> 
>          sb_start_intwrite(inode->i_sb);
>          set_inode_flag(inode, FI_NO_ALLOC);
>          i_size_write(inode, 0);
> retry:
>          if (F2FS_HAS_BLOCKS(inode))
>                  err = f2fs_truncate(inode);
> 

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Daeho Jeong]

> Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
> callpath:
>
> - evict_inode
>   - dquot_initialize failed
>   - sb_start_intwrite
>   - f2fs_truncate
>    - dquot_initialize lock dqio_sem
>
> How about this?
>

Got it~
Then we need this in both f2fs_evict_inode() and f2fs_drop_inode().

Thanks,

> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> old mode 100644
> new mode 100755
> index b24b9bc..0e49593
> --- a/fs/f2fs/f2fs.h
> +++ b/fs/f2fs/f2fs.h
> @@ -728,6 +728,7 @@ enum {
>       FI_ENABLE_COMPRESS,    /* enable compression in "user" compression mode */
>       FI_COMPRESS_RELEASED,    /* compressed blocks were released */
>       FI_ALIGNED_WRITE,    /* enable aligned write */
> +    FI_QUOTA_INIT_FAIL,    /* inidicate failed to initialize quota in drop_inode()/evict_inode() */
>       FI_MAX,            /* max flag, never be used */
>   };
>
> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> old mode 100644
> new mode 100755
> index 13deae0..2fb53f54
> --- a/fs/f2fs/file.c
> +++ b/fs/f2fs/file.c
> @@ -789,9 +789,11 @@ int f2fs_truncate(struct inode *inode)
>           return -EIO;
>       }
>
> -    err = dquot_initialize(inode);
> -    if (err)
> -        return err;
> +    if (!is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL)) {
> +        err = dquot_initialize(inode);
> +        if (err)
> +            return err;
> +    }
>
>       /* we should check inline_data size */
>       if (!f2fs_may_inline_data(inode)) {
> diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
> old mode 100644
> new mode 100755
> index 1213f15..16cf50c
> --- a/fs/f2fs/inode.c
> +++ b/fs/f2fs/inode.c
> @@ -758,6 +758,7 @@ void f2fs_evict_inode(struct inode *inode)
>       if (err) {
>           err = 0;
>           set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
> +        set_inode_flag(inode, FI_QUOTA_INIT_FAIL);
>       }
>
>       f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
> @@ -770,6 +771,8 @@ void f2fs_evict_inode(struct inode *inode)
>   retry:
>       if (F2FS_HAS_BLOCKS(inode))
>           err = f2fs_truncate(inode);
> +    if (is_inode_flag_set(inode, FI_QUOTA_INIT_FAIL))
> +        clear_inode_flag(inode, FI_QUOTA_INIT_FAIL);
>
>       if (time_to_inject(sbi, FAULT_EVICT_INODE)) {
>           f2fs_show_injection_info(sbi, FAULT_EVICT_INODE);
>
> Thanks,
>
>
> > In fact, I picked up the idea from here.
> >
> >          err = dquot_initialize(inode);
> >          if (err) {
> >                  err = 0;
> >                  set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
> >          }
> >
> >          f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
> >          f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
> >          f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
> >
> >          sb_start_intwrite(inode->i_sb);
> >          set_inode_flag(inode, FI_NO_ALLOC);
> >          i_size_write(inode, 0);
> > retry:
> >          if (F2FS_HAS_BLOCKS(inode))
> >                  err = f2fs_truncate(inode);
> >

Re: [f2fs-dev] [PATCH] f2fs: remove circular locking between sb_internal and fs_reclaim

[Daeho Jeong]

On Tue, Oct 26, 2021 at 10:56 AM Daeho Jeong <daeho43@gmail.com> wrote:
>
> > Yes, I doubt the problem may also happen in f2fs_evict_inode() with below
> > callpath:
> >
> > - evict_inode
> >   - dquot_initialize failed
> >   - sb_start_intwrite
> >   - f2fs_truncate
> >    - dquot_initialize lock dqio_sem
> >
> > How about this?
> >
>
> Got it~
> Then we need this in both f2fs_evict_inode() and f2fs_drop_inode().
>

It turns out this deadlock issue was related to the Android kernel only. :(
That was related to one of Android tracepoints, which triggered
internal memory reclaim inside of it.
We made a workaround for that in Android kernel.

Thanks,