Re: [fuse-devel] [PATCH] fuse: avoid deadlock when write fuse inode

Re: [fuse-devel] [PATCH] fuse: avoid deadlock when write fuse inode

[Miklos Szeredi]

On Tue, Feb 2, 2021 at 5:41 AM Huang Jianan via fuse-devel
<fuse-devel@lists.sourceforge.net> wrote:
>
> We found the following deadlock situations in low memory scenarios:
> Thread A                         Thread B
> - __writeback_single_inode
>  - fuse_write_inode
>   - fuse_simple_request
>    - __fuse_request_send
>     - request_wait_answer
>                                  - fuse_dev_splice_read
>                                   - fuse_copy_fill
>                                    - __alloc_pages_direct_reclaim
>                                     - do_shrink_slab
>                                      - super_cache_scan
>                                       - shrink_dentry_list
>                                        - dentry_unlink_inode
>                                         - iput_final
>                                          - inode_wait_for_writeback

On what kernel are you seeing this?

I don't see how it can happen on upstream kernels, since there's a
"write_inode_now(inode, 1)" call in fuse_release() and nothing can
dirty the inode after the file has been released.

Thanks,
Miklos

Re: [fuse-devel] [PATCH] fuse: avoid deadlock when write fuse inode

[Rokudo Yan]

Hi, Miklos

The similar issue occurs in our Android device(4G RAM + 3G zram + 8 arm cores + kernel-4.14) too.
Under the monkey test, kswapd and fuse daemon thread deadlocked when free pages is extreme low
(less than 1/2 of the min watermark), the backtrace of the 2 threads is as follows. kswapd
try to evict inode to free some memory(blocked at inode_wait_for_writeback), and fuse daemon thread
handle the fuse inode write request, which is throttled when do direct reclaim in page allocation
slow path(blocked at throttle_direct_reclaim). As the __GFP_FS is set, the thread is throttled until
kswapd free enough pages until watermark ok(check allow_direct_reclaim), which cause the deadlock.
Although the kernel version is 4.14, the same issue exists in the upstream kernel too.

kswapd0         D 26485194.538158 157 1287917 23577482 0x1a20840 0x0 157 438599862461462
<ffffff8beec866b4> __switch_to+0x134/0x150
<ffffff8befb838cc> __schedule+0xd5c/0x1100
<ffffff8befb83ce0> schedule+0x70/0x90
<ffffff8befb849b4> bit_wait+0x14/0x54
<ffffff8befb84350> __wait_on_bit+0x74/0xe0
<ffffff8beeeae0b4> inode_wait_for_writeback+0xa0/0xe4
<ffffff8beee9b95c> evict+0xa4/0x284
<ffffff8beee99b58> iput+0x25c/0x2ac
<ffffff8beee9602c> dentry_unlink_inode+0xd8/0xe4
<ffffff8beee93274> __dentry_kill+0xe8/0x22c
<ffffff8beee9374c> shrink_dentry_list+0x19c/0x3b0
<ffffff8beee9340c> prune_dcache_sb+0x54/0x80
<ffffff8beee79c50> super_cache_scan+0x114/0x164
<ffffff8beee16504> shrink_slab+0x454/0x528
<ffffff8beee1b81c> shrink_node+0x144/0x318
<ffffff8beee1a100> kswapd+0x830/0x9e0
<ffffff8beecde9f0> kthread+0x17c/0x18c
<ffffff8beec856a4> ret_from_fork+0x10/0x18
<ffffffffffffffff> 0xffffffffffffffff

Thread-19       D 7542.719029 2888 24823 5064 0x1404840 0x1000008 24235 438599754021693
<ffffff8beec866b4> __switch_to+0x134/0x150
<ffffff8befb838cc> __schedule+0xd5c/0x1100
<ffffff8befb83ce0> schedule+0x70/0x90
<ffffff8beee18258> try_to_free_pages+0x264/0x4b0
<ffffff8beee06978> __alloc_pages_nodemask+0x7a4/0x10d0
<ffffff8beefac784> fuse_copy_fill+0x15c/0x210
<ffffff8beefabbcc> fuse_dev_do_read+0x434/0xc24
<ffffff8beefab56c> fuse_dev_splice_read+0x84/0x1d8
<ffffff8beeeb5788> SyS_splice+0x67c/0x8bc
<ffffff8beec83fc0> el0_svc_naked+0x34/0x38
<ffffffffffffffff> 0xffffffffffffffff

code snippet:
static bool throttle_direct_reclaim(...)
{
...
	/*
	 * If the caller cannot enter the filesystem, it's possible that it
	 * is due to the caller holding an FS lock or performing a journal
	 * transaction in the case of a filesystem like ext[3|4]. In this case,
	 * it is not safe to block on pfmemalloc_wait as kswapd could be
	 * blocked waiting on the same lock. Instead, throttle for up to a
	 * second before continuing.
	 */
	if (!(gfp_mask & __GFP_FS)) {
		wait_event_interruptible_timeout(pgdat->pfmemalloc_wait,
			allow_direct_reclaim(pgdat), HZ);

		goto check_pending;
	}

	/* Throttle until kswapd wakes the process */
	wait_event_killable(zone->zone_pgdat->pfmemalloc_wait,
		allow_direct_reclaim(pgdat));
...
}

Thanks,
yanwu

On Wed, 24 Mar 2021 16:28:35 +0100 Miklos Szeredi via <miklos@szeredi.hu> wrote:
> On what kernel are you seeing this?

> I don't see how it can happen on upstream kernels, since there's a
>"write_inode_now(inode, 1)" call in fuse_release() and nothing can
> dirty the inode after the file has been released.

> Thanks,
> Miklos

>On Tue, Feb 2, 2021 at 5:41 AM Huang Jianan via fuse-devel
><fuse-devel@lists.sourceforge.net> wrote:
>>
>> We found the following deadlock situations in low memory scenarios:
>> Thread A                         Thread B
>> - __writeback_single_inode
>>  - fuse_write_inode
>>   - fuse_simple_request
>>    - __fuse_request_send
>>     - request_wait_answer
>>                                  - fuse_dev_splice_read
>>                                   - fuse_copy_fill
>>                                    - __alloc_pages_direct_reclaim
>>                                     - do_shrink_slab
>>                                      - super_cache_scan
>>                                       - shrink_dentry_list
>>                                        - dentry_unlink_inode
>>                                         - iput_final
>>                                          - inode_wait_for_writeback

Re: [fuse-devel] [PATCH] fuse: avoid deadlock when write fuse inode

[Miklos Szeredi]

On Thu, 10 Mar 2022 at 12:11, Rokudo Yan <wu-yan@tcl.com> wrote:
>
> Hi, Miklos
>
> The similar issue occurs in our Android device(4G RAM + 3G zram + 8 arm cores + kernel-4.14) too.
> Under the monkey test, kswapd and fuse daemon thread deadlocked when free pages is extreme low
> (less than 1/2 of the min watermark), the backtrace of the 2 threads is as follows. kswapd
> try to evict inode to free some memory(blocked at inode_wait_for_writeback), and fuse daemon thread
> handle the fuse inode write request, which is throttled when do direct reclaim in page allocation
> slow path(blocked at throttle_direct_reclaim). As the __GFP_FS is set, the thread is throttled until
> kswapd free enough pages until watermark ok(check allow_direct_reclaim), which cause the deadlock.
> Although the kernel version is 4.14, the same issue exists in the upstream kernel too.
>
> kswapd0         D 26485194.538158 157 1287917 23577482 0x1a20840 0x0 157 438599862461462
> <ffffff8beec866b4> __switch_to+0x134/0x150
> <ffffff8befb838cc> __schedule+0xd5c/0x1100
> <ffffff8befb83ce0> schedule+0x70/0x90
> <ffffff8befb849b4> bit_wait+0x14/0x54
> <ffffff8befb84350> __wait_on_bit+0x74/0xe0
> <ffffff8beeeae0b4> inode_wait_for_writeback+0xa0/0xe4

This is the one I don't understand.  Fuse inodes must never be dirty
on eviction for the reason stated in my previous reply:

> > I don't see how it can happen on upstream kernels, since there's a
> >"write_inode_now(inode, 1)" call in fuse_release() and nothing can
> > dirty the inode after the file has been released.

If you could trace the source of this dirtyness I think that would
explain this deadlock.

Thanks,
Miklos