* memalloc_nofs_save() only on async
@ 2021-03-03 13:10 Benjamin Coddington
2021-03-03 13:23 ` Trond Myklebust
0 siblings, 1 reply; 2+ messages in thread
From: Benjamin Coddington @ 2021-03-03 13:10 UTC (permalink / raw)
To: Trond Myklebust; +Cc: Linux NFS Mailing List
Hi Trond,
I'd like to go back to setting sk->sk_allocation = GFP_NOIO (see
a1231fda7e94). That would cover sync tasks as well as async, but I'm not
sure what memalloc_nofs_save/restore gives us and if we should just try
to
apply that to all tasks.
We're getting some folks deadlocked on the xprt_sending queue with
stacks like:
#0 [ffffacab45f17108] __schedule at ffffffffae4d1826
#1 [ffffacab45f171a0] schedule at ffffffffae4d1cb8
#2 [ffffacab45f171b0] rpc_wait_bit_killable at ffffffffc067d44e
[sunrpc]
#3 [ffffacab45f171c8] __wait_on_bit at ffffffffae4d216c
#4 [ffffacab45f17200] out_of_line_wait_on_bit at ffffffffae4d2211
#5 [ffffacab45f17250] __rpc_execute at ffffffffc067f3fc [sunrpc]
#6 [ffffacab45f172a8] rpc_run_task at ffffffffc06732c4 [sunrpc]
#7 [ffffacab45f172e8] nfs4_proc_layoutreturn at ffffffffc08f5d44
[nfsv4]
#8 [ffffacab45f17388] pnfs_send_layoutreturn at ffffffffc091946e
[nfsv4]
#9 [ffffacab45f173d8] _pnfs_return_layout at ffffffffc091ba8b [nfsv4]
#10 [ffffacab45f17450] nfs4_evict_inode at ffffffffc0906a05 [nfsv4]
#11 [ffffacab45f17460] evict at ffffffffadef8592
#12 [ffffacab45f17480] dispose_list at ffffffffadef86a8
#13 [ffffacab45f174a0] prune_icache_sb at ffffffffadef99a2
#14 [ffffacab45f174c8] super_cache_scan at ffffffffadede183
#15 [ffffacab45f17518] do_shrink_slab at ffffffffade3d5d8
#16 [ffffacab45f17588] shrink_slab at ffffffffade3dab5
#17 [ffffacab45f17608] shrink_node at ffffffffade42a8c
#18 [ffffacab45f17678] do_try_to_free_pages at ffffffffade42e43
#19 [ffffacab45f176c8] try_to_free_pages at ffffffffade431c8
#20 [ffffacab45f17768] __alloc_pages_slowpath at ffffffffade81981
#21 [ffffacab45f17868] __alloc_pages_nodemask at ffffffffade82555
#22 [ffffacab45f178c8] skb_page_frag_refill at ffffffffae31bea7
#23 [ffffacab45f178e0] sk_page_frag_refill at ffffffffae31c71d
#24 [ffffacab45f178f8] tcp_sendmsg_locked at ffffffffae3cbe65
#25 [ffffacab45f179a0] tcp_sendmsg at ffffffffae3cc8f7
#26 [ffffacab45f179c0] sock_sendmsg at ffffffffae317cce
#27 [ffffacab45f179d8] xs_sendpages at ffffffffc0679741 [sunrpc]
#28 [ffffacab45f17ac8] xs_tcp_send_request at ffffffffc067adb4 [sunrpc]
#29 [ffffacab45f17b20] xprt_transmit at ffffffffc067674c [sunrpc]
#30 [ffffacab45f17b90] call_transmit at ffffffffc0672064 [sunrpc]
#31 [ffffacab45f17ba0] __rpc_execute at ffffffffc067f365 [sunrpc]
#32 [ffffacab45f17bf8] rpc_run_task at ffffffffc06732c4 [sunrpc]
#33 [ffffacab45f17c38] nfs4_call_sync_custom at ffffffffc08e50bb [nfsv4]
#34 [ffffacab45f17c48] nfs4_call_sync_sequence at ffffffffc08e5143
[nfsv4]
#35 [ffffacab45f17cb8] _nfs4_proc_getattr at ffffffffc08e7f08 [nfsv4]
#36 [ffffacab45f17d78] nfs4_proc_getattr at ffffffffc08f200a [nfsv4]
#37 [ffffacab45f17de8] __nfs_revalidate_inode at ffffffffc08741d7 [nfs]
#38 [ffffacab45f17e18] nfs_getattr at ffffffffc0874458 [nfs]
#39 [ffffacab45f17e60] vfs_statx_fd at ffffffffadedf8a4
#40 [ffffacab45f17e98] __do_sys_newfstat at ffffffffadedfedd
#41 [ffffacab45f17f38] do_syscall_64 at ffffffffadc0419b
#42 [ffffacab45f17f50] entry_SYSCALL_64_after_hwframe at
ffffffffae6000ad
RIP: 00007f721ddcdd37 RSP: 00007ffc0d54cab8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00007f721e09b3c0 RCX: 00007f721ddcdd37
RDX: 00007ffc0d54cac0 RSI: 00007ffc0d54cac0 RDI: 0000000000000001
RBP: 00007f721e09f6c0 R8: 00007f721f87cf00 R9: 0000000000000000
R10: 00007ffc0d54a32a R11: 0000000000000246 R12: 00007f721e09b3c0
R13: 000055606b81376e R14: 0000000000000013 R15: 00007f721e09b3c0
ORIG_RAX: 0000000000000005 CS: 0033 SS: 002b
Ben
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: memalloc_nofs_save() only on async
2021-03-03 13:10 memalloc_nofs_save() only on async Benjamin Coddington
@ 2021-03-03 13:23 ` Trond Myklebust
0 siblings, 0 replies; 2+ messages in thread
From: Trond Myklebust @ 2021-03-03 13:23 UTC (permalink / raw)
To: bcodding; +Cc: linux-nfs
On Wed, 2021-03-03 at 08:10 -0500, Benjamin Coddington wrote:
> Hi Trond,
>
> I'd like to go back to setting sk->sk_allocation = GFP_NOIO (see
> a1231fda7e94). That would cover sync tasks as well as async, but I'm
> not
> sure what memalloc_nofs_save/restore gives us and if we should just
> try
> to
> apply that to all tasks.
>
> We're getting some folks deadlocked on the xprt_sending queue with
> stacks like:
>
> #0 [ffffacab45f17108] __schedule at ffffffffae4d1826
> #1 [ffffacab45f171a0] schedule at ffffffffae4d1cb8
> #2 [ffffacab45f171b0] rpc_wait_bit_killable at ffffffffc067d44e
> [sunrpc]
> #3 [ffffacab45f171c8] __wait_on_bit at ffffffffae4d216c
> #4 [ffffacab45f17200] out_of_line_wait_on_bit at ffffffffae4d2211
> #5 [ffffacab45f17250] __rpc_execute at ffffffffc067f3fc [sunrpc]
> #6 [ffffacab45f172a8] rpc_run_task at ffffffffc06732c4 [sunrpc]
> #7 [ffffacab45f172e8] nfs4_proc_layoutreturn at ffffffffc08f5d44
> [nfsv4]
> #8 [ffffacab45f17388] pnfs_send_layoutreturn at ffffffffc091946e
> [nfsv4]
> #9 [ffffacab45f173d8] _pnfs_return_layout at ffffffffc091ba8b
> [nfsv4]
> #10 [ffffacab45f17450] nfs4_evict_inode at ffffffffc0906a05 [nfsv4]
> #11 [ffffacab45f17460] evict at ffffffffadef8592
> #12 [ffffacab45f17480] dispose_list at ffffffffadef86a8
> #13 [ffffacab45f174a0] prune_icache_sb at ffffffffadef99a2
> #14 [ffffacab45f174c8] super_cache_scan at ffffffffadede183
> #15 [ffffacab45f17518] do_shrink_slab at ffffffffade3d5d8
> #16 [ffffacab45f17588] shrink_slab at ffffffffade3dab5
> #17 [ffffacab45f17608] shrink_node at ffffffffade42a8c
> #18 [ffffacab45f17678] do_try_to_free_pages at ffffffffade42e43
> #19 [ffffacab45f176c8] try_to_free_pages at ffffffffade431c8
> #20 [ffffacab45f17768] __alloc_pages_slowpath at ffffffffade81981
> #21 [ffffacab45f17868] __alloc_pages_nodemask at ffffffffade82555
> #22 [ffffacab45f178c8] skb_page_frag_refill at ffffffffae31bea7
> #23 [ffffacab45f178e0] sk_page_frag_refill at ffffffffae31c71d
> #24 [ffffacab45f178f8] tcp_sendmsg_locked at ffffffffae3cbe65
> #25 [ffffacab45f179a0] tcp_sendmsg at ffffffffae3cc8f7
> #26 [ffffacab45f179c0] sock_sendmsg at ffffffffae317cce
> #27 [ffffacab45f179d8] xs_sendpages at ffffffffc0679741 [sunrpc]
> #28 [ffffacab45f17ac8] xs_tcp_send_request at ffffffffc067adb4
> [sunrpc]
> #29 [ffffacab45f17b20] xprt_transmit at ffffffffc067674c [sunrpc]
> #30 [ffffacab45f17b90] call_transmit at ffffffffc0672064 [sunrpc]
> #31 [ffffacab45f17ba0] __rpc_execute at ffffffffc067f365 [sunrpc]
> #32 [ffffacab45f17bf8] rpc_run_task at ffffffffc06732c4 [sunrpc]
> #33 [ffffacab45f17c38] nfs4_call_sync_custom at ffffffffc08e50bb
> [nfsv4]
> #34 [ffffacab45f17c48] nfs4_call_sync_sequence at ffffffffc08e5143
> [nfsv4]
> #35 [ffffacab45f17cb8] _nfs4_proc_getattr at ffffffffc08e7f08 [nfsv4]
> #36 [ffffacab45f17d78] nfs4_proc_getattr at ffffffffc08f200a [nfsv4]
> #37 [ffffacab45f17de8] __nfs_revalidate_inode at ffffffffc08741d7
> [nfs]
> #38 [ffffacab45f17e18] nfs_getattr at ffffffffc0874458 [nfs]
> #39 [ffffacab45f17e60] vfs_statx_fd at ffffffffadedf8a4
> #40 [ffffacab45f17e98] __do_sys_newfstat at ffffffffadedfedd
> #41 [ffffacab45f17f38] do_syscall_64 at ffffffffadc0419b
> #42 [ffffacab45f17f50] entry_SYSCALL_64_after_hwframe at
> ffffffffae6000ad
> RIP: 00007f721ddcdd37 RSP: 00007ffc0d54cab8 RFLAGS: 00000246
> RAX: ffffffffffffffda RBX: 00007f721e09b3c0 RCX:
> 00007f721ddcdd37
> RDX: 00007ffc0d54cac0 RSI: 00007ffc0d54cac0 RDI:
> 0000000000000001
> RBP: 00007f721e09f6c0 R8: 00007f721f87cf00 R9:
> 0000000000000000
> R10: 00007ffc0d54a32a R11: 0000000000000246 R12:
> 00007f721e09b3c0
> R13: 000055606b81376e R14: 0000000000000013 R15:
> 00007f721e09b3c0
> ORIG_RAX: 0000000000000005 CS: 0033 SS: 002b
>
> Ben
>
Please just wrap that call to __rpc_execute() in rpc_execute() with a
memalloc_nofs_save()/memalloc_nofs_restore() pair. That should cause
the mm layer to do the correct thing here, and prevent re-entry into
the NFS code.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-03-04 0:29 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-03 13:10 memalloc_nofs_save() only on async Benjamin Coddington
2021-03-03 13:23 ` Trond Myklebust
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).