Re: NFS regression between 5.17 and 5.18

[Trond Myklebust <trondmy@hammerspace.com>]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

On Thu, 2022-04-28 at 15:47 -0400, Dennis Dalessandro wrote:
> On 4/28/22 11:42 AM, Dennis Dalessandro wrote:
> > On 4/28/22 10:57 AM, Chuck Lever III wrote:
> > > > On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro
> > > > <dennis.dalessandro@cornelisnetworks.com> wrote:
> > > > 
> > > > Hi NFS folks,
> > > > 
> > > > I've noticed a pretty nasty regression in our NFS capability
> > > > between 5.17 and
> > > > 5.18-rc1. I've tried to bisect but not having any luck. The
> > > > problem I'm seeing
> > > > is it takes 3 minutes to copy a file from NFS to the local
> > > > disk. When it should
> > > > take less than half a second, which it did up through 5.17.
> > > > 
> > > > It doesn't seem to be network related, but can't rule that out
> > > > completely.
> > > > 
> > > > I tried to bisect but the problem can be intermittent. Some
> > > > runs I'll see a
> > > > problem in 3 out of 100 cycles, sometimes 0 out of 100.
> > > > Sometimes I'll see it
> > > > 100 out of 100.
> > > 
> > > It's not clear from your problem report whether the problem
> > > appears
> > > when it's the server running v5.18-rc or the client.
> > 
> > That's because I don't know which it is. I'll do a quick test and
> > find out. I
> > was testing the same kernel across both nodes.
> 
> Looks like it is the client.
> 
> server  client  result
> ------  ------  ------
> 5.17    5.17    Pass
> 5.17    5.18    Fail
> 5.18    5.18    Fail
> 5.18    5.17    Pass
> 
> Is there a patch for the client issue you mentioned that I could try?
> 
> -Denny

Try this one


-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-SUNRPC-Fix-an-RPC-RDMA-performance-regression.patch --]
[-- Type: text/x-patch; name="0001-SUNRPC-Fix-an-RPC-RDMA-performance-regression.patch", Size: 2444 bytes --]

From 0505ecf76f5131525e696d3dae3bd456a8b8b3ad Mon Sep 17 00:00:00 2001
From: Trond Myklebust <trond.myklebust@hammerspace.com>
Date: Thu, 28 Apr 2022 15:46:01 -0400
Subject: [PATCH] SUNRPC: Fix an RPC/RDMA performance regression

Use the standard gfp mask instead of using GFP_NOWAIT. The latter causes
issues when under memory pressure.

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 net/sunrpc/auth_gss/auth_gss.c  | 11 ++++-------
 net/sunrpc/sched.c              |  1 +
 net/sunrpc/xprtrdma/transport.c |  6 +-----
 3 files changed, 6 insertions(+), 12 deletions(-)

diff --git a/net/sunrpc/auth_gss/auth_gss.c b/net/sunrpc/auth_gss/auth_gss.c
index de7e5b41ab8f..a31a27816cc0 100644
--- a/net/sunrpc/auth_gss/auth_gss.c
+++ b/net/sunrpc/auth_gss/auth_gss.c
@@ -1340,14 +1340,11 @@ gss_hash_cred(struct auth_cred *acred, unsigned int hashbits)
 /*
  * Lookup RPCSEC_GSS cred for the current process
  */
-static struct rpc_cred *
-gss_lookup_cred(struct rpc_auth *auth, struct auth_cred *acred, int flags)
+static struct rpc_cred *gss_lookup_cred(struct rpc_auth *auth,
+					struct auth_cred *acred, int flags)
 {
-	gfp_t gfp = GFP_KERNEL;
-
-	if (flags & RPCAUTH_LOOKUP_ASYNC)
-		gfp = GFP_NOWAIT | __GFP_NOWARN;
-	return rpcauth_lookup_credcache(auth, acred, flags, gfp);
+	return rpcauth_lookup_credcache(auth, acred, flags,
+					rpc_task_gfp_mask());
 }
 
 static struct rpc_cred *
diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
index 7f70c1e608b7..25b9221950ff 100644
--- a/net/sunrpc/sched.c
+++ b/net/sunrpc/sched.c
@@ -63,6 +63,7 @@ gfp_t rpc_task_gfp_mask(void)
 		return GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN;
 	return GFP_KERNEL;
 }
+EXPORT_SYMBOL_GPL(rpc_task_gfp_mask);
 
 unsigned long
 rpc_task_timeout(const struct rpc_task *task)
diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c
index 6b7e10e5a141..bcb37b51adf6 100644
--- a/net/sunrpc/xprtrdma/transport.c
+++ b/net/sunrpc/xprtrdma/transport.c
@@ -571,11 +571,7 @@ xprt_rdma_allocate(struct rpc_task *task)
 	struct rpc_rqst *rqst = task->tk_rqstp;
 	struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt);
 	struct rpcrdma_req *req = rpcr_to_rdmar(rqst);
-	gfp_t flags;
-
-	flags = RPCRDMA_DEF_GFP;
-	if (RPC_IS_ASYNC(task))
-		flags = GFP_NOWAIT | __GFP_NOWARN;
+	gfp_t flags = rpc_task_gfp_mask();
 
 	if (!rpcrdma_check_regbuf(r_xprt, req->rl_sendbuf, rqst->rq_callsize,
 				  flags))
-- 
2.35.1