Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Chuck Lever <chuck.lever@oracle.com>
To: Trond Myklebust <trondmy@hammerspace.com>
Cc: Bruce Fields <bfields@fieldses.org>, Neil Brown <neilb@suse.de>,
	Anna Schumaker <anna.schumaker@netapp.com>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] SUNRPC: backchannel RPC request must reference XPRT
Date: Tue, 15 Oct 2019 17:47:09 -0400
Message-ID: <94FD3327-AEE7-4160-87E4-4E7569FB8D18@oracle.com> (raw)
In-Reply-To: <711ebfa5340c6e29ff640e855db5ad8e41a09a60.camel@hammerspace.com>



> On Oct 15, 2019, at 5:16 PM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> Hi Neil,
> 
> On Tue, 2019-10-15 at 10:36 +1100, NeilBrown wrote:
>> The backchannel RPC requests - that are queued waiting
>> for the reply to be sent by the "NFSv4 callback" thread -
>> have a pointer to the xprt, but it is not reference counted.
>> It is possible for the xprt to be freed while there are
>> still queued requests.
>> 
>> I think this has been a problem since
>> Commit fb7a0b9addbd ("nfs41: New backchannel helper routines")
>> when the code was introduced, but I suspect it became more of
>> a problem after
>> Commit 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
>> transports")
>> (or there abouts).
>> Before this second patch, the nfs client would hold a reference to
>> the xprt to keep it alive.  After multipath was introduced,
>> a client holds a reference to a swtich, and the switch can have
>> multiple
>> xprts which can be added and removed.
>> 
>> I'm not sure of all the causal issues, but this patch has
>> fixed a customer problem were an NFSv4.1 client would run out
>> of memory with tens of thousands of backchannel rpc requests
>> queued for an xprt that had been freed.  This was a 64K-page
>> machine so each rpc_rqst consumed more than 128K of memory.
>> 
>> Fixes: 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
>> transports")
>> cc: stable@vger.kernel.org (v4.6)
>> Signed-off-by: NeilBrown <neilb@suse.de>
>> ---
>> net/sunrpc/backchannel_rqst.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/net/sunrpc/backchannel_rqst.c
>> b/net/sunrpc/backchannel_rqst.c
>> index 339e8c077c2d..c95ca39688b6 100644
>> --- a/net/sunrpc/backchannel_rqst.c
>> +++ b/net/sunrpc/backchannel_rqst.c
>> @@ -61,6 +61,7 @@ static void xprt_free_allocation(struct rpc_rqst
>> *req)
>> 	free_page((unsigned long)xbufp->head[0].iov_base);
>> 	xbufp = &req->rq_snd_buf;
>> 	free_page((unsigned long)xbufp->head[0].iov_base);
>> +	xprt_put(req->rq_xprt);
>> 	kfree(req);
>> }
> 
> Would it perhaps make better sense to move the xprt_get() to
> xprt_lookup_bc_request() and to release it in xprt_free_bc_rqst()?

/me wonders if the same problem exists for the RPC/RDMA backchannel....


> Otherwise as far as I can tell, we will have freed slots on the xprt-
>> bc_pa_list that hold a reference to the transport itself, meaning that
> the latter never gets released.
> 
>> 
>> @@ -85,7 +86,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt
>> *xprt, gfp_t gfp_flags)
>> 	if (req == NULL)
>> 		return NULL;
>> 
>> -	req->rq_xprt = xprt;
>> +	req->rq_xprt = xprt_get(xprt);
>> 	INIT_LIST_HEAD(&req->rq_bc_list);
>> 
>> 	/* Preallocate one XDR receive buffer */
> -- 
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com

--
Chuck Lever




  reply index

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-09  5:15 NFSv4.1 backchannel xprt problems NeilBrown
2019-10-11 16:56 ` bfields
2019-10-14 23:36   ` [PATCH] SUNRPC: backchannel RPC request must reference XPRT NeilBrown
2019-10-15 21:16     ` Trond Myklebust
2019-10-15 21:47       ` Chuck Lever [this message]
2019-10-15 23:23       ` NeilBrown
2019-10-16  3:04         ` Trond Myklebust
2019-10-16  4:38           ` NeilBrown

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=94FD3327-AEE7-4160-87E4-4E7569FB8D18@oracle.com \
    --to=chuck.lever@oracle.com \
    --cc=anna.schumaker@netapp.com \
    --cc=bfields@fieldses.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=neilb@suse.de \
    --cc=trondmy@hammerspace.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git