From: Trond Myklebust <trondmy@hammerspace.com>
To: "bfields@fieldses.org" <bfields@fieldses.org>,
"neilb@suse.de" <neilb@suse.de>,
"anna.schumaker@netapp.com" <anna.schumaker@netapp.com>
Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] SUNRPC: backchannel RPC request must reference XPRT
Date: Tue, 15 Oct 2019 21:16:07 +0000 [thread overview]
Message-ID: <711ebfa5340c6e29ff640e855db5ad8e41a09a60.camel@hammerspace.com> (raw)
In-Reply-To: <87imoqrjb8.fsf@notabene.neil.brown.name>
Hi Neil,
On Tue, 2019-10-15 at 10:36 +1100, NeilBrown wrote:
> The backchannel RPC requests - that are queued waiting
> for the reply to be sent by the "NFSv4 callback" thread -
> have a pointer to the xprt, but it is not reference counted.
> It is possible for the xprt to be freed while there are
> still queued requests.
>
> I think this has been a problem since
> Commit fb7a0b9addbd ("nfs41: New backchannel helper routines")
> when the code was introduced, but I suspect it became more of
> a problem after
> Commit 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
> transports")
> (or there abouts).
> Before this second patch, the nfs client would hold a reference to
> the xprt to keep it alive. After multipath was introduced,
> a client holds a reference to a swtich, and the switch can have
> multiple
> xprts which can be added and removed.
>
> I'm not sure of all the causal issues, but this patch has
> fixed a customer problem were an NFSv4.1 client would run out
> of memory with tens of thousands of backchannel rpc requests
> queued for an xprt that had been freed. This was a 64K-page
> machine so each rpc_rqst consumed more than 128K of memory.
>
> Fixes: 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
> transports")
> cc: stable@vger.kernel.org (v4.6)
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
> net/sunrpc/backchannel_rqst.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/net/sunrpc/backchannel_rqst.c
> b/net/sunrpc/backchannel_rqst.c
> index 339e8c077c2d..c95ca39688b6 100644
> --- a/net/sunrpc/backchannel_rqst.c
> +++ b/net/sunrpc/backchannel_rqst.c
> @@ -61,6 +61,7 @@ static void xprt_free_allocation(struct rpc_rqst
> *req)
> free_page((unsigned long)xbufp->head[0].iov_base);
> xbufp = &req->rq_snd_buf;
> free_page((unsigned long)xbufp->head[0].iov_base);
> + xprt_put(req->rq_xprt);
> kfree(req);
> }
Would it perhaps make better sense to move the xprt_get() to
xprt_lookup_bc_request() and to release it in xprt_free_bc_rqst()?
Otherwise as far as I can tell, we will have freed slots on the xprt-
>bc_pa_list that hold a reference to the transport itself, meaning that
the latter never gets released.
>
> @@ -85,7 +86,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt
> *xprt, gfp_t gfp_flags)
> if (req == NULL)
> return NULL;
>
> - req->rq_xprt = xprt;
> + req->rq_xprt = xprt_get(xprt);
> INIT_LIST_HEAD(&req->rq_bc_list);
>
> /* Preallocate one XDR receive buffer */
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com
next prev parent reply other threads:[~2019-10-15 21:16 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-09 5:15 NFSv4.1 backchannel xprt problems NeilBrown
2019-10-11 16:56 ` J. Bruce Fields
2019-10-14 23:36 ` [PATCH] SUNRPC: backchannel RPC request must reference XPRT NeilBrown
2019-10-15 21:16 ` Trond Myklebust [this message]
2019-10-15 21:47 ` Chuck Lever
2019-10-15 23:23 ` NeilBrown
2019-10-16 3:04 ` Trond Myklebust
2019-10-16 4:38 ` NeilBrown
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=711ebfa5340c6e29ff640e855db5ad8e41a09a60.camel@hammerspace.com \
--to=trondmy@hammerspace.com \
--cc=anna.schumaker@netapp.com \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
--cc=neilb@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).