[PATCH RFC] xprtrdma: Release in-flight MRs on disconnect

[PATCH RFC] xprtrdma: Release in-flight MRs on disconnect

[Chuck Lever]

Dan Aloni reports that when a server disconnects abruptly, a few
memory regions are left DMA mapped. Over time this leak could pin
enough I/O resources to slow or even deadlock an NFS/RDMA client.

I found that if a transport disconnects before pending Send and
FastReg WRs can be posted, the to-be-registered MRs are stranded on
the req's rl_registered list and never released -- since they
weren't posted, there's no Send completion to DMA unmap them.

Reported-by: Dan Aloni <dan@kernelim.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/sunrpc/xprtrdma/verbs.c |    2 ++
 1 file changed, 2 insertions(+)

Hi Dan, does this help?


diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 95c66a339e34..53962e41896d 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -936,6 +936,8 @@ static void rpcrdma_req_reset(struct rpcrdma_req *req)
 
 	rpcrdma_regbuf_dma_unmap(req->rl_sendbuf);
 	rpcrdma_regbuf_dma_unmap(req->rl_recvbuf);
+
+	frwr_reset(req);
 }
 
 /* ASSUMPTION: the rb_allreqs list is stable for the duration,

Re: [PATCH RFC] xprtrdma: Release in-flight MRs on disconnect

[Dan Aloni]

On Mon, Aug 17, 2020 at 11:19:26AM -0400, Chuck Lever wrote:
> Dan Aloni reports that when a server disconnects abruptly, a few
> memory regions are left DMA mapped. Over time this leak could pin
> enough I/O resources to slow or even deadlock an NFS/RDMA client.
> 
> I found that if a transport disconnects before pending Send and
> FastReg WRs can be posted, the to-be-registered MRs are stranded on
> the req's rl_registered list and never released -- since they
> weren't posted, there's no Send completion to DMA unmap them.
> 
> Reported-by: Dan Aloni <dan@kernelim.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  net/sunrpc/xprtrdma/verbs.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> Hi Dan, does this help?

Yeah, works as expected with your fix.

Thanks!

-- 
Dan Aloni

Re: [PATCH RFC] xprtrdma: Release in-flight MRs on disconnect

[Chuck Lever]

> On Aug 17, 2020, at 11:19 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
> 
> Dan Aloni reports that when a server disconnects abruptly, a few
> memory regions are left DMA mapped. Over time this leak could pin
> enough I/O resources to slow or even deadlock an NFS/RDMA client.
> 
> I found that if a transport disconnects before pending Send and
> FastReg WRs can be posted, the to-be-registered MRs are stranded on
> the req's rl_registered list and never released -- since they
> weren't posted, there's no Send completion to DMA unmap them.
> 
> Reported-by: Dan Aloni <dan@kernelim.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
> net/sunrpc/xprtrdma/verbs.c |    2 ++
> 1 file changed, 2 insertions(+)
> 
> Hi Dan, does this help?

Hi Anna, can you grab this for v5.9-rc ?


> diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
> index 95c66a339e34..53962e41896d 100644
> --- a/net/sunrpc/xprtrdma/verbs.c
> +++ b/net/sunrpc/xprtrdma/verbs.c
> @@ -936,6 +936,8 @@ static void rpcrdma_req_reset(struct rpcrdma_req *req)
> 
> 	rpcrdma_regbuf_dma_unmap(req->rl_sendbuf);
> 	rpcrdma_regbuf_dma_unmap(req->rl_recvbuf);
> +
> +	frwr_reset(req);
> }
> 
> /* ASSUMPTION: the rb_allreqs list is stable for the duration,
> 
> 

--
Chuck Lever