Re: [PATCH] xprtrdma: make sure MRs are unmapped before freeing them

[Chuck Lever <chuck.lever@oracle.com>]

Hi Dan-

> On Aug 14, 2020, at 1:37 PM, Dan Aloni <dan@kernelim.com> wrote:
> 
> It was observed that on disconnections, these unmaps don't occur. The
> relevant path is rpcrdma_mrs_destroy(), being called from
> rpcrdma_xprt_disconnect().

MRs are supposed to be unmapped right after they are used, so
during disconnect they should all be unmapped already. How often
do you see a DMA mapped MR in this code path? Do you have a
reproducer I can try?


> Signed-off-by: Dan Aloni <dan@kernelim.com>
> ---
> net/sunrpc/xprtrdma/frwr_ops.c | 22 ++++++++++++++++------
> 1 file changed, 16 insertions(+), 6 deletions(-)
> 
> diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c
> index 7f94c9a19fd3..3899a5310214 100644
> --- a/net/sunrpc/xprtrdma/frwr_ops.c
> +++ b/net/sunrpc/xprtrdma/frwr_ops.c
> @@ -49,6 +49,19 @@
> # define RPCDBG_FACILITY	RPCDBG_TRANS
> #endif
> 
> +static void frwr_mr_unmap(struct rpcrdma_mr *mr)
> +{
> +	struct rpcrdma_xprt *r_xprt = mr->mr_xprt;
> +
> +	if (mr->mr_dir == DMA_NONE)
> +		return;
> +
> +	trace_xprtrdma_mr_unmap(mr);
> +	ib_dma_unmap_sg(r_xprt->rx_ep->re_id->device,
> +			mr->mr_sg, mr->mr_nents, mr->mr_dir);
> +	mr->mr_dir = DMA_NONE;
> +}
> +
> /**
>  * frwr_release_mr - Destroy one MR
>  * @mr: MR allocated by frwr_mr_init
> @@ -58,6 +71,8 @@ void frwr_release_mr(struct rpcrdma_mr *mr)
> {
> 	int rc;
> 
> +	frwr_mr_unmap(mr);
> +
> 	rc = ib_dereg_mr(mr->frwr.fr_mr);
> 	if (rc)
> 		trace_xprtrdma_frwr_dereg(mr, rc);
> @@ -71,12 +86,7 @@ static void frwr_mr_recycle(struct rpcrdma_mr *mr)
> 
> 	trace_xprtrdma_mr_recycle(mr);
> 
> -	if (mr->mr_dir != DMA_NONE) {
> -		trace_xprtrdma_mr_unmap(mr);
> -		ib_dma_unmap_sg(r_xprt->rx_ep->re_id->device,
> -				mr->mr_sg, mr->mr_nents, mr->mr_dir);
> -		mr->mr_dir = DMA_NONE;
> -	}
> +	frwr_mr_unmap(mr);
> 
> 	spin_lock(&r_xprt->rx_buf.rb_lock);
> 	list_del(&mr->mr_all);
> -- 
> 2.25.4
> 

--
Chuck Lever