linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Chuck Lever <chuck.lever@oracle.com>
To: linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org
Subject: [PATCH v1 05/11] xprtrdma: Allocate Protection Domain in rpcrdma_ep_create()
Date: Fri, 21 Feb 2020 17:00:33 -0500	[thread overview]
Message-ID: <20200221220033.2072.22880.stgit@manet.1015granger.net> (raw)
In-Reply-To: <20200221214906.2072.32572.stgit@manet.1015granger.net>

Make a Protection Domain (PD) a per-connection resource rather than
a per-transport resource. In other words, when the connection
terminates, the PD is destroyed.

Thus there is one less HW resource that remains allocated to a
transport after a connection is closed.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 net/sunrpc/xprtrdma/verbs.c |   26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)

diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index f361213a8157..36fe7baea014 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -363,14 +363,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
 		rc = PTR_ERR(ia->ri_id);
 		goto out_err;
 	}
-
-	ia->ri_pd = ib_alloc_pd(ia->ri_id->device, 0);
-	if (IS_ERR(ia->ri_pd)) {
-		rc = PTR_ERR(ia->ri_pd);
-		pr_err("rpcrdma: ib_alloc_pd() returned %d\n", rc);
-		goto out_err;
-	}
-
 	return 0;
 
 out_err:
@@ -403,9 +395,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
 
 	rpcrdma_ep_destroy(r_xprt);
 
-	ib_dealloc_pd(ia->ri_pd);
-	ia->ri_pd = NULL;
-
 	/* Allow waiters to continue */
 	complete(&ia->ri_remove_done);
 
@@ -423,11 +412,6 @@ static void rpcrdma_update_cm_private(struct rpcrdma_xprt *r_xprt,
 	if (ia->ri_id && !IS_ERR(ia->ri_id))
 		rdma_destroy_id(ia->ri_id);
 	ia->ri_id = NULL;
-
-	/* If the pd is still busy, xprtrdma missed freeing a resource */
-	if (ia->ri_pd && !IS_ERR(ia->ri_pd))
-		ib_dealloc_pd(ia->ri_pd);
-	ia->ri_pd = NULL;
 }
 
 static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt,
@@ -514,6 +498,12 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt,
 	ep->rep_remote_cma.flow_control = 0;
 	ep->rep_remote_cma.rnr_retry_count = 0;
 
+	ia->ri_pd = ib_alloc_pd(id->device, 0);
+	if (IS_ERR(ia->ri_pd)) {
+		rc = PTR_ERR(ia->ri_pd);
+		goto out_destroy;
+	}
+
 	rc = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr);
 	if (rc)
 		goto out_destroy;
@@ -540,6 +530,10 @@ static void rpcrdma_ep_destroy(struct rpcrdma_xprt *r_xprt)
 	if (ep->rep_attr.send_cq)
 		ib_free_cq(ep->rep_attr.send_cq);
 	ep->rep_attr.send_cq = NULL;
+
+	if (ia->ri_pd)
+		ib_dealloc_pd(ia->ri_pd);
+	ia->ri_pd = NULL;
 }
 
 /* Re-establish a connection after a device removal event.


  parent reply	other threads:[~2020-02-21 22:00 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-21 22:00 [PATCH v1 00/11] NFS/RDMA client side connection overhaul Chuck Lever
2020-02-21 22:00 ` [PATCH v1 01/11] xprtrdma: Invoke rpcrdma_ep_create() in the connect worker Chuck Lever
2020-02-21 22:00 ` [PATCH v1 02/11] xprtrdma: Refactor frwr_init_mr() Chuck Lever
2020-02-21 22:00 ` [PATCH v1 03/11] xprtrdma: Clean up the post_send path Chuck Lever
2020-02-21 22:00 ` [PATCH v1 04/11] xprtrdma: Refactor rpcrdma_ep_connect() and rpcrdma_ep_disconnect() Chuck Lever
2020-02-21 22:00 ` Chuck Lever [this message]
2020-03-01 18:11   ` [PATCH v1 05/11] xprtrdma: Allocate Protection Domain in rpcrdma_ep_create() Tom Talpey
2020-03-01 18:29     ` Chuck Lever
2020-03-01 18:38       ` Tom Talpey
2020-02-21 22:00 ` [PATCH v1 06/11] xprtrdma: Invoke rpcrdma_ia_open in the connect worker Chuck Lever
2020-02-21 22:00 ` [PATCH v1 07/11] xprtrdma: Remove rpcrdma_ia::ri_flags Chuck Lever
2020-02-21 22:00 ` [PATCH v1 08/11] xprtrdma: Disconnect on flushed completion Chuck Lever
2020-02-21 22:00 ` [PATCH v1 09/11] xprtrdma: Merge struct rpcrdma_ia into struct rpcrdma_ep Chuck Lever
2020-02-21 22:01 ` [PATCH v1 10/11] xprtrdma: Extract sockaddr from struct rdma_cm_id Chuck Lever
2020-02-24 16:15   ` Anna Schumaker
2020-02-24 16:18     ` Chuck Lever
2020-02-24 16:23       ` Anna Schumaker
2020-02-21 22:01 ` [PATCH v1 11/11] xprtrdma: kmalloc rpcrdma_ep separate from rpcrdma_xprt Chuck Lever
2020-03-01 18:09 ` [PATCH v1 00/11] NFS/RDMA client side connection overhaul Tom Talpey
2020-03-01 18:12   ` Chuck Lever
2020-03-11 15:27 ` Chuck Lever
2020-03-11 17:16   ` Schumaker, Anna

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200221220033.2072.22880.stgit@manet.1015granger.net \
    --to=chuck.lever@oracle.com \
    --cc=linux-nfs@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).