[net] rds: ib: update WR sizes when bringing up connection

[net] rds: ib: update WR sizes when bringing up connection

[Dag Moxnes]

Currently WR sizes are updated from rds_ib_sysctl_max_send_wr and
rds_ib_sysctl_max_recv_wr when a connection is shut down. As a result,
a connection being down while rds_ib_sysctl_max_send_wr or
rds_ib_sysctl_max_recv_wr are updated, will not update the sizes when
it comes back up.

Move resizing of WRs to rds_ib_setup_qp so that connections will be setup
with the most current WR sizes.

Signed-off-by: Dag Moxnes <dag.moxnes@oracle.com>
---
 net/rds/ib_cm.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 233f136816..18c6fac6ea 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -450,6 +450,7 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
 	struct ib_qp_init_attr attr;
 	struct ib_cq_init_attr cq_attr = {};
 	struct rds_ib_device *rds_ibdev;
+	unsigned long max_wrs;
 	int ret, fr_queue_space;
 
 	/*
@@ -469,10 +470,15 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
 	/* add the conn now so that connection establishment has the dev */
 	rds_ib_add_conn(rds_ibdev, conn);
 
-	if (rds_ibdev->max_wrs < ic->i_send_ring.w_nr + 1)
-		rds_ib_ring_resize(&ic->i_send_ring, rds_ibdev->max_wrs - 1);
-	if (rds_ibdev->max_wrs < ic->i_recv_ring.w_nr + 1)
-		rds_ib_ring_resize(&ic->i_recv_ring, rds_ibdev->max_wrs - 1);
+	max_wrs = rds_ibdev->max_wrs < rds_ib_sysctl_max_send_wr + 1 ?
+		rds_ibdev->max_wrs - 1 : rds_ib_sysctl_max_send_wr;
+	if (ic->i_send_ring.w_nr != max_wrs)
+		rds_ib_ring_resize(&ic->i_send_ring, max_wrs);
+
+	max_wrs = rds_ibdev->max_wrs < rds_ib_sysctl_max_recv_wr + 1 ?
+		rds_ibdev->max_wrs - 1 : rds_ib_sysctl_max_recv_wr;
+	if (ic->i_recv_ring.w_nr != max_wrs)
+		rds_ib_ring_resize(&ic->i_recv_ring, max_wrs);
 
 	/* Protection domain and memory range */
 	ic->i_pd = rds_ibdev->pd;
@@ -1099,8 +1105,9 @@ void rds_ib_conn_path_shutdown(struct rds_conn_path *cp)
 	ic->i_flowctl = 0;
 	atomic_set(&ic->i_credits, 0);
 
-	rds_ib_ring_init(&ic->i_send_ring, rds_ib_sysctl_max_send_wr);
-	rds_ib_ring_init(&ic->i_recv_ring, rds_ib_sysctl_max_recv_wr);
+	/* Re-init rings, but retain sizes. */
+	rds_ib_ring_init(&ic->i_send_ring, ic->i_send_ring.w_nr);
+	rds_ib_ring_init(&ic->i_recv_ring, ic->i_recv_ring.w_nr);
 
 	if (ic->i_ibinc) {
 		rds_inc_put(&ic->i_ibinc->ii_inc);
@@ -1147,8 +1154,8 @@ int rds_ib_conn_alloc(struct rds_connection *conn, gfp_t gfp)
 	 * rds_ib_conn_shutdown() waits for these to be emptied so they
 	 * must be initialized before it can be called.
 	 */
-	rds_ib_ring_init(&ic->i_send_ring, rds_ib_sysctl_max_send_wr);
-	rds_ib_ring_init(&ic->i_recv_ring, rds_ib_sysctl_max_recv_wr);
+	rds_ib_ring_init(&ic->i_send_ring, 0);
+	rds_ib_ring_init(&ic->i_recv_ring, 0);
 
 	ic->conn = conn;
 	conn->c_transport_data = ic;
-- 
2.20.1

Re: [net] rds: ib: update WR sizes when bringing up connection

[santosh.shilimkar]

On 11/15/19 12:56 AM, Dag Moxnes wrote:
> Currently WR sizes are updated from rds_ib_sysctl_max_send_wr and
> rds_ib_sysctl_max_recv_wr when a connection is shut down. As a result,
> a connection being down while rds_ib_sysctl_max_send_wr or
> rds_ib_sysctl_max_recv_wr are updated, will not update the sizes when
> it comes back up.
> 
> Move resizing of WRs to rds_ib_setup_qp so that connections will be setup
> with the most current WR sizes.
> 
> Signed-off-by: Dag Moxnes <dag.moxnes@oracle.com>
> ---
Looks correct to me. Thanks Dag.

Acked-by: Santosh Shilimkar <santosh.shilimkar@oracle.com>

Re: [net] rds: ib: update WR sizes when bringing up connection

[David Miller]

From: Dag Moxnes <dag.moxnes@oracle.com>
Date: Fri, 15 Nov 2019 09:56:01 +0100

> Currently WR sizes are updated from rds_ib_sysctl_max_send_wr and
> rds_ib_sysctl_max_recv_wr when a connection is shut down. As a result,
> a connection being down while rds_ib_sysctl_max_send_wr or
> rds_ib_sysctl_max_recv_wr are updated, will not update the sizes when
> it comes back up.
> 
> Move resizing of WRs to rds_ib_setup_qp so that connections will be setup
> with the most current WR sizes.
> 
> Signed-off-by: Dag Moxnes <dag.moxnes@oracle.com>

Applied.