* [PATCH for-next 00/13] rtrs: misc fix and cleanup
@ 2020-10-12 13:18 Jack Wang
2020-10-12 13:18 ` [PATCH for-next 01/13] RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed Jack Wang
` (12 more replies)
0 siblings, 13 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma; +Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang
Hi Jason, hi Doug,
Please consider to include following changes to upstream.
Danil Kipnis (1):
RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed
Gioh Kim (6):
RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess
RDMA/rtrs: removed unused filed list of rtrs_iu
RDMA/rtrs: remove unnecessary argument dir of rtrs_iu_free
RDMA/rtrs-clt: remove duplicated switch-case handling for CM error
events
RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established
RDMA/rtrs-clt: remove duplicated code
Guoqing Jiang (4):
RDMA/rtrs-srv: fix typo
RDMA/rtrs-srv: kill rtrs_srv_change_state_get_old
RDMA/rtrs: introduce rtrs_post_send
RDMA/rtrs-clt: remove 'addr' from rtrs_clt_add_path_to_arr
Jack Wang (2):
RDMA/rtrs-clt: remove outdated comment in create_con_cq_qp
RDMA/rtrs-clt: avoid run destroy_con_cq_qp/create_con_cq_qp in
parallel
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 103 +++++++++++--------------
drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 +
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 5 +-
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 33 +++-----
drivers/infiniband/ulp/rtrs/rtrs-srv.h | 2 +-
drivers/infiniband/ulp/rtrs/rtrs.c | 61 ++++++---------
6 files changed, 82 insertions(+), 123 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH for-next 01/13] RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 02/13] RDMA/rtrs-clt: remove outdated comment in create_con_cq_qp Jack Wang
` (11 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma; +Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang
From: Danil Kipnis <danil.kipnis@cloud.ionos.com>
We call destroy_con_cq_qp(con) in rtrs_rdma_addr_resolved() in case route
couldn't be resolved and then again in create_cm() because nothing happens.
Don't call destroy_con_cq_qp from rtrs_rdma_addr_resolved, create_cm()
does the clean up already.
Signed-off-by: Danil Kipnis <danil.kipnis@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 776e89231c52..9980bb4a6f78 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1640,10 +1640,8 @@ static int rtrs_rdma_addr_resolved(struct rtrs_clt_con *con)
return err;
}
err = rdma_resolve_route(con->c.cm_id, RTRS_CONNECT_TIMEOUT_MS);
- if (err) {
+ if (err)
rtrs_err(s, "Resolving route failed, err: %d\n", err);
- destroy_con_cq_qp(con);
- }
return err;
}
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 02/13] RDMA/rtrs-clt: remove outdated comment in create_con_cq_qp
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
2020-10-12 13:18 ` [PATCH for-next 01/13] RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 03/13] RDMA/rtrs-clt: avoid run destroy_con_cq_qp/create_con_cq_qp in parallel Jack Wang
` (10 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
As run destroy_con_cq_qp many times doesn't work, remove
the comments.
Suggested-by: Gioh Kim <gioh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 9980bb4a6f78..fb840b152b37 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1520,15 +1520,6 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
int err, cq_vector;
struct rtrs_msg_rkey_rsp *rsp;
- /*
- * This function can fail, but still destroy_con_cq_qp() should
- * be called, this is because create_con_cq_qp() is called on cm
- * event path, thus caller/waiter never knows: have we failed before
- * create_con_cq_qp() or after. To solve this dilemma without
- * creating any additional flags just allow destroy_con_cq_qp() be
- * called many times.
- */
-
if (con->c.cid == 0) {
/*
* One completion for each receive and two for each send
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 03/13] RDMA/rtrs-clt: avoid run destroy_con_cq_qp/create_con_cq_qp in parallel
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
2020-10-12 13:18 ` [PATCH for-next 01/13] RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed Jack Wang
2020-10-12 13:18 ` [PATCH for-next 02/13] RDMA/rtrs-clt: remove outdated comment in create_con_cq_qp Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess Jack Wang
` (9 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma; +Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang
For redmine #54314, too kworker.
addr_resolver kworker reconnect kworker
rtrs_clt_rdma_cm_handler
rtrs_rdma_addr_resolved
create_con_cq_qp: s.dev_ref++
"s.dev_ref is 1"
wait in create_cm fails with TIMEOUT
destroy_con_cq_qp: --s.dev_ref
"s.dev_ref is 0"
destroy_con_cq_qp: sess->s.dev = NULL
rtrs_cq_qp_create -> create_qp(con, sess->dev->ib_pd...)
sess->dev is NULL, panic.
To fix the problem using mutex to serialize create_con_cq_qp
and destroy_con_cq_qp.
redmine: https://redmine.pb.local/issues/54314
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 15 +++++++++++++--
drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 +
2 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index fb840b152b37..4677e8ed29ae 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1499,6 +1499,7 @@ static int create_con(struct rtrs_clt_sess *sess, unsigned int cid)
con->c.cid = cid;
con->c.sess = &sess->s;
atomic_set(&con->io_cnt, 0);
+ mutex_init(&con->con_mutex);
sess->s.con[cid] = &con->c;
@@ -1510,6 +1511,7 @@ static void destroy_con(struct rtrs_clt_con *con)
struct rtrs_clt_sess *sess = to_clt_sess(con->c.sess);
sess->s.con[con->c.cid] = NULL;
+ mutex_destroy(&con->con_mutex);
kfree(con);
}
@@ -1520,6 +1522,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
int err, cq_vector;
struct rtrs_msg_rkey_rsp *rsp;
+ lockdep_assert_held(&con->con_mutex);
if (con->c.cid == 0) {
/*
* One completion for each receive and two for each send
@@ -1593,7 +1596,7 @@ static void destroy_con_cq_qp(struct rtrs_clt_con *con)
* Be careful here: destroy_con_cq_qp() can be called even
* create_con_cq_qp() failed, see comments there.
*/
-
+ lockdep_assert_held(&con->con_mutex);
rtrs_cq_qp_destroy(&con->c);
if (con->rsp_ius) {
rtrs_iu_free(con->rsp_ius, DMA_FROM_DEVICE,
@@ -1625,7 +1628,9 @@ static int rtrs_rdma_addr_resolved(struct rtrs_clt_con *con)
struct rtrs_sess *s = con->c.sess;
int err;
+ mutex_lock(&con->con_mutex);
err = create_con_cq_qp(con);
+ mutex_unlock(&con->con_mutex);
if (err) {
rtrs_err(s, "create_con_cq_qp(), err: %d\n", err);
return err;
@@ -1938,8 +1943,9 @@ static int create_cm(struct rtrs_clt_con *con)
errr:
stop_cm(con);
- /* Is safe to call destroy if cq_qp is not inited */
+ mutex_lock(&con->con_mutex);
destroy_con_cq_qp(con);
+ mutex_unlock(&con->con_mutex);
destroy_cm:
destroy_cm(con);
@@ -2046,7 +2052,9 @@ static void rtrs_clt_stop_and_destroy_conns(struct rtrs_clt_sess *sess)
if (!sess->s.con[cid])
break;
con = to_clt_con(sess->s.con[cid]);
+ mutex_lock(&con->con_mutex);
destroy_con_cq_qp(con);
+ mutex_unlock(&con->con_mutex);
destroy_cm(con);
destroy_con(con);
}
@@ -2213,7 +2221,10 @@ static int init_conns(struct rtrs_clt_sess *sess)
struct rtrs_clt_con *con = to_clt_con(sess->s.con[cid]);
stop_cm(con);
+
+ mutex_lock(&con->con_mutex);
destroy_con_cq_qp(con);
+ mutex_unlock(&con->con_mutex);
destroy_cm(con);
destroy_con(con);
}
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h
index 167acd3c90fc..b8dbd701b3cb 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h
@@ -72,6 +72,7 @@ struct rtrs_clt_con {
struct rtrs_iu *rsp_ius;
u32 queue_size;
unsigned int cpu;
+ struct mutex con_mutex;
atomic_t io_cnt;
int cm_err;
};
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (2 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 03/13] RDMA/rtrs-clt: avoid run destroy_con_cq_qp/create_con_cq_qp in parallel Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-13 14:22 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu Jack Wang
` (8 subsequent siblings)
12 siblings, 1 reply; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
dev_ref of rtrs_sess is used for counting connections sharing
the rtrs_ib_dev object. But rtrs_ib_dev has its own reference counter.
We can use rtrs_ib_dev_get/put.
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 35 +++++++++++---------------
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 -
2 files changed, 14 insertions(+), 22 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 4677e8ed29ae..b1c0c1400c8a 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1523,6 +1523,19 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
struct rtrs_msg_rkey_rsp *rsp;
lockdep_assert_held(&con->con_mutex);
+ /*
+ * The whole session uses device from user connection.
+ * Be careful not to close user connection before ib dev
+ * is gracefully put.
+ */
+ sess->s.dev = rtrs_ib_dev_find_or_add(con->c.cm_id->device,
+ &dev_pd);
+ if (!sess->s.dev) {
+ rtrs_wrn(sess->clt,
+ "rtrs_ib_dev_find_get_or_add(): no memory\n");
+ return -ENOMEM;
+ }
+
if (con->c.cid == 0) {
/*
* One completion for each receive and two for each send
@@ -1531,23 +1544,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
* in case qp gets into error state
*/
wr_queue_size = SERVICE_CON_QUEUE_DEPTH * 3 + 2;
- /* We must be the first here */
- if (WARN_ON(sess->s.dev))
- return -EINVAL;
- /*
- * The whole session uses device from user connection.
- * Be careful not to close user connection before ib dev
- * is gracefully put.
- */
- sess->s.dev = rtrs_ib_dev_find_or_add(con->c.cm_id->device,
- &dev_pd);
- if (!sess->s.dev) {
- rtrs_wrn(sess->clt,
- "rtrs_ib_dev_find_get_or_add(): no memory\n");
- return -ENOMEM;
- }
- sess->s.dev_ref = 1;
query_fast_reg_mode(sess);
} else {
/*
@@ -1560,8 +1557,6 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
if (WARN_ON(!sess->queue_depth))
return -EINVAL;
- /* Shared between connections */
- sess->s.dev_ref++;
wr_queue_size =
min_t(int, sess->s.dev->ib_dev->attrs.max_qp_wr,
/* QD * (REQ + RSP + FR REGS or INVS) + drain */
@@ -1604,10 +1599,8 @@ static void destroy_con_cq_qp(struct rtrs_clt_con *con)
con->rsp_ius = NULL;
con->queue_size = 0;
}
- if (sess->s.dev_ref && !--sess->s.dev_ref) {
- rtrs_ib_dev_put(sess->s.dev);
+ if (rtrs_ib_dev_put(sess->s.dev))
sess->s.dev = NULL;
- }
}
static void stop_cm(struct rtrs_clt_con *con)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
index 0a93c87ef92b..287ff78921c7 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
@@ -103,7 +103,6 @@ struct rtrs_sess {
unsigned int con_num;
unsigned int recon_cnt;
struct rtrs_ib_dev *dev;
- int dev_ref;
struct ib_cqe *hb_cqe;
void (*hb_err_handler)(struct rtrs_con *con);
struct workqueue_struct *hb_wq;
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (3 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:25 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 06/13] RDMA/rtrs: remove unnecessary argument dir of rtrs_iu_free Jack Wang
` (7 subsequent siblings)
12 siblings, 1 reply; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
list filed is not used anywhere
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
index 287ff78921c7..7821ac4e827b 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
@@ -114,7 +114,6 @@ struct rtrs_sess {
/* rtrs information unit */
struct rtrs_iu {
- struct list_head list;
struct ib_cqe cqe;
dma_addr_t dma_addr;
void *buf;
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 06/13] RDMA/rtrs: remove unnecessary argument dir of rtrs_iu_free
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (4 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 07/13] RDMA/rtrs-clt: remove duplicated switch-case handling for CM error events Jack Wang
` (6 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
The direction of DMA operation is already in the rtrs_iu
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++--------
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 3 +--
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 14 ++++++--------
drivers/infiniband/ulp/rtrs/rtrs.c | 9 ++++-----
4 files changed, 17 insertions(+), 23 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index b1c0c1400c8a..6507cfb960d4 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1236,8 +1236,7 @@ static void free_sess_reqs(struct rtrs_clt_sess *sess)
if (req->mr)
ib_dereg_mr(req->mr);
kfree(req->sge);
- rtrs_iu_free(req->iu, DMA_TO_DEVICE,
- sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(req->iu, sess->s.dev->ib_dev, 1);
}
kfree(sess->reqs);
sess->reqs = NULL;
@@ -1594,8 +1593,7 @@ static void destroy_con_cq_qp(struct rtrs_clt_con *con)
lockdep_assert_held(&con->con_mutex);
rtrs_cq_qp_destroy(&con->c);
if (con->rsp_ius) {
- rtrs_iu_free(con->rsp_ius, DMA_FROM_DEVICE,
- sess->s.dev->ib_dev, con->queue_size);
+ rtrs_iu_free(con->rsp_ius, sess->s.dev->ib_dev, con->queue_size);
con->rsp_ius = NULL;
con->queue_size = 0;
}
@@ -2238,7 +2236,7 @@ static void rtrs_clt_info_req_done(struct ib_cq *cq, struct ib_wc *wc)
struct rtrs_iu *iu;
iu = container_of(wc->wr_cqe, struct rtrs_iu, cqe);
- rtrs_iu_free(iu, DMA_TO_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(iu, sess->s.dev->ib_dev, 1);
if (unlikely(wc->status != IB_WC_SUCCESS)) {
rtrs_err(sess->clt, "Sess info request send failed: %s\n",
@@ -2367,7 +2365,7 @@ static void rtrs_clt_info_rsp_done(struct ib_cq *cq, struct ib_wc *wc)
out:
rtrs_clt_update_wc_stats(con);
- rtrs_iu_free(iu, DMA_FROM_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(iu, sess->s.dev->ib_dev, 1);
rtrs_clt_change_state(sess, state);
}
@@ -2429,9 +2427,9 @@ static int rtrs_send_sess_info(struct rtrs_clt_sess *sess)
out:
if (tx_iu)
- rtrs_iu_free(tx_iu, DMA_TO_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(tx_iu, sess->s.dev->ib_dev, 1);
if (rx_iu)
- rtrs_iu_free(rx_iu, DMA_FROM_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(rx_iu, sess->s.dev->ib_dev, 1);
if (unlikely(err))
/* If we've never taken async path because of malloc problems */
rtrs_clt_change_state(sess, RTRS_CLT_CONNECTING_ERR);
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
index 7821ac4e827b..f49b04c69d59 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
@@ -286,8 +286,7 @@ struct rtrs_msg_rdma_hdr {
struct rtrs_iu *rtrs_iu_alloc(u32 queue_size, size_t size, gfp_t t,
struct ib_device *dev, enum dma_data_direction,
void (*done)(struct ib_cq *cq, struct ib_wc *wc));
-void rtrs_iu_free(struct rtrs_iu *iu, enum dma_data_direction dir,
- struct ib_device *dev, u32 queue_size);
+void rtrs_iu_free(struct rtrs_iu *iu, struct ib_device *dev, u32 queue_size);
int rtrs_iu_post_recv(struct rtrs_con *con, struct rtrs_iu *iu);
int rtrs_iu_post_send(struct rtrs_con *con, struct rtrs_iu *iu, size_t size,
struct ib_send_wr *head);
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
index 28f6414dfa3d..d85a55851446 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
@@ -575,8 +575,7 @@ static void unmap_cont_bufs(struct rtrs_srv_sess *sess)
struct rtrs_srv_mr *srv_mr;
srv_mr = &sess->mrs[i];
- rtrs_iu_free(srv_mr->iu, DMA_TO_DEVICE,
- sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(srv_mr->iu, sess->s.dev->ib_dev, 1);
ib_dereg_mr(srv_mr->mr);
ib_dma_unmap_sg(sess->s.dev->ib_dev, srv_mr->sgt.sgl,
srv_mr->sgt.nents, DMA_BIDIRECTIONAL);
@@ -680,8 +679,7 @@ static int map_cont_bufs(struct rtrs_srv_sess *sess)
sgt = &srv_mr->sgt;
mr = srv_mr->mr;
free_iu:
- rtrs_iu_free(srv_mr->iu, DMA_TO_DEVICE,
- sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(srv_mr->iu, sess->s.dev->ib_dev, 1);
dereg_mr:
ib_dereg_mr(mr);
unmap_sg:
@@ -733,7 +731,7 @@ static void rtrs_srv_info_rsp_done(struct ib_cq *cq, struct ib_wc *wc)
struct rtrs_iu *iu;
iu = container_of(wc->wr_cqe, struct rtrs_iu, cqe);
- rtrs_iu_free(iu, DMA_TO_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(iu, sess->s.dev->ib_dev, 1);
if (unlikely(wc->status != IB_WC_SUCCESS)) {
rtrs_err(s, "Sess info response send failed: %s\n",
@@ -859,7 +857,7 @@ static int process_info_req(struct rtrs_srv_con *con,
if (unlikely(err)) {
rtrs_err(s, "rtrs_iu_post_send(), err: %d\n", err);
iu_free:
- rtrs_iu_free(tx_iu, DMA_TO_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(tx_iu, sess->s.dev->ib_dev, 1);
}
rwr_free:
kfree(rwr);
@@ -904,7 +902,7 @@ static void rtrs_srv_info_req_done(struct ib_cq *cq, struct ib_wc *wc)
goto close;
out:
- rtrs_iu_free(iu, DMA_FROM_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(iu, sess->s.dev->ib_dev, 1);
return;
close:
close_sess(sess);
@@ -927,7 +925,7 @@ static int post_recv_info_req(struct rtrs_srv_con *con)
err = rtrs_iu_post_recv(&con->c, rx_iu);
if (unlikely(err)) {
rtrs_err(s, "rtrs_iu_post_recv(), err: %d\n", err);
- rtrs_iu_free(rx_iu, DMA_FROM_DEVICE, sess->s.dev->ib_dev, 1);
+ rtrs_iu_free(rx_iu, sess->s.dev->ib_dev, 1);
return err;
}
diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c
index ff1093d6e4bc..48f648f573b6 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs.c
@@ -31,6 +31,7 @@ struct rtrs_iu *rtrs_iu_alloc(u32 queue_size, size_t size, gfp_t gfp_mask,
return NULL;
for (i = 0; i < queue_size; i++) {
iu = &ius[i];
+ iu->direction = dir;
iu->buf = kzalloc(size, gfp_mask);
if (!iu->buf)
goto err;
@@ -41,17 +42,15 @@ struct rtrs_iu *rtrs_iu_alloc(u32 queue_size, size_t size, gfp_t gfp_mask,
iu->cqe.done = done;
iu->size = size;
- iu->direction = dir;
}
return ius;
err:
- rtrs_iu_free(ius, dir, dma_dev, i);
+ rtrs_iu_free(ius, dma_dev, i);
return NULL;
}
EXPORT_SYMBOL_GPL(rtrs_iu_alloc);
-void rtrs_iu_free(struct rtrs_iu *ius, enum dma_data_direction dir,
- struct ib_device *ibdev, u32 queue_size)
+void rtrs_iu_free(struct rtrs_iu *ius, struct ib_device *ibdev, u32 queue_size)
{
struct rtrs_iu *iu;
int i;
@@ -61,7 +60,7 @@ void rtrs_iu_free(struct rtrs_iu *ius, enum dma_data_direction dir,
for (i = 0; i < queue_size; i++) {
iu = &ius[i];
- ib_dma_unmap_single(ibdev, iu->dma_addr, iu->size, dir);
+ ib_dma_unmap_single(ibdev, iu->dma_addr, iu->size, iu->direction);
kfree(iu->buf);
}
kfree(ius);
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 07/13] RDMA/rtrs-clt: remove duplicated switch-case handling for CM error events
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (5 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 06/13] RDMA/rtrs: remove unnecessary argument dir of rtrs_iu_free Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established Jack Wang
` (5 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
The events returning the same error value are put together.
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 6507cfb960d4..7764a01185ef 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1836,20 +1836,22 @@ static int rtrs_clt_rdma_cm_handler(struct rdma_cm_id *cm_id,
case RDMA_CM_EVENT_REJECTED:
cm_err = rtrs_rdma_conn_rejected(con, ev);
break;
+ case RDMA_CM_EVENT_DISCONNECTED:
+ /* No message for disconnecting */
+ cm_err = -ECONNRESET;
+ break;
case RDMA_CM_EVENT_CONNECT_ERROR:
case RDMA_CM_EVENT_UNREACHABLE:
+ case RDMA_CM_EVENT_ADDR_CHANGE:
+ case RDMA_CM_EVENT_TIMEWAIT_EXIT:
rtrs_wrn(s, "CM error event %d\n", ev->event);
cm_err = -ECONNRESET;
break;
case RDMA_CM_EVENT_ADDR_ERROR:
case RDMA_CM_EVENT_ROUTE_ERROR:
+ rtrs_wrn(s, "CM error event %d\n", ev->event);
cm_err = -EHOSTUNREACH;
break;
- case RDMA_CM_EVENT_DISCONNECTED:
- case RDMA_CM_EVENT_ADDR_CHANGE:
- case RDMA_CM_EVENT_TIMEWAIT_EXIT:
- cm_err = -ECONNRESET;
- break;
case RDMA_CM_EVENT_DEVICE_REMOVAL:
/*
* Device removal is a special case. Queue close and return 0.
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (6 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 07/13] RDMA/rtrs-clt: remove duplicated switch-case handling for CM error events Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:23 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 09/13] RDMA/rtrs-clt: remove duplicated code Jack Wang
` (4 subsequent siblings)
12 siblings, 1 reply; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
When rtrs_rdma_conn_established returns error (non-zero value),
the error value is stored in con->cm_err and it cannot trigger
rtrs_rdma_error_recovery. Finally the error of rtrs_rdma_con_established
will be forgot.
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 7764a01185ef..f63f239bbf55 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1822,8 +1822,8 @@ static int rtrs_clt_rdma_cm_handler(struct rdma_cm_id *cm_id,
cm_err = rtrs_rdma_route_resolved(con);
break;
case RDMA_CM_EVENT_ESTABLISHED:
- con->cm_err = rtrs_rdma_conn_established(con, ev);
- if (likely(!con->cm_err)) {
+ cm_err = rtrs_rdma_conn_established(con, ev);
+ if (likely(!cm_err)) {
/*
* Report success and wake up. Here we abuse state_wq,
* i.e. wake up without state change, but we set cm_err.
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 09/13] RDMA/rtrs-clt: remove duplicated code
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (7 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 10/13] RDMA/rtrs-srv: fix typo Jack Wang
` (3 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Gioh Kim
From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
process_info_rsp checks that sg_cnt is zero twice.
Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index f63f239bbf55..102df6898339 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -2257,8 +2257,12 @@ static int process_info_rsp(struct rtrs_clt_sess *sess,
int i, sgi;
sg_cnt = le16_to_cpu(msg->sg_cnt);
- if (unlikely(!sg_cnt))
+ if (unlikely(!sg_cnt || (sess->queue_depth % sg_cnt))) {
+ rtrs_err(sess->clt, "Incorrect sg_cnt %d, is not multiple\n",
+ sg_cnt);
return -EINVAL;
+ }
+
/*
* Check if IB immediate data size is enough to hold the mem_id and
* the offset inside the memory chunk.
@@ -2271,11 +2275,6 @@ static int process_info_rsp(struct rtrs_clt_sess *sess,
MAX_IMM_PAYL_BITS, sg_cnt, sess->chunk_size);
return -EINVAL;
}
- if (unlikely(!sg_cnt || (sess->queue_depth % sg_cnt))) {
- rtrs_err(sess->clt, "Incorrect sg_cnt %d, is not multiple\n",
- sg_cnt);
- return -EINVAL;
- }
total_len = 0;
for (sgi = 0, i = 0; sgi < sg_cnt && i < sess->queue_depth; sgi++) {
const struct rtrs_sg_desc *desc = &msg->desc[sgi];
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 10/13] RDMA/rtrs-srv: fix typo
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (8 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 09/13] RDMA/rtrs-clt: remove duplicated code Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 11/13] RDMA/rtrs-srv: kill rtrs_srv_change_state_get_old Jack Wang
` (2 subsequent siblings)
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Guoqing Jiang
From: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
It should mean region here.
Signed-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-srv.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
index dc95b0932f0d..6fb39e46e248 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
@@ -62,7 +62,7 @@ struct rtrs_srv_op {
/*
* server side memory region context, when always_invalidate=Y, we need
- * queue_depth of memory regrion to invalidate each memory region.
+ * queue_depth of memory region to invalidate each memory region.
*/
struct rtrs_srv_mr {
struct ib_mr *mr;
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 11/13] RDMA/rtrs-srv: kill rtrs_srv_change_state_get_old
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (9 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 10/13] RDMA/rtrs-srv: fix typo Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 12/13] RDMA/rtrs: introduce rtrs_post_send Jack Wang
2020-10-12 13:18 ` [PATCH for-next 13/13] RDMA/rtrs-clt: remove 'addr' from rtrs_clt_add_path_to_arr Jack Wang
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Guoqing Jiang
From: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
This function isn't needed since no caller checks the old_state of sess.
Signed-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 19 +++----------------
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
index d85a55851446..52e02123697c 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
@@ -111,28 +111,18 @@ static bool __rtrs_srv_change_state(struct rtrs_srv_sess *sess,
return changed;
}
-static bool rtrs_srv_change_state_get_old(struct rtrs_srv_sess *sess,
- enum rtrs_srv_state new_state,
- enum rtrs_srv_state *old_state)
+static bool rtrs_srv_change_state(struct rtrs_srv_sess *sess,
+ enum rtrs_srv_state new_state)
{
bool changed;
spin_lock_irq(&sess->state_lock);
- *old_state = sess->state;
changed = __rtrs_srv_change_state(sess, new_state);
spin_unlock_irq(&sess->state_lock);
return changed;
}
-static bool rtrs_srv_change_state(struct rtrs_srv_sess *sess,
- enum rtrs_srv_state new_state)
-{
- enum rtrs_srv_state old_state;
-
- return rtrs_srv_change_state_get_old(sess, new_state, &old_state);
-}
-
static void free_id(struct rtrs_srv_op *id)
{
if (!id)
@@ -469,10 +459,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id,
void close_sess(struct rtrs_srv_sess *sess)
{
- enum rtrs_srv_state old_state;
-
- if (rtrs_srv_change_state_get_old(sess, RTRS_SRV_CLOSING,
- &old_state))
+ if (rtrs_srv_change_state(sess, RTRS_SRV_CLOSING))
queue_work(rtrs_wq, &sess->close_work);
WARN_ON(sess->state != RTRS_SRV_CLOSING);
}
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 12/13] RDMA/rtrs: introduce rtrs_post_send
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (10 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 11/13] RDMA/rtrs-srv: kill rtrs_srv_change_state_get_old Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
2020-10-12 13:18 ` [PATCH for-next 13/13] RDMA/rtrs-clt: remove 'addr' from rtrs_clt_add_path_to_arr Jack Wang
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Guoqing Jiang
From: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Since the three functions share the similar logic, let's introduce one
common function for it.
Signed-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs.c | 52 +++++++++++-------------------
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c
index 48f648f573b6..2e3a849e0a77 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs.c
@@ -104,6 +104,22 @@ int rtrs_post_recv_empty(struct rtrs_con *con, struct ib_cqe *cqe)
}
EXPORT_SYMBOL_GPL(rtrs_post_recv_empty);
+static int rtrs_post_send(struct ib_qp *qp, struct ib_send_wr *head,
+ struct ib_send_wr *wr)
+{
+ if (head) {
+ struct ib_send_wr *tail = head;
+
+ while (tail->next)
+ tail = tail->next;
+ tail->next = wr;
+ } else {
+ head = wr;
+ }
+
+ return ib_post_send(qp, head, NULL);
+}
+
int rtrs_iu_post_send(struct rtrs_con *con, struct rtrs_iu *iu, size_t size,
struct ib_send_wr *head)
{
@@ -126,17 +142,7 @@ int rtrs_iu_post_send(struct rtrs_con *con, struct rtrs_iu *iu, size_t size,
.send_flags = IB_SEND_SIGNALED,
};
- if (head) {
- struct ib_send_wr *tail = head;
-
- while (tail->next)
- tail = tail->next;
- tail->next = ≀
- } else {
- head = ≀
- }
-
- return ib_post_send(con->qp, head, NULL);
+ return rtrs_post_send(con->qp, head, &wr);
}
EXPORT_SYMBOL_GPL(rtrs_iu_post_send);
@@ -168,17 +174,7 @@ int rtrs_iu_post_rdma_write_imm(struct rtrs_con *con, struct rtrs_iu *iu,
if (WARN_ON(sge[i].length == 0))
return -EINVAL;
- if (head) {
- struct ib_send_wr *tail = head;
-
- while (tail->next)
- tail = tail->next;
- tail->next = &wr.wr;
- } else {
- head = &wr.wr;
- }
-
- return ib_post_send(con->qp, head, NULL);
+ return rtrs_post_send(con->qp, head, &wr.wr);
}
EXPORT_SYMBOL_GPL(rtrs_iu_post_rdma_write_imm);
@@ -195,17 +191,7 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe,
.ex.imm_data = cpu_to_be32(imm_data),
};
- if (head) {
- struct ib_send_wr *tail = head;
-
- while (tail->next)
- tail = tail->next;
- tail->next = ≀
- } else {
- head = ≀
- }
-
- return ib_post_send(con->qp, head, NULL);
+ return rtrs_post_send(con->qp, head, &wr);
}
EXPORT_SYMBOL_GPL(rtrs_post_rdma_write_imm_empty);
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH for-next 13/13] RDMA/rtrs-clt: remove 'addr' from rtrs_clt_add_path_to_arr
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
` (11 preceding siblings ...)
2020-10-12 13:18 ` [PATCH for-next 12/13] RDMA/rtrs: introduce rtrs_post_send Jack Wang
@ 2020-10-12 13:18 ` Jack Wang
12 siblings, 0 replies; 17+ messages in thread
From: Jack Wang @ 2020-10-12 13:18 UTC (permalink / raw)
To: linux-rdma
Cc: bvanassche, leon, dledford, jgg, danil.kipnis, jinpu.wang, Guoqing Jiang
From: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Remove the argument since it is not used in the function.
Signed-off-by: Guoqing Jiang <guoqing.jiang@cloud.ionos.com>
Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 102df6898339..a9f1dc3295c4 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -2154,8 +2154,7 @@ static void rtrs_clt_remove_path_from_arr(struct rtrs_clt_sess *sess)
mutex_unlock(&clt->paths_mutex);
}
-static void rtrs_clt_add_path_to_arr(struct rtrs_clt_sess *sess,
- struct rtrs_addr *addr)
+static void rtrs_clt_add_path_to_arr(struct rtrs_clt_sess *sess)
{
struct rtrs_clt *clt = sess->clt;
@@ -2930,7 +2929,7 @@ int rtrs_clt_create_path_from_sysfs(struct rtrs_clt *clt,
* IO will never grab it. Also it is very important to add
* path before init, since init fires LINK_CONNECTED event.
*/
- rtrs_clt_add_path_to_arr(sess, addr);
+ rtrs_clt_add_path_to_arr(sess);
err = init_sess(sess);
if (err)
--
2.25.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established
2020-10-12 13:18 ` [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established Jack Wang
@ 2020-10-12 13:23 ` Jinpu Wang
0 siblings, 0 replies; 17+ messages in thread
From: Jinpu Wang @ 2020-10-12 13:23 UTC (permalink / raw)
To: linux-rdma
Cc: Bart Van Assche, Leon Romanovsky, Doug Ledford, Jason Gunthorpe,
Danil Kipnis, Gioh Kim
On Mon, Oct 12, 2020 at 3:18 PM Jack Wang <jinpu.wang@cloud.ionos.com> wrote:
>
> From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
>
> When rtrs_rdma_conn_established returns error (non-zero value),
> the error value is stored in con->cm_err and it cannot trigger
> rtrs_rdma_error_recovery. Finally the error of rtrs_rdma_con_established
> will be forgot.
>
> Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
> Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
sorry, forgot to change the subject line to rtrs.
Jason, do you want a resend with the fixed subject.
> ---
> drivers/infiniband/ulp/rtrs/rtrs-clt.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> index 7764a01185ef..f63f239bbf55 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> @@ -1822,8 +1822,8 @@ static int rtrs_clt_rdma_cm_handler(struct rdma_cm_id *cm_id,
> cm_err = rtrs_rdma_route_resolved(con);
> break;
> case RDMA_CM_EVENT_ESTABLISHED:
> - con->cm_err = rtrs_rdma_conn_established(con, ev);
> - if (likely(!con->cm_err)) {
> + cm_err = rtrs_rdma_conn_established(con, ev);
> + if (likely(!cm_err)) {
> /*
> * Report success and wake up. Here we abuse state_wq,
> * i.e. wake up without state change, but we set cm_err.
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu
2020-10-12 13:18 ` [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu Jack Wang
@ 2020-10-12 13:25 ` Jinpu Wang
0 siblings, 0 replies; 17+ messages in thread
From: Jinpu Wang @ 2020-10-12 13:25 UTC (permalink / raw)
To: linux-rdma
Cc: Bart Van Assche, Leon Romanovsky, Doug Ledford, Jason Gunthorpe,
Danil Kipnis, Gioh Kim
On Mon, Oct 12, 2020 at 3:18 PM Jack Wang <jinpu.wang@cloud.ionos.com> wrote:
>
> From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
>
> list filed is not used anywhere
>
> Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
> Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
sorry, this one is already in Jason's tree, please ignore.
> ---
> drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> index 287ff78921c7..7821ac4e827b 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> @@ -114,7 +114,6 @@ struct rtrs_sess {
>
> /* rtrs information unit */
> struct rtrs_iu {
> - struct list_head list;
> struct ib_cqe cqe;
> dma_addr_t dma_addr;
> void *buf;
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess
2020-10-12 13:18 ` [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess Jack Wang
@ 2020-10-13 14:22 ` Jinpu Wang
0 siblings, 0 replies; 17+ messages in thread
From: Jinpu Wang @ 2020-10-13 14:22 UTC (permalink / raw)
To: linux-rdma
Cc: Bart Van Assche, Leon Romanovsky, Doug Ledford, Jason Gunthorpe,
Danil Kipnis, Gioh Kim
On Mon, Oct 12, 2020 at 3:18 PM Jack Wang <jinpu.wang@cloud.ionos.com> wrote:
>
> From: Gioh Kim <gi-oh.kim@cloud.ionos.com>
>
> dev_ref of rtrs_sess is used for counting connections sharing
> the rtrs_ib_dev object. But rtrs_ib_dev has its own reference counter.
> We can use rtrs_ib_dev_get/put.
>
> Signed-off-by: Gioh Kim <gi-oh.kim@cloud.ionos.com>
> Signed-off-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Please ignore this one, we found problem during tests, sorry.
> ---
> drivers/infiniband/ulp/rtrs/rtrs-clt.c | 35 +++++++++++---------------
> drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 -
> 2 files changed, 14 insertions(+), 22 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> index 4677e8ed29ae..b1c0c1400c8a 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
> @@ -1523,6 +1523,19 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
> struct rtrs_msg_rkey_rsp *rsp;
>
> lockdep_assert_held(&con->con_mutex);
> + /*
> + * The whole session uses device from user connection.
> + * Be careful not to close user connection before ib dev
> + * is gracefully put.
> + */
> + sess->s.dev = rtrs_ib_dev_find_or_add(con->c.cm_id->device,
> + &dev_pd);
> + if (!sess->s.dev) {
> + rtrs_wrn(sess->clt,
> + "rtrs_ib_dev_find_get_or_add(): no memory\n");
> + return -ENOMEM;
> + }
> +
> if (con->c.cid == 0) {
> /*
> * One completion for each receive and two for each send
> @@ -1531,23 +1544,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
> * in case qp gets into error state
> */
> wr_queue_size = SERVICE_CON_QUEUE_DEPTH * 3 + 2;
> - /* We must be the first here */
> - if (WARN_ON(sess->s.dev))
> - return -EINVAL;
>
> - /*
> - * The whole session uses device from user connection.
> - * Be careful not to close user connection before ib dev
> - * is gracefully put.
> - */
> - sess->s.dev = rtrs_ib_dev_find_or_add(con->c.cm_id->device,
> - &dev_pd);
> - if (!sess->s.dev) {
> - rtrs_wrn(sess->clt,
> - "rtrs_ib_dev_find_get_or_add(): no memory\n");
> - return -ENOMEM;
> - }
> - sess->s.dev_ref = 1;
> query_fast_reg_mode(sess);
> } else {
> /*
> @@ -1560,8 +1557,6 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
> if (WARN_ON(!sess->queue_depth))
> return -EINVAL;
>
> - /* Shared between connections */
> - sess->s.dev_ref++;
> wr_queue_size =
> min_t(int, sess->s.dev->ib_dev->attrs.max_qp_wr,
> /* QD * (REQ + RSP + FR REGS or INVS) + drain */
> @@ -1604,10 +1599,8 @@ static void destroy_con_cq_qp(struct rtrs_clt_con *con)
> con->rsp_ius = NULL;
> con->queue_size = 0;
> }
> - if (sess->s.dev_ref && !--sess->s.dev_ref) {
> - rtrs_ib_dev_put(sess->s.dev);
> + if (rtrs_ib_dev_put(sess->s.dev))
> sess->s.dev = NULL;
> - }
> }
>
> static void stop_cm(struct rtrs_clt_con *con)
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> index 0a93c87ef92b..287ff78921c7 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
> @@ -103,7 +103,6 @@ struct rtrs_sess {
> unsigned int con_num;
> unsigned int recon_cnt;
> struct rtrs_ib_dev *dev;
> - int dev_ref;
> struct ib_cqe *hb_cqe;
> void (*hb_err_handler)(struct rtrs_con *con);
> struct workqueue_struct *hb_wq;
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-10-13 14:22 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-12 13:18 [PATCH for-next 00/13] rtrs: misc fix and cleanup Jack Wang
2020-10-12 13:18 ` [PATCH for-next 01/13] RDMA/rtrs-clt: remove destroy_con_cq_qo in case route resolving failed Jack Wang
2020-10-12 13:18 ` [PATCH for-next 02/13] RDMA/rtrs-clt: remove outdated comment in create_con_cq_qp Jack Wang
2020-10-12 13:18 ` [PATCH for-next 03/13] RDMA/rtrs-clt: avoid run destroy_con_cq_qp/create_con_cq_qp in parallel Jack Wang
2020-10-12 13:18 ` [PATCH for-next 04/13] RDMA/rtrs-clt: remove unnecessary dev_ref of rtrs_sess Jack Wang
2020-10-13 14:22 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 05/13] RDMA/rtrs: removed unused filed list of rtrs_iu Jack Wang
2020-10-12 13:25 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 06/13] RDMA/rtrs: remove unnecessary argument dir of rtrs_iu_free Jack Wang
2020-10-12 13:18 ` [PATCH for-next 07/13] RDMA/rtrs-clt: remove duplicated switch-case handling for CM error events Jack Wang
2020-10-12 13:18 ` [PATCH for-next 08/13] RDMA/ibtrs-clt: missing error from rtrs_rdma_conn_established Jack Wang
2020-10-12 13:23 ` Jinpu Wang
2020-10-12 13:18 ` [PATCH for-next 09/13] RDMA/rtrs-clt: remove duplicated code Jack Wang
2020-10-12 13:18 ` [PATCH for-next 10/13] RDMA/rtrs-srv: fix typo Jack Wang
2020-10-12 13:18 ` [PATCH for-next 11/13] RDMA/rtrs-srv: kill rtrs_srv_change_state_get_old Jack Wang
2020-10-12 13:18 ` [PATCH for-next 12/13] RDMA/rtrs: introduce rtrs_post_send Jack Wang
2020-10-12 13:18 ` [PATCH for-next 13/13] RDMA/rtrs-clt: remove 'addr' from rtrs_clt_add_path_to_arr Jack Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).