From: Jack Wang <jinpu.wang@ionos.com>
To: linux-rdma@vger.kernel.org
Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com,
jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com,
Aleksei Marov <aleksei.marov@ionos.com>,
Gioh Kim <gi-oh.kim@ionos.com>
Subject: [PATCHv2 for-rc 6/6] RDMA/rtrs: Move sq_wr_avail to rtrs_con
Date: Mon, 12 Jul 2021 08:07:50 +0200 [thread overview]
Message-ID: <20210712060750.16494-7-jinpu.wang@ionos.com> (raw)
In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com>
In order to account HB for sq_wr_avail properly,
move sq_wr_avail from rtrs_srv_con to rtrs_con.
Although rtrs-clt do not care sq_wr_avail, but still init it
to max_send_wr.
Fixes: b38041d50add ("RDMA/rtrs: Do not signal for heatbeat")
Signed-off-by: Jack Wang <jinpu.wang@ionos.com>
Reviewed-by: Aleksei Marov <aleksei.marov@ionos.com>
Reviewed-by: Gioh Kim <gi-oh.kim@ionos.com>
Reviewed-by: Md Haris Iqbal <haris.iqbal@ionos.com>
---
drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 +
drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 +
drivers/infiniband/ulp/rtrs/rtrs-srv.c | 8 ++++----
drivers/infiniband/ulp/rtrs/rtrs-srv.h | 1 -
drivers/infiniband/ulp/rtrs/rtrs.c | 1 +
5 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index f023676e05e4..ece3205531b8 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1680,6 +1680,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con)
sess->queue_depth * 3 + 1);
max_send_sge = 2;
}
+ atomic_set(&con->c.sq_wr_avail, max_send_wr);
cq_num = max_send_wr + max_recv_wr;
/* alloc iu to recv new rkey reply when server reports flags set */
if (sess->flags & RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) {
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
index 76581ebaed1d..d12ddfa50747 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h
@@ -97,6 +97,7 @@ struct rtrs_con {
unsigned int cid;
int nr_cqe;
atomic_t wr_cnt;
+ atomic_t sq_wr_avail;
};
struct rtrs_sess {
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
index 44ed15f38896..cd9a4ccf4c28 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
@@ -507,11 +507,11 @@ bool rtrs_srv_resp_rdma(struct rtrs_srv_op *id, int status)
ib_update_fast_reg_key(mr->mr, ib_inc_rkey(mr->mr->rkey));
}
if (unlikely(atomic_sub_return(1,
- &con->sq_wr_avail) < 0)) {
+ &con->c.sq_wr_avail) < 0)) {
rtrs_err(s, "IB send queue full: sess=%s cid=%d\n",
kobject_name(&sess->kobj),
con->c.cid);
- atomic_add(1, &con->sq_wr_avail);
+ atomic_add(1, &con->c.sq_wr_avail);
spin_lock(&con->rsp_wr_wait_lock);
list_add_tail(&id->wait_list, &con->rsp_wr_wait_list);
spin_unlock(&con->rsp_wr_wait_lock);
@@ -1268,7 +1268,7 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc)
* post_send() RDMA write completions of IO reqs (read/write)
* and hb.
*/
- atomic_add(s->signal_interval, &con->sq_wr_avail);
+ atomic_add(s->signal_interval, &con->c.sq_wr_avail);
if (unlikely(!list_empty_careful(&con->rsp_wr_wait_list)))
rtrs_rdma_process_wr_wait_list(con);
@@ -1680,7 +1680,7 @@ static int create_con(struct rtrs_srv_sess *sess,
*/
}
cq_num = max_send_wr + max_recv_wr;
- atomic_set(&con->sq_wr_avail, max_send_wr);
+ atomic_set(&con->c.sq_wr_avail, max_send_wr);
cq_vector = rtrs_srv_get_next_cq_vector(sess);
/* TODO: SOFTIRQ can be faster, but be careful with softirq context */
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
index 6785c3b6363e..e81774f5acd3 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
@@ -42,7 +42,6 @@ struct rtrs_srv_stats {
struct rtrs_srv_con {
struct rtrs_con c;
- atomic_t sq_wr_avail;
struct list_head rsp_wr_wait_list;
spinlock_t rsp_wr_wait_lock;
};
diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c
index b56dc5b82db0..ca542e477d38 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs.c
@@ -191,6 +191,7 @@ static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con,
struct rtrs_sess *sess = con->sess;
enum ib_send_flags sflags;
+ atomic_dec_if_positive(&con->sq_wr_avail);
sflags = (atomic_inc_return(&con->wr_cnt) % sess->signal_interval) ?
0 : IB_SEND_SIGNALED;
--
2.25.1
next prev parent reply other threads:[~2021-07-12 6:07 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-12 6:07 [PATCHv2 for-rc 0/6] Bugfixes for send queue overflow by heartbeat Jack Wang
2021-07-12 6:07 ` [PATCHv2 for-rc 1/6] RDMA/rtrs: Add error messages for failed operations Jack Wang
2021-07-12 6:07 ` [PATCHv2 for-rc 2/6] RDMA/rtrs: move wr_cnt from rtrs_srv_con to rtrs_con Jack Wang
2021-07-12 6:07 ` [PATCHv2 for-rc 3/6] RDMA/rtrs: Enable the same selective signal for heartbeat and IO Jack Wang
2021-07-12 6:07 ` [PATCHv2 for-rc 4/6] RDMA/rtrs: Make rtrs_post_rdma_write_imm_empty static Jack Wang
2021-07-12 6:07 ` [PATCHv2 for-rc 5/6] RDMA/rtrs: Remove unused flags parameter Jack Wang
2021-07-12 6:07 ` Jack Wang [this message]
2021-07-12 17:35 ` [PATCHv2 for-rc 0/6] Bugfixes for send queue overflow by heartbeat Jason Gunthorpe
2021-07-12 18:58 ` Jinpu Wang
2021-07-15 17:41 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210712060750.16494-7-jinpu.wang@ionos.com \
--to=jinpu.wang@ionos.com \
--cc=aleksei.marov@ionos.com \
--cc=bvanassche@acm.org \
--cc=dledford@redhat.com \
--cc=gi-oh.kim@ionos.com \
--cc=haris.iqbal@ionos.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.