All of lore.kernel.org
 help / color / mirror / Atom feed
From: Bob Pearson <rpearsonhpe@gmail.com>
To: yanjun.zhu@linux.dev, jgg@ziepe.ca, leon@kernel.org,
	linux-rdma@vger.kernel.org, jhack@hpe.com
Cc: Bob Pearson <rpearsonhpe@gmail.com>
Subject: [PATCH for-next v2 08/12] RDMA/rxe: Don't call direct between tasks
Date: Wed, 27 Mar 2024 10:51:54 -0500	[thread overview]
Message-ID: <20240327155157.590886-10-rpearsonhpe@gmail.com> (raw)
In-Reply-To: <20240327155157.590886-2-rpearsonhpe@gmail.com>

Replace calls to rxe_run_task() with rxe_sched_task().
This prevents the tasks from all running on the same cpu.

This change slightly reduces performance for single qp send and write
benchmarks in loopback mode but greatly improves the performance
with multiple qps because if run task is used all the work tends
to be performed on one cpu. For actual on the wire benchmarks there
is no noticeable performance change.

Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>
---
 drivers/infiniband/sw/rxe/rxe_comp.c  | 13 ++-----------
 drivers/infiniband/sw/rxe/rxe_resp.c  | 12 +-----------
 drivers/infiniband/sw/rxe/rxe_verbs.c |  2 +-
 3 files changed, 4 insertions(+), 23 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index c41743fbd5f1..26c06f840184 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -131,18 +131,9 @@ void retransmit_timer(struct timer_list *t)
 
 void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb)
 {
-	int must_sched;
-
-	must_sched = skb_queue_len(&qp->resp_pkts) > 0;
-	if (must_sched != 0)
-		rxe_counter_inc(SKB_TO_PKT(skb)->rxe, RXE_CNT_SENDER_SCHED);
-
+	rxe_counter_inc(SKB_TO_PKT(skb)->rxe, RXE_CNT_SENDER_SCHED);
 	skb_queue_tail(&qp->resp_pkts, skb);
-
-	if (must_sched)
-		rxe_sched_task(&qp->send_task);
-	else
-		rxe_run_task(&qp->send_task);
+	rxe_sched_task(&qp->send_task);
 }
 
 static inline enum comp_state get_wqe(struct rxe_qp *qp,
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index 3ce7a32b5dcf..c6a7fa3054fa 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -49,18 +49,8 @@ static char *resp_state_name[] = {
 /* rxe_recv calls here to add a request packet to the input queue */
 void rxe_resp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb)
 {
-	int must_sched;
-	struct rxe_pkt_info *pkt = SKB_TO_PKT(skb);
-
 	skb_queue_tail(&qp->req_pkts, skb);
-
-	must_sched = (pkt->opcode == IB_OPCODE_RC_RDMA_READ_REQUEST) ||
-			(skb_queue_len(&qp->req_pkts) > 1);
-
-	if (must_sched)
-		rxe_sched_task(&qp->recv_task);
-	else
-		rxe_run_task(&qp->recv_task);
+	rxe_sched_task(&qp->recv_task);
 }
 
 static inline enum resp_states get_req(struct rxe_qp *qp,
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index d07f7bd3b2ae..c7d4d8ab5a09 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -935,7 +935,7 @@ static int rxe_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
 
 	if (qp->is_user) {
 		/* Utilize process context to do protocol processing */
-		rxe_run_task(&qp->send_task);
+		rxe_sched_task(&qp->send_task);
 	} else {
 		err = rxe_post_send_kernel(qp, wr, bad_wr);
 		if (err)
-- 
2.43.0


  parent reply	other threads:[~2024-03-27 15:57 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-03-27 15:51 [PATCH for-next v2 00/12] RDMA/rxe: Various fixes and cleanups Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 01/12] RDMA/rxe: Fix seg fault in rxe_comp_queue_pkt Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 02/12] RDMA/rxe: Allow good work requests to be executed Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 03/12] RDMA/rxe: Remove redundant scheduling of rxe_completer Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 04/12] RDMA/rxe: Merge request and complete tasks Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 05/12] RDMA/rxe: Remove save/rollback_state in rxe_requester Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 06/12] RDMA/rxe: Don't schedule rxe_completer from rxe_requester Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 07/12] RDMA/rxe: Don't call rxe_requester from rxe_completer Bob Pearson
2024-03-27 15:51 ` Bob Pearson [this message]
2024-03-27 15:51 ` [PATCH for-next v2 09/12] RDMA/rxe: Fix incorrect rxe_put in error path Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 10/12] RDMA/rxe: Make rxe_loopback match rxe_send behavior Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 11/12] RDMA/rxe: Get rid of pkt resend on err Bob Pearson
2024-03-27 15:51 ` [PATCH for-next v2 12/12] RDMA/rxe: Let destroy qp succeed with stuck packet Bob Pearson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240327155157.590886-10-rpearsonhpe@gmail.com \
    --to=rpearsonhpe@gmail.com \
    --cc=jgg@ziepe.ca \
    --cc=jhack@hpe.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=yanjun.zhu@linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.