* [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
@ 2022-01-20 14:34 Bernard Metzler
2022-01-28 17:00 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Bernard Metzler @ 2022-01-20 14:34 UTC (permalink / raw)
To: linux-rdma; +Cc: Bernard Metzler, jgg, leon, jared.holzman
Code unconditionally resumed fenced SQ processing after next RDMA
Read completion, even if other RDMA Read responses are still
outstanding, or ORQ is full. Also adds comments for better readability
of fence processing, and removes orq_get_tail() helper, which is not
needed anymore.
Reported-by: Jared Holzman <jared.holzman@excelero.com>
Signed-off-by: Bernard Metzler <bmt@zurich.ibm.com>
---
drivers/infiniband/sw/siw/siw.h | 7 +------
drivers/infiniband/sw/siw/siw_qp_rx.c | 20 +++++++++++---------
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index 368959ae9a8c..df03d84c6868 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -644,14 +644,9 @@ static inline struct siw_sqe *orq_get_current(struct siw_qp *qp)
return &qp->orq[qp->orq_get % qp->attrs.orq_size];
}
-static inline struct siw_sqe *orq_get_tail(struct siw_qp *qp)
-{
- return &qp->orq[qp->orq_put % qp->attrs.orq_size];
-}
-
static inline struct siw_sqe *orq_get_free(struct siw_qp *qp)
{
- struct siw_sqe *orq_e = orq_get_tail(qp);
+ struct siw_sqe *orq_e = &qp->orq[qp->orq_put % qp->attrs.orq_size];
if (READ_ONCE(orq_e->flags) == 0)
return orq_e;
diff --git a/drivers/infiniband/sw/siw/siw_qp_rx.c b/drivers/infiniband/sw/siw/siw_qp_rx.c
index 60116f20653c..875ea6f1b04a 100644
--- a/drivers/infiniband/sw/siw/siw_qp_rx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_rx.c
@@ -1153,11 +1153,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
spin_lock_irqsave(&qp->orq_lock, flags);
- rreq = orq_get_current(qp);
-
/* free current orq entry */
+ rreq = orq_get_current(qp);
WRITE_ONCE(rreq->flags, 0);
+ qp->orq_get++;
+
if (qp->tx_ctx.orq_fence) {
if (unlikely(tx_waiting->wr_status != SIW_WR_QUEUED)) {
pr_warn("siw: [QP %u]: fence resume: bad status %d\n",
@@ -1165,10 +1166,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
rv = -EPROTO;
goto out;
}
- /* resume SQ processing */
+ /* resume SQ processing, if possible */
if (tx_waiting->sqe.opcode == SIW_OP_READ ||
tx_waiting->sqe.opcode == SIW_OP_READ_LOCAL_INV) {
- rreq = orq_get_tail(qp);
+
+ /* SQ processing was stopped because of a full ORQ */
+ rreq = orq_get_free(qp);
if (unlikely(!rreq)) {
pr_warn("siw: [QP %u]: no ORQE\n", qp_id(qp));
rv = -EPROTO;
@@ -1181,15 +1184,14 @@ static int siw_check_tx_fence(struct siw_qp *qp)
resume_tx = 1;
} else if (siw_orq_empty(qp)) {
+ /*
+ * SQ processing was stopped by fenced work request.
+ * Resume since all previous Read's are now completed.
+ */
qp->tx_ctx.orq_fence = 0;
resume_tx = 1;
- } else {
- pr_warn("siw: [QP %u]: fence resume: orq idx: %d:%d\n",
- qp_id(qp), qp->orq_get, qp->orq_put);
- rv = -EPROTO;
}
}
- qp->orq_get++;
out:
spin_unlock_irqrestore(&qp->orq_lock, flags);
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
2022-01-20 14:34 [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic Bernard Metzler
@ 2022-01-28 17:00 ` Jason Gunthorpe
0 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2022-01-28 17:00 UTC (permalink / raw)
To: Bernard Metzler; +Cc: linux-rdma, leon, jared.holzman
On Thu, Jan 20, 2022 at 03:34:34PM +0100, Bernard Metzler wrote:
> Code unconditionally resumed fenced SQ processing after next RDMA
> Read completion, even if other RDMA Read responses are still
> outstanding, or ORQ is full. Also adds comments for better readability
> of fence processing, and removes orq_get_tail() helper, which is not
> needed anymore.
>
> Reported-by: Jared Holzman <jared.holzman@excelero.com>
> Signed-off-by: Bernard Metzler <bmt@zurich.ibm.com>
> ---
> drivers/infiniband/sw/siw/siw.h | 7 +------
> drivers/infiniband/sw/siw/siw_qp_rx.c | 20 +++++++++++---------
> 2 files changed, 12 insertions(+), 15 deletions(-)
Fixes line??
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
2022-01-30 17:08 Bernard Metzler
@ 2022-02-01 14:18 ` Jason Gunthorpe
0 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2022-02-01 14:18 UTC (permalink / raw)
To: Bernard Metzler; +Cc: linux-rdma, leon, jared.holzman
On Sun, Jan 30, 2022 at 06:08:15PM +0100, Bernard Metzler wrote:
> Code unconditionally resumed fenced SQ processing after next RDMA
> Read completion, even if other RDMA Read responses are still
> outstanding, or ORQ is full. Also adds comments for better readability
> of fence processing, and removes orq_get_tail() helper, which is not
> needed anymore.
>
> Fixes: 8b6a361b8c48 ("rdma/siw: receive path")
> Fixes: a531975279f3 ("rdma/siw: main include file")
> Reported-by: Jared Holzman <jared.holzman@excelero.com>
> Signed-off-by: Bernard Metzler <bmt@zurich.ibm.com>
> ---
> drivers/infiniband/sw/siw/siw.h | 7 +------
> drivers/infiniband/sw/siw/siw_qp_rx.c | 20 +++++++++++---------
> 2 files changed, 12 insertions(+), 15 deletions(-)
Applied to for-rc, thanks
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
@ 2022-01-30 17:08 Bernard Metzler
2022-02-01 14:18 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Bernard Metzler @ 2022-01-30 17:08 UTC (permalink / raw)
To: linux-rdma; +Cc: Bernard Metzler, jgg, leon, jared.holzman
Code unconditionally resumed fenced SQ processing after next RDMA
Read completion, even if other RDMA Read responses are still
outstanding, or ORQ is full. Also adds comments for better readability
of fence processing, and removes orq_get_tail() helper, which is not
needed anymore.
Fixes: 8b6a361b8c48 ("rdma/siw: receive path")
Fixes: a531975279f3 ("rdma/siw: main include file")
Reported-by: Jared Holzman <jared.holzman@excelero.com>
Signed-off-by: Bernard Metzler <bmt@zurich.ibm.com>
---
drivers/infiniband/sw/siw/siw.h | 7 +------
drivers/infiniband/sw/siw/siw_qp_rx.c | 20 +++++++++++---------
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index 368959ae9a8c..df03d84c6868 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -644,14 +644,9 @@ static inline struct siw_sqe *orq_get_current(struct siw_qp *qp)
return &qp->orq[qp->orq_get % qp->attrs.orq_size];
}
-static inline struct siw_sqe *orq_get_tail(struct siw_qp *qp)
-{
- return &qp->orq[qp->orq_put % qp->attrs.orq_size];
-}
-
static inline struct siw_sqe *orq_get_free(struct siw_qp *qp)
{
- struct siw_sqe *orq_e = orq_get_tail(qp);
+ struct siw_sqe *orq_e = &qp->orq[qp->orq_put % qp->attrs.orq_size];
if (READ_ONCE(orq_e->flags) == 0)
return orq_e;
diff --git a/drivers/infiniband/sw/siw/siw_qp_rx.c b/drivers/infiniband/sw/siw/siw_qp_rx.c
index 60116f20653c..875ea6f1b04a 100644
--- a/drivers/infiniband/sw/siw/siw_qp_rx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_rx.c
@@ -1153,11 +1153,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
spin_lock_irqsave(&qp->orq_lock, flags);
- rreq = orq_get_current(qp);
-
/* free current orq entry */
+ rreq = orq_get_current(qp);
WRITE_ONCE(rreq->flags, 0);
+ qp->orq_get++;
+
if (qp->tx_ctx.orq_fence) {
if (unlikely(tx_waiting->wr_status != SIW_WR_QUEUED)) {
pr_warn("siw: [QP %u]: fence resume: bad status %d\n",
@@ -1165,10 +1166,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
rv = -EPROTO;
goto out;
}
- /* resume SQ processing */
+ /* resume SQ processing, if possible */
if (tx_waiting->sqe.opcode == SIW_OP_READ ||
tx_waiting->sqe.opcode == SIW_OP_READ_LOCAL_INV) {
- rreq = orq_get_tail(qp);
+
+ /* SQ processing was stopped because of a full ORQ */
+ rreq = orq_get_free(qp);
if (unlikely(!rreq)) {
pr_warn("siw: [QP %u]: no ORQE\n", qp_id(qp));
rv = -EPROTO;
@@ -1181,15 +1184,14 @@ static int siw_check_tx_fence(struct siw_qp *qp)
resume_tx = 1;
} else if (siw_orq_empty(qp)) {
+ /*
+ * SQ processing was stopped by fenced work request.
+ * Resume since all previous Read's are now completed.
+ */
qp->tx_ctx.orq_fence = 0;
resume_tx = 1;
- } else {
- pr_warn("siw: [QP %u]: fence resume: orq idx: %d:%d\n",
- qp_id(qp), qp->orq_get, qp->orq_put);
- rv = -EPROTO;
}
}
- qp->orq_get++;
out:
spin_unlock_irqrestore(&qp->orq_lock, flags);
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* RE: [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
2022-01-20 14:08 Bernard Metzler
@ 2022-01-20 14:34 ` Bernard Metzler
0 siblings, 0 replies; 6+ messages in thread
From: Bernard Metzler @ 2022-01-20 14:34 UTC (permalink / raw)
To: Bernard Metzler, linux-rdma; +Cc: jgg, leon, jared.holzman
Please ignore the patch I just sent, it contains a stray '/'
within comments. I'll resend with fixed line. Sorry for the noise.
Bernard.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic.
@ 2022-01-20 14:08 Bernard Metzler
2022-01-20 14:34 ` Bernard Metzler
0 siblings, 1 reply; 6+ messages in thread
From: Bernard Metzler @ 2022-01-20 14:08 UTC (permalink / raw)
To: linux-rdma; +Cc: Bernard Metzler, jgg, leon, jared.holzman
Code unconditionally resumed fenced SQ processing after next RDMA
Read completion, even if other RDMA Read responses are still
outstanding, or ORQ is full. Also adds comments for better readability
of fence processing, and removes orq_get_tail() helper, which is not
needed anymore.
Reported-by: Jared Holzman <jared.holzman@excelero.com>
Signed-off-by: Bernard Metzler <bmt@zurich.ibm.com>
---
drivers/infiniband/sw/siw/siw.h | 7 +------
drivers/infiniband/sw/siw/siw_qp_rx.c | 20 +++++++++++---------
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index 368959ae9a8c..df03d84c6868 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -644,14 +644,9 @@ static inline struct siw_sqe *orq_get_current(struct siw_qp *qp)
return &qp->orq[qp->orq_get % qp->attrs.orq_size];
}
-static inline struct siw_sqe *orq_get_tail(struct siw_qp *qp)
-{
- return &qp->orq[qp->orq_put % qp->attrs.orq_size];
-}
-
static inline struct siw_sqe *orq_get_free(struct siw_qp *qp)
{
- struct siw_sqe *orq_e = orq_get_tail(qp);
+ struct siw_sqe *orq_e = &qp->orq[qp->orq_put % qp->attrs.orq_size];
if (READ_ONCE(orq_e->flags) == 0)
return orq_e;
diff --git a/drivers/infiniband/sw/siw/siw_qp_rx.c b/drivers/infiniband/sw/siw/siw_qp_rx.c
index 60116f20653c..a699579a958b 100644
--- a/drivers/infiniband/sw/siw/siw_qp_rx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_rx.c
@@ -1153,11 +1153,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
spin_lock_irqsave(&qp->orq_lock, flags);
- rreq = orq_get_current(qp);
-
/* free current orq entry */
+ rreq = orq_get_current(qp);
WRITE_ONCE(rreq->flags, 0);
+ qp->orq_get++;
+
if (qp->tx_ctx.orq_fence) {
if (unlikely(tx_waiting->wr_status != SIW_WR_QUEUED)) {
pr_warn("siw: [QP %u]: fence resume: bad status %d\n",
@@ -1165,10 +1166,12 @@ static int siw_check_tx_fence(struct siw_qp *qp)
rv = -EPROTO;
goto out;
}
- /* resume SQ processing */
+ /* resume SQ processing, if possible */
if (tx_waiting->sqe.opcode == SIW_OP_READ ||
tx_waiting->sqe.opcode == SIW_OP_READ_LOCAL_INV) {
- rreq = orq_get_tail(qp);
+
+ /* SQ processing was stopped because of a full ORQ */
+ rreq = orq_get_free(qp);
if (unlikely(!rreq)) {
pr_warn("siw: [QP %u]: no ORQE\n", qp_id(qp));
rv = -EPROTO;
@@ -1181,15 +1184,14 @@ static int siw_check_tx_fence(struct siw_qp *qp)
resume_tx = 1;
} else if (siw_orq_empty(qp)) {
+ /*
+ * SQ processing was stopped by fenced work request.
+ /* Resume since all previous Read's are now completed.
+ */
qp->tx_ctx.orq_fence = 0;
resume_tx = 1;
- } else {
- pr_warn("siw: [QP %u]: fence resume: orq idx: %d:%d\n",
- qp_id(qp), qp->orq_get, qp->orq_put);
- rv = -EPROTO;
}
}
- qp->orq_get++;
out:
spin_unlock_irqrestore(&qp->orq_lock, flags);
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-02-01 14:19 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-20 14:34 [PATCH] RDMA/siw: Fix broken RDMA Read Fence/Resume logic Bernard Metzler
2022-01-28 17:00 ` Jason Gunthorpe
-- strict thread matches above, loose matches on Subject: below --
2022-01-30 17:08 Bernard Metzler
2022-02-01 14:18 ` Jason Gunthorpe
2022-01-20 14:08 Bernard Metzler
2022-01-20 14:34 ` Bernard Metzler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.