* [PATCH] RDMA/nes: Adding queue drain functions
@ 2016-03-29 17:58 Tatyana Nikolova
2016-03-29 20:49 ` Steve Wise
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Tatyana Nikolova @ 2016-03-29 17:58 UTC (permalink / raw)
To: Doug Ledford
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w, leon-2ukJVAZIZ/Y
Adding sq and rq drain functions, which block until all
previously posted wr-s in the specified queue have completed.
A completion object is signaled to unblock the thread,
when the last cqe for the corresponding queue is processed.
Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
drivers/infiniband/hw/nes/nes_verbs.c | 34 ++++++++++++++++++++++++++++++++++
drivers/infiniband/hw/nes/nes_verbs.h | 2 ++
2 files changed, 36 insertions(+)
diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c
index fba69a3..170d43a 100644
--- a/drivers/infiniband/hw/nes/nes_verbs.c
+++ b/drivers/infiniband/hw/nes/nes_verbs.c
@@ -1315,6 +1315,8 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd,
nes_debug(NES_DBG_QP, "Invalid QP type: %d\n", init_attr->qp_type);
return ERR_PTR(-EINVAL);
}
+ init_completion(&nesqp->sq_drained);
+ init_completion(&nesqp->rq_drained);
nesqp->sig_all = (init_attr->sq_sig_type == IB_SIGNAL_ALL_WR);
init_timer(&nesqp->terminate_timer);
@@ -3452,6 +3454,29 @@ out:
return err;
}
+/**
+ * nes_drain_sq - drain sq
+ * @ibqp: pointer to ibqp
+ */
+static void nes_drain_sq(struct ib_qp *ibqp)
+{
+ struct nes_qp *nesqp = to_nesqp(ibqp);
+
+ if (nesqp->hwqp.sq_tail != nesqp->hwqp.sq_head)
+ wait_for_completion(&nesqp->sq_drained);
+}
+
+/**
+ * nes_drain_rq - drain rq
+ * @ibqp: pointer to ibqp
+ */
+static void nes_drain_rq(struct ib_qp *ibqp)
+{
+ struct nes_qp *nesqp = to_nesqp(ibqp);
+
+ if (nesqp->hwqp.rq_tail != nesqp->hwqp.rq_head)
+ wait_for_completion(&nesqp->rq_drained);
+}
/**
* nes_poll_cq
@@ -3581,6 +3606,13 @@ static int nes_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry)
wq_tail = nesqp->hwqp.rq_tail;
}
}
+
+ if (nesqp->iwarp_state > NES_CQP_QP_IWARP_STATE_RTS) {
+ if (nesqp->hwqp.sq_tail == nesqp->hwqp.sq_head)
+ complete(&nesqp->sq_drained);
+ if (nesqp->hwqp.rq_tail == nesqp->hwqp.rq_head)
+ complete(&nesqp->rq_drained);
+ }
entry->wr_id = wrid;
entry++;
@@ -3754,6 +3786,8 @@ struct nes_ib_device *nes_init_ofa_device(struct net_device *netdev)
nesibdev->ibdev.req_notify_cq = nes_req_notify_cq;
nesibdev->ibdev.post_send = nes_post_send;
nesibdev->ibdev.post_recv = nes_post_recv;
+ nesibdev->ibdev.drain_sq = nes_drain_sq;
+ nesibdev->ibdev.drain_rq = nes_drain_rq;
nesibdev->ibdev.iwcm = kzalloc(sizeof(*nesibdev->ibdev.iwcm), GFP_KERNEL);
if (nesibdev->ibdev.iwcm == NULL) {
diff --git a/drivers/infiniband/hw/nes/nes_verbs.h b/drivers/infiniband/hw/nes/nes_verbs.h
index 7029088..e02a566 100644
--- a/drivers/infiniband/hw/nes/nes_verbs.h
+++ b/drivers/infiniband/hw/nes/nes_verbs.h
@@ -189,6 +189,8 @@ struct nes_qp {
u8 pau_pending;
u8 pau_state;
__u64 nesuqp_addr;
+ struct completion sq_drained;
+ struct completion rq_drained;
};
struct ib_mr *nes_reg_phys_mr(struct ib_pd *ib_pd,
--
1.8.5.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 14+ messages in thread
* RE: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 17:58 [PATCH] RDMA/nes: Adding queue drain functions Tatyana Nikolova
@ 2016-03-29 20:49 ` Steve Wise
2016-03-29 21:18 ` Nikolova, Tatyana E
2016-03-31 1:19 ` Leon Romanovsky
2016-03-30 7:11 ` Leon Romanovsky
` (2 subsequent siblings)
3 siblings, 2 replies; 14+ messages in thread
From: Steve Wise @ 2016-03-29 20:49 UTC (permalink / raw)
To: 'Tatyana Nikolova', 'Doug Ledford'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w, leon-2ukJVAZIZ/Y
>
> Adding sq and rq drain functions, which block until all
> previously posted wr-s in the specified queue have completed.
> A completion object is signaled to unblock the thread,
> when the last cqe for the corresponding queue is processed.
>
> Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Looks good to me. No locking needed though?
Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 20:49 ` Steve Wise
@ 2016-03-29 21:18 ` Nikolova, Tatyana E
2016-03-31 1:19 ` Leon Romanovsky
1 sibling, 0 replies; 14+ messages in thread
From: Nikolova, Tatyana E @ 2016-03-29 21:18 UTC (permalink / raw)
To: Steve Wise, 'Doug Ledford'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Latif, Faisal, leon-2ukJVAZIZ/Y
There is locking in place in nes_poll_cq() which also protects the added drain queue functionality.
Thank you,
Tatyana
-----Original Message-----
From: Steve Wise [mailto:swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org]
Sent: Tuesday, March 29, 2016 3:49 PM
To: Nikolova, Tatyana E <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; 'Doug Ledford' <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Latif, Faisal <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>; leon-2ukJVAZIZ/Y@public.gmane.org
Subject: RE: [PATCH] RDMA/nes: Adding queue drain functions
>
> Adding sq and rq drain functions, which block until all previously
> posted wr-s in the specified queue have completed.
> A completion object is signaled to unblock the thread, when the last
> cqe for the corresponding queue is processed.
>
> Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Looks good to me. No locking needed though?
Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 20:49 ` Steve Wise
2016-03-29 21:18 ` Nikolova, Tatyana E
@ 2016-03-31 1:19 ` Leon Romanovsky
[not found] ` <20160331011902.GC2670-2ukJVAZIZ/Y@public.gmane.org>
1 sibling, 1 reply; 14+ messages in thread
From: Leon Romanovsky @ 2016-03-31 1:19 UTC (permalink / raw)
To: Steve Wise
Cc: 'Tatyana Nikolova', 'Doug Ledford',
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w
On Tue, Mar 29, 2016 at 03:49:18PM -0500, Steve Wise wrote:
> >
> > Adding sq and rq drain functions, which block until all
> > previously posted wr-s in the specified queue have completed.
> > A completion object is signaled to unblock the thread,
> > when the last cqe for the corresponding queue is processed.
> >
> > Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> Looks good to me. No locking needed though?
>
> Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Steve,
I see that this implementation follows your reference implementation of
iw_cxgb4 which was iWARP specific. Can you point me to the relevant
information which explain why this specific case exists?
Thanks.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 17:58 [PATCH] RDMA/nes: Adding queue drain functions Tatyana Nikolova
2016-03-29 20:49 ` Steve Wise
@ 2016-03-30 7:11 ` Leon Romanovsky
2016-04-03 16:17 ` sagig
2016-05-13 19:59 ` Doug Ledford
3 siblings, 0 replies; 14+ messages in thread
From: Leon Romanovsky @ 2016-03-30 7:11 UTC (permalink / raw)
To: Tatyana Nikolova
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w
On Tue, Mar 29, 2016 at 12:58:37PM -0500, Tatyana Nikolova wrote:
> Adding sq and rq drain functions, which block until all
> previously posted wr-s in the specified queue have completed.
> A completion object is signaled to unblock the thread,
> when the last cqe for the corresponding queue is processed.
>
> Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
> drivers/infiniband/hw/nes/nes_verbs.c | 34 ++++++++++++++++++++++++++++++++++
> drivers/infiniband/hw/nes/nes_verbs.h | 2 ++
> 2 files changed, 36 insertions(+)
>
> diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c
> index fba69a3..170d43a 100644
> --- a/drivers/infiniband/hw/nes/nes_verbs.c
> +++ b/drivers/infiniband/hw/nes/nes_verbs.c
> @@ -1315,6 +1315,8 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd,
> nes_debug(NES_DBG_QP, "Invalid QP type: %d\n", init_attr->qp_type);
> return ERR_PTR(-EINVAL);
> }
> + init_completion(&nesqp->sq_drained);
> + init_completion(&nesqp->rq_drained);
>
> nesqp->sig_all = (init_attr->sq_sig_type == IB_SIGNAL_ALL_WR);
> init_timer(&nesqp->terminate_timer);
> @@ -3452,6 +3454,29 @@ out:
> return err;
> }
>
> +/**
> + * nes_drain_sq - drain sq
> + * @ibqp: pointer to ibqp
> + */
> +static void nes_drain_sq(struct ib_qp *ibqp)
> +{
> + struct nes_qp *nesqp = to_nesqp(ibqp);
> +
> + if (nesqp->hwqp.sq_tail != nesqp->hwqp.sq_head)
I'm not sure that you need these checks in all places.
It won't protect from anything.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 17:58 [PATCH] RDMA/nes: Adding queue drain functions Tatyana Nikolova
2016-03-29 20:49 ` Steve Wise
2016-03-30 7:11 ` Leon Romanovsky
@ 2016-04-03 16:17 ` sagig
[not found] ` <570141FE.4090400-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-05-13 19:59 ` Doug Ledford
3 siblings, 1 reply; 14+ messages in thread
From: sagig @ 2016-04-03 16:17 UTC (permalink / raw)
To: Tatyana Nikolova, Doug Ledford
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w, leon-2ukJVAZIZ/Y
Looks fine.
Reviewed-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
P.S.
Which ULP was used to test the patch?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] RDMA/nes: Adding queue drain functions
2016-03-29 17:58 [PATCH] RDMA/nes: Adding queue drain functions Tatyana Nikolova
` (2 preceding siblings ...)
2016-04-03 16:17 ` sagig
@ 2016-05-13 19:59 ` Doug Ledford
3 siblings, 0 replies; 14+ messages in thread
From: Doug Ledford @ 2016-05-13 19:59 UTC (permalink / raw)
To: Tatyana Nikolova
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
faisal.latif-ral2JQCrhuEAvxtiuMwx3w, leon-2ukJVAZIZ/Y
[-- Attachment #1: Type: text/plain, Size: 790 bytes --]
On 03/29/2016 01:58 PM, Tatyana Nikolova wrote:
> Adding sq and rq drain functions, which block until all
> previously posted wr-s in the specified queue have completed.
> A completion object is signaled to unblock the thread,
> when the last cqe for the corresponding queue is processed.
>
> Signed-off-by: Tatyana Nikolova <tatyana.e.nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
> drivers/infiniband/hw/nes/nes_verbs.c | 34 ++++++++++++++++++++++++++++++++++
> drivers/infiniband/hw/nes/nes_verbs.h | 2 ++
> 2 files changed, 36 insertions(+)
Thanks, applied.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2016-05-13 19:59 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-03-29 17:58 [PATCH] RDMA/nes: Adding queue drain functions Tatyana Nikolova
2016-03-29 20:49 ` Steve Wise
2016-03-29 21:18 ` Nikolova, Tatyana E
2016-03-31 1:19 ` Leon Romanovsky
[not found] ` <20160331011902.GC2670-2ukJVAZIZ/Y@public.gmane.org>
2016-04-04 13:54 ` Steve Wise
2016-04-04 14:52 ` Leon Romanovsky
[not found] ` <20160404145254.GE5264-2ukJVAZIZ/Y@public.gmane.org>
2016-04-04 15:01 ` Steve Wise
2016-04-04 17:21 ` Leon Romanovsky
2016-03-30 7:11 ` Leon Romanovsky
2016-04-03 16:17 ` sagig
[not found] ` <570141FE.4090400-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-04 1:31 ` Leon Romanovsky
[not found] ` <20160404013152.GD5264-2ukJVAZIZ/Y@public.gmane.org>
2016-04-04 6:39 ` sagig
[not found] ` <57020C15.3000601-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2016-04-04 14:42 ` Steve Wise
2016-05-13 19:59 ` Doug Ledford
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.