linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 for-next] RDMA/hns: Support owner mode doorbell
@ 2020-09-27  9:45 Weihang Li
  2020-09-29 16:53 ` Jason Gunthorpe
  0 siblings, 1 reply; 3+ messages in thread
From: Weihang Li @ 2020-09-27  9:45 UTC (permalink / raw)
  To: dledford, jgg; +Cc: leon, linux-rdma, linuxarm

From: Lang Cheng <chenglang@huawei.com>

The doorbell needs to store PI information into QPC, so the RoCEE should
wait for the results of storing, that is, it needs two bus operations to
complete a doorbell. When ROCEE is in SDI mode, multiple doorbells may be
interlocked because the RoCEE can only handle bus operations serially. So a
flag to mark if HIP09 is working in SDI mode is added. When the SDI flag is
set, the ROCEE will ignore the PI information of the doorbell, continue to
fetch wqe and verify its validity by it's owner_bit.

Signed-off-by: Lang Cheng <chenglang@huawei.com>
Signed-off-by: Weihang Li <liweihang@huawei.com>
---
Previous version:
v1: https://patchwork.kernel.org/patch/11799327/

Changes since v1:
- Fix comments from Leon about the unused enum.

 drivers/infiniband/hw/hns/hns_roce_device.h |  4 +++-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c  | 28 ++++++++++++++++++++++------
 drivers/infiniband/hw/hns/hns_roce_qp.c     |  3 +++
 3 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index a8183ef..7c88e25 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -137,9 +137,10 @@ enum {
 	SERV_TYPE_UD,
 };
 
-enum {
+enum hns_roce_qp_caps {
 	HNS_ROCE_QP_CAP_RQ_RECORD_DB = BIT(0),
 	HNS_ROCE_QP_CAP_SQ_RECORD_DB = BIT(1),
+	HNS_ROCE_QP_CAP_OWNER_DB = BIT(2),
 };
 
 enum hns_roce_cq_flags {
@@ -229,6 +230,7 @@ enum {
 	HNS_ROCE_CAP_FLAG_FRMR                  = BIT(8),
 	HNS_ROCE_CAP_FLAG_QP_FLOW_CTRL		= BIT(9),
 	HNS_ROCE_CAP_FLAG_ATOMIC		= BIT(10),
+	HNS_ROCE_CAP_FLAG_SDI_MODE		= BIT(14),
 };
 
 #define HNS_ROCE_DB_TYPE_COUNT			2
diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index 6d30850..cc89bdb 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -474,9 +474,6 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,
 	roce_set_bit(ud_sq_wqe->byte_4, V2_UD_SEND_WQE_BYTE_4_SE_S,
 		     (wr->send_flags & IB_SEND_SOLICITED) ? 1 : 0);
 
-	roce_set_bit(ud_sq_wqe->byte_4, V2_UD_SEND_WQE_BYTE_4_OWNER_S,
-		     owner_bit);
-
 	roce_set_field(ud_sq_wqe->byte_16, V2_UD_SEND_WQE_BYTE_16_PD_M,
 		       V2_UD_SEND_WQE_BYTE_16_PD_S, to_hr_pd(qp->ibqp.pd)->pdn);
 
@@ -517,7 +514,18 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,
 
 	set_extend_sge(qp, wr, &curr_idx, valid_num_sge);
 
+	/*
+	 * The pipeline can sequentially post all valid WQEs into WQ buffer,
+	 * including new WQEs waiting for the doorbell to update the PI again.
+	 * Therefore, the owner bit of WQE MUST be updated after all fields
+	 * and extSGEs have been written into DDR instead of cache.
+	 */
+	if (qp->en_flags & HNS_ROCE_QP_CAP_OWNER_DB)
+		wmb();
+
 	*sge_idx = curr_idx;
+	roce_set_bit(ud_sq_wqe->byte_4, V2_UD_SEND_WQE_BYTE_4_OWNER_S,
+		     owner_bit);
 
 	return 0;
 }
@@ -591,9 +599,6 @@ static inline int set_rc_wqe(struct hns_roce_qp *qp,
 	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_CQE_S,
 		     (wr->send_flags & IB_SEND_SIGNALED) ? 1 : 0);
 
-	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_OWNER_S,
-		     owner_bit);
-
 	if (wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP ||
 	    wr->opcode == IB_WR_ATOMIC_FETCH_AND_ADD)
 		set_atomic_seg(wr, rc_sq_wqe, valid_num_sge);
@@ -601,7 +606,18 @@ static inline int set_rc_wqe(struct hns_roce_qp *qp,
 		ret = set_rwqe_data_seg(&qp->ibqp, wr, rc_sq_wqe,
 					&curr_idx, valid_num_sge);
 
+	/*
+	 * The pipeline can sequentially post all valid WQEs into WQ buffer,
+	 * including new WQEs waiting for the doorbell to update the PI again.
+	 * Therefore, the owner bit of WQE MUST be updated after all fields
+	 * and extSGEs have been written into DDR instead of cache.
+	 */
+	if (qp->en_flags & HNS_ROCE_QP_CAP_OWNER_DB)
+		wmb();
+
 	*sge_idx = curr_idx;
+	roce_set_bit(rc_sq_wqe->byte_4, V2_RC_SEND_WQE_BYTE_4_OWNER_S,
+		     owner_bit);
 
 	return ret;
 }
diff --git a/drivers/infiniband/hw/hns/hns_roce_qp.c b/drivers/infiniband/hw/hns/hns_roce_qp.c
index 7c3b548..5ae3c5a 100644
--- a/drivers/infiniband/hw/hns/hns_roce_qp.c
+++ b/drivers/infiniband/hw/hns/hns_roce_qp.c
@@ -727,6 +727,9 @@ static int alloc_qp_db(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp,
 	struct ib_device *ibdev = &hr_dev->ib_dev;
 	int ret;
 
+	if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_SDI_MODE)
+		hr_qp->en_flags |= HNS_ROCE_QP_CAP_OWNER_DB;
+
 	if (udata) {
 		if (user_qp_has_sdb(hr_dev, init_attr, udata, resp, ucmd)) {
 			ret = hns_roce_db_map_user(uctx, udata, ucmd->sdb_addr,
-- 
2.8.1


^ permalink raw reply related	[flat|nested] 3+ messages in thread
* Re: [PATCH v2 for-next] RDMA/hns: Support owner mode doorbell
  2020-09-27  9:45 [PATCH v2 for-next] RDMA/hns: Support owner mode doorbell Weihang Li
@ 2020-09-29 16:53 ` Jason Gunthorpe
  2020-10-12 11:56   ` liweihang
  0 siblings, 1 reply; 3+ messages in thread
From: Jason Gunthorpe @ 2020-09-29 16:53 UTC (permalink / raw)
  To: Weihang Li; +Cc: dledford, leon, linux-rdma, linuxarm

On Sun, Sep 27, 2020 at 05:45:01PM +0800, Weihang Li wrote:
> @@ -517,7 +514,18 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,
>  
>  	set_extend_sge(qp, wr, &curr_idx, valid_num_sge);
>  
> +	/*
> +	 * The pipeline can sequentially post all valid WQEs into WQ buffer,
> +	 * including new WQEs waiting for the doorbell to update the PI again.
> +	 * Therefore, the owner bit of WQE MUST be updated after all fields
> +	 * and extSGEs have been written into DDR instead of cache.
> +	 */
> +	if (qp->en_flags & HNS_ROCE_QP_CAP_OWNER_DB)
> +		wmb();

Should this be dma_wmb() ?

Jason

^ permalink raw reply	[flat|nested] 3+ messages in thread
* Re: [PATCH v2 for-next] RDMA/hns: Support owner mode doorbell
  2020-09-29 16:53 ` Jason Gunthorpe
@ 2020-10-12 11:56   ` liweihang
  0 siblings, 0 replies; 3+ messages in thread
From: liweihang @ 2020-10-12 11:56 UTC (permalink / raw)
  To: Jason Gunthorpe; +Cc: dledford, leon, linux-rdma, Linuxarm

On 2020/9/30 0:54, Jason Gunthorpe wrote:
> On Sun, Sep 27, 2020 at 05:45:01PM +0800, Weihang Li wrote:
>> @@ -517,7 +514,18 @@ static inline int set_ud_wqe(struct hns_roce_qp *qp,
>>  
>>  	set_extend_sge(qp, wr, &curr_idx, valid_num_sge);
>>  
>> +	/*
>> +	 * The pipeline can sequentially post all valid WQEs into WQ buffer,
>> +	 * including new WQEs waiting for the doorbell to update the PI again.
>> +	 * Therefore, the owner bit of WQE MUST be updated after all fields
>> +	 * and extSGEs have been written into DDR instead of cache.
>> +	 */
>> +	if (qp->en_flags & HNS_ROCE_QP_CAP_OWNER_DB)
>> +		wmb();
> 
> Should this be dma_wmb() ?
> 
> Jason
> 

Yes, dma_wmb() is better, thank you.

Weihang

^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-10-12 11:56 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-27  9:45 [PATCH v2 for-next] RDMA/hns: Support owner mode doorbell Weihang Li
2020-09-29 16:53 ` Jason Gunthorpe
2020-10-12 11:56   ` liweihang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).