* [PATCH for-rc 1/4] RDMA/hns: Fix port active speed
2023-08-04 1:27 [PATCH for-rc 0/4] Bugfixes for HNS RoCE Junxian Huang
@ 2023-08-04 1:27 ` Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 2/4] RDMA/hns: Fix incorrect post-send with direct wqe of wr-list Junxian Huang
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Junxian Huang @ 2023-08-04 1:27 UTC (permalink / raw)
To: jgg, leon; +Cc: linux-rdma, linuxarm, linux-kernel, huangjunxian6
From: Chengchang Tang <tangchengchang@huawei.com>
HW supports a variety of different speed, but the current speed
is fixed.
The real speed should be querried from ethernet.
Fixes: 9a4435375cd1 ("IB/hns: Add driver files for hns RoCE driver")
Signed-off-by: Chengchang Tang <tangchengchang@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
drivers/infiniband/hw/hns/hns_roce_main.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_main.c b/drivers/infiniband/hw/hns/hns_roce_main.c
index 485e110ca433..9141eadf33d2 100644
--- a/drivers/infiniband/hw/hns/hns_roce_main.c
+++ b/drivers/infiniband/hw/hns/hns_roce_main.c
@@ -219,6 +219,7 @@ static int hns_roce_query_port(struct ib_device *ib_dev, u32 port_num,
unsigned long flags;
enum ib_mtu mtu;
u32 port;
+ int ret;
port = port_num - 1;
@@ -231,8 +232,10 @@ static int hns_roce_query_port(struct ib_device *ib_dev, u32 port_num,
IB_PORT_BOOT_MGMT_SUP;
props->max_msg_sz = HNS_ROCE_MAX_MSG_LEN;
props->pkey_tbl_len = 1;
- props->active_width = IB_WIDTH_4X;
- props->active_speed = 1;
+ ret = ib_get_eth_speed(ib_dev, port_num, &props->active_speed,
+ &props->active_width);
+ if (ret)
+ ibdev_warn(ib_dev, "failed to get speed, ret = %d.\n", ret);
spin_lock_irqsave(&hr_dev->iboe.lock, flags);
--
2.30.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH for-rc 2/4] RDMA/hns: Fix incorrect post-send with direct wqe of wr-list
2023-08-04 1:27 [PATCH for-rc 0/4] Bugfixes for HNS RoCE Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 1/4] RDMA/hns: Fix port active speed Junxian Huang
@ 2023-08-04 1:27 ` Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 3/4] RDMA/hns: Fix inaccurate error label name in init instance Junxian Huang
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Junxian Huang @ 2023-08-04 1:27 UTC (permalink / raw)
To: jgg, leon; +Cc: linux-rdma, linuxarm, linux-kernel, huangjunxian6
Currently, direct wqe is not supported for wr-list. RoCE driver excludes
direct wqe for wr-list by judging whether the number of wr is 1.
For a wr-list where the second wr is a length-error atomic wr, the
post-send driver handles the first wr and adds 1 to the wr number counter
firstly. While handling the second wr, the driver finds out a length error
and terminates the wr handle process, remaining the counter at 1. This
causes the driver mistakenly judges there is only 1 wr and thus enters
the direct wqe process, carrying the current length-error atomic wqe.
This patch fixes the error by adding a judgement whether the current wr
is a bad wr. If so, use the normal doorbell process but not direct wqe
despite the wr number is 1.
Fixes: 01584a5edcc4 ("RDMA/hns: Add support of direct wqe")
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index 8f7eb11066b4..f95551b6d407 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -750,7 +750,8 @@ static int hns_roce_v2_post_send(struct ib_qp *ibqp,
qp->sq.head += nreq;
qp->next_sge = sge_idx;
- if (nreq == 1 && (qp->en_flags & HNS_ROCE_QP_CAP_DIRECT_WQE))
+ if (nreq == 1 && !ret &&
+ (qp->en_flags & HNS_ROCE_QP_CAP_DIRECT_WQE))
write_dwqe(hr_dev, qp, wqe);
else
update_sq_db(hr_dev, qp);
--
2.30.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH for-rc 3/4] RDMA/hns: Fix inaccurate error label name in init instance
2023-08-04 1:27 [PATCH for-rc 0/4] Bugfixes for HNS RoCE Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 1/4] RDMA/hns: Fix port active speed Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 2/4] RDMA/hns: Fix incorrect post-send with direct wqe of wr-list Junxian Huang
@ 2023-08-04 1:27 ` Junxian Huang
2023-08-04 1:27 ` [PATCH for-rc 4/4] RDMA/hns: Fix CQ and QP cache affinity Junxian Huang
2023-08-07 13:47 ` [PATCH for-rc 0/4] Bugfixes for HNS RoCE Leon Romanovsky
4 siblings, 0 replies; 6+ messages in thread
From: Junxian Huang @ 2023-08-04 1:27 UTC (permalink / raw)
To: jgg, leon; +Cc: linux-rdma, linuxarm, linux-kernel, huangjunxian6
This patch fixes inaccurate error label name in init instance.
Fixes: 70f92521584f ("RDMA/hns: Use the reserved loopback QPs to free MR before destroying MPT")
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index f95551b6d407..1d998298e28f 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -6723,14 +6723,14 @@ static int __hns_roce_hw_v2_init_instance(struct hnae3_handle *handle)
ret = hns_roce_init(hr_dev);
if (ret) {
dev_err(hr_dev->dev, "RoCE Engine init failed!\n");
- goto error_failed_cfg;
+ goto error_failed_roce_init;
}
if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) {
ret = free_mr_init(hr_dev);
if (ret) {
dev_err(hr_dev->dev, "failed to init free mr!\n");
- goto error_failed_roce_init;
+ goto error_failed_free_mr_init;
}
}
@@ -6738,10 +6738,10 @@ static int __hns_roce_hw_v2_init_instance(struct hnae3_handle *handle)
return 0;
-error_failed_roce_init:
+error_failed_free_mr_init:
hns_roce_exit(hr_dev);
-error_failed_cfg:
+error_failed_roce_init:
kfree(hr_dev->priv);
error_failed_kzalloc:
--
2.30.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH for-rc 4/4] RDMA/hns: Fix CQ and QP cache affinity
2023-08-04 1:27 [PATCH for-rc 0/4] Bugfixes for HNS RoCE Junxian Huang
` (2 preceding siblings ...)
2023-08-04 1:27 ` [PATCH for-rc 3/4] RDMA/hns: Fix inaccurate error label name in init instance Junxian Huang
@ 2023-08-04 1:27 ` Junxian Huang
2023-08-07 13:47 ` [PATCH for-rc 0/4] Bugfixes for HNS RoCE Leon Romanovsky
4 siblings, 0 replies; 6+ messages in thread
From: Junxian Huang @ 2023-08-04 1:27 UTC (permalink / raw)
To: jgg, leon; +Cc: linux-rdma, linuxarm, linux-kernel, huangjunxian6
From: Chengchang Tang <tangchengchang@huawei.com>
Currently, the affinity between QP cache and CQ cache is not
considered when assigning QPN, it will affect the message rate of HW.
Allocate QPN from QP cache with better CQ affinity to get better
performance.
Fixes: 71586dd20010 ("RDMA/hns: Create QP with selected QPN for bank load balance")
Signed-off-by: Chengchang Tang <tangchengchang@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
drivers/infiniband/hw/hns/hns_roce_device.h | 1 +
drivers/infiniband/hw/hns/hns_roce_qp.c | 28 ++++++++++++++++-----
2 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index 84239b907de2..bb94eb076858 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -97,6 +97,7 @@
#define HNS_ROCE_CQ_BANK_NUM 4
#define CQ_BANKID_SHIFT 2
+#define CQ_BANKID_MASK GENMASK(1, 0)
enum {
SERV_TYPE_RC,
diff --git a/drivers/infiniband/hw/hns/hns_roce_qp.c b/drivers/infiniband/hw/hns/hns_roce_qp.c
index d855a917f4cf..cdc1c6de43a1 100644
--- a/drivers/infiniband/hw/hns/hns_roce_qp.c
+++ b/drivers/infiniband/hw/hns/hns_roce_qp.c
@@ -170,14 +170,29 @@ static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp,
}
}
-static u8 get_least_load_bankid_for_qp(struct hns_roce_bank *bank)
+static u8 get_affinity_cq_bank(u8 qp_bank)
{
- u32 least_load = bank[0].inuse;
+ return (qp_bank >> 1) & CQ_BANKID_MASK;
+}
+
+static u8 get_least_load_bankid_for_qp(struct ib_qp_init_attr *init_attr,
+ struct hns_roce_bank *bank)
+{
+#define INVALID_LOAD_QPNUM 0xFFFFFFFF
+ struct ib_cq *scq = init_attr->send_cq;
+ u32 least_load = INVALID_LOAD_QPNUM;
+ unsigned long cqn = 0;
u8 bankid = 0;
u32 bankcnt;
u8 i;
- for (i = 1; i < HNS_ROCE_QP_BANK_NUM; i++) {
+ if (scq)
+ cqn = to_hr_cq(scq)->cqn;
+
+ for (i = 0; i < HNS_ROCE_QP_BANK_NUM; i++) {
+ if (scq && (get_affinity_cq_bank(i) != (cqn & CQ_BANKID_MASK)))
+ continue;
+
bankcnt = bank[i].inuse;
if (bankcnt < least_load) {
least_load = bankcnt;
@@ -209,7 +224,8 @@ static int alloc_qpn_with_bankid(struct hns_roce_bank *bank, u8 bankid,
return 0;
}
-static int alloc_qpn(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp)
+static int alloc_qpn(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp,
+ struct ib_qp_init_attr *init_attr)
{
struct hns_roce_qp_table *qp_table = &hr_dev->qp_table;
unsigned long num = 0;
@@ -220,7 +236,7 @@ static int alloc_qpn(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp)
num = 1;
} else {
mutex_lock(&qp_table->bank_mutex);
- bankid = get_least_load_bankid_for_qp(qp_table->bank);
+ bankid = get_least_load_bankid_for_qp(init_attr, qp_table->bank);
ret = alloc_qpn_with_bankid(&qp_table->bank[bankid], bankid,
&num);
@@ -1082,7 +1098,7 @@ static int hns_roce_create_qp_common(struct hns_roce_dev *hr_dev,
goto err_buf;
}
- ret = alloc_qpn(hr_dev, hr_qp);
+ ret = alloc_qpn(hr_dev, hr_qp, init_attr);
if (ret) {
ibdev_err(ibdev, "failed to alloc QPN, ret = %d.\n", ret);
goto err_qpn;
--
2.30.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH for-rc 0/4] Bugfixes for HNS RoCE
2023-08-04 1:27 [PATCH for-rc 0/4] Bugfixes for HNS RoCE Junxian Huang
` (3 preceding siblings ...)
2023-08-04 1:27 ` [PATCH for-rc 4/4] RDMA/hns: Fix CQ and QP cache affinity Junxian Huang
@ 2023-08-07 13:47 ` Leon Romanovsky
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-08-07 13:47 UTC (permalink / raw)
To: Jason Gunthorpe, Junxian Huang; +Cc: linux-rdma, linuxarm, linux-kernel
On Fri, 04 Aug 2023 09:27:07 +0800, Junxian Huang wrote:
> 1. #1: The first patch fixes port speed error by getting the speed
> from ethernet instead of using a fixed speed.
>
> 2. #2: The second patch fixes an error of using direct wqe for wr-list.
>
> 3. #3: The third patch modifies some inaccurate error label names.
>
> [...]
Applied, thanks!
[1/4] RDMA/hns: Fix port active speed
https://git.kernel.org/rdma/rdma/c/df1bcf90a66a10
[2/4] RDMA/hns: Fix incorrect post-send with direct wqe of wr-list
https://git.kernel.org/rdma/rdma/c/706efac4477cdb
[3/4] RDMA/hns: Fix inaccurate error label name in init instance
https://git.kernel.org/rdma/rdma/c/c9c0bd3c177d93
[4/4] RDMA/hns: Fix CQ and QP cache affinity
https://git.kernel.org/rdma/rdma/c/9e03dbea2b0634
Best regards,
--
Leon Romanovsky <leon@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread