* [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys
@ 2023-01-17 13:14 Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Eric Dumazet, Jakub Kicinski, linux-kernel, linux-rdma,
Michael Guralnik, netdev, Or Har-Toov, Paolo Abeni,
Saeed Mahameed
Changelog:
v2:
* Took a liberty and rewrote net/mlx5 patches
* change logic around terminate_scatter_list_mkey
* Added capability checks to be before executing command
v1: https://lore.kernel.org/all/cover.1672917578.git.leonro@nvidia.com
* Use already stored mkeys.terminate_scatter_list_mkey.
v0: https://lore.kernel.org/all/cover.1672819469.git.leonro@nvidia.com
-----------------------------------------------------------------------
This series from Or extends mlx5 driver to rely on firmware to get
special mkey values.
Thanks
Or Har-Toov (4):
net/mlx5: Expose bits for querying special mkeys
net/mlx5: Change define name for 0x100 lkey value
net/mlx5e: Use query_special_contexts for mkeys
RDMA/mlx5: Use query_special_contexts for mkeys
drivers/infiniband/hw/mlx5/cmd.c | 45 +++++++++++--------
drivers/infiniband/hw/mlx5/cmd.h | 3 +-
drivers/infiniband/hw/mlx5/main.c | 10 ++---
drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +++-
drivers/infiniband/hw/mlx5/odp.c | 27 ++++-------
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 22 ++++++++-
include/linux/mlx5/mlx5_ifc.h | 10 ++++-
include/linux/mlx5/qp.h | 2 +-
10 files changed, 81 insertions(+), 51 deletions(-)
--
2.39.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying special mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Add needed HW bits to query the values of all special mkeys.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
include/linux/mlx5/mlx5_ifc.h | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 2d17b6a6d82d..8bbf15433bb2 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1480,7 +1480,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
u8 relaxed_ordering_write[0x1];
u8 relaxed_ordering_read[0x1];
u8 log_max_mkey[0x6];
- u8 reserved_at_f0[0x8];
+ u8 reserved_at_f0[0x6];
+ u8 terminate_scatter_list_mkey[0x1];
+ u8 repeated_mkey[0x1];
u8 dump_fill_mkey[0x1];
u8 reserved_at_f9[0x2];
u8 fast_teardown[0x1];
@@ -5202,7 +5204,11 @@ struct mlx5_ifc_query_special_contexts_out_bits {
u8 null_mkey[0x20];
- u8 reserved_at_a0[0x60];
+ u8 terminate_scatter_list_mkey[0x20];
+
+ u8 repeated_mkey[0x20];
+
+ u8 reserved_at_a0[0x20];
};
struct mlx5_ifc_query_special_contexts_in_bits {
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Change define of 0x100 lkey value from MLX5_INVALID_LKEY to be
MLX5_TERMINATE_SCATTER_LIST_LKEY as 0x100 is the value of
terminate_scatter_list_mkey.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/infiniband/hw/mlx5/odp.c | 10 +++++-----
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 ++-
include/linux/mlx5/qp.h | 2 +-
5 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index e6e021af6aa9..b4ebeadce67c 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -986,7 +986,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
{
int ret = 0, npages = 0;
u64 io_virt;
- u32 key;
+ __be32 key;
u32 byte_count;
size_t bcnt;
int inline_segment;
@@ -1000,7 +1000,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
struct mlx5_wqe_data_seg *dseg = wqe;
io_virt = be64_to_cpu(dseg->addr);
- key = be32_to_cpu(dseg->lkey);
+ key = dseg->lkey;
byte_count = be32_to_cpu(dseg->byte_count);
inline_segment = !!(byte_count & MLX5_INLINE_SEG);
bcnt = byte_count & ~MLX5_INLINE_SEG;
@@ -1014,8 +1014,8 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
}
/* receive WQE end of sg list. */
- if (receive_queue && bcnt == 0 && key == MLX5_INVALID_LKEY &&
- io_virt == 0)
+ if (receive_queue && bcnt == 0 &&
+ key == MLX5_TERMINATE_SCATTER_LIST_LKEY && io_virt == 0)
break;
if (!inline_segment && total_wqe_bytes) {
@@ -1034,7 +1034,7 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
continue;
}
- ret = pagefault_single_data_segment(dev, NULL, key,
+ ret = pagefault_single_data_segment(dev, NULL, be32_to_cpu(key),
io_virt, bcnt,
&pfault->bytes_committed,
bytes_mapped);
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index 757756c50cc6..bcceb14a07f9 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -447,7 +447,7 @@ int mlx5_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
if (i < srq->msrq.max_avail_gather) {
scat[i].byte_count = 0;
- scat[i].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
scat[i].addr = 0;
}
}
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index 855f3f4fefad..bc44551493e2 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -1252,7 +1252,7 @@ int mlx5_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
if (i < qp->rq.max_gs) {
scat[i].byte_count = 0;
- scat[i].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
scat[i].addr = 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index cff5f2e29e1e..db4e66c38395 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -825,7 +825,8 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
/* check if num_frags is not a pow of two */
if (rq->wqe.info.num_frags < (1 << rq->wqe.info.log_num_frags)) {
wqe->data[f].byte_count = 0;
- wqe->data[f].lkey = cpu_to_be32(MLX5_INVALID_LKEY);
+ wqe->data[f].lkey =
+ MLX5_TERMINATE_SCATTER_LIST_LKEY;
wqe->data[f].addr = 0;
}
}
diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index 4657d5c54abe..df55fbb65717 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -36,7 +36,7 @@
#include <linux/mlx5/device.h>
#include <linux/mlx5/driver.h>
-#define MLX5_INVALID_LKEY 0x100
+#define MLX5_TERMINATE_SCATTER_LIST_LKEY cpu_to_be32(0x100)
/* UMR (3 WQE_BB's) + SIG (3 WQE_BB's) + PSV (mem) + PSV (wire) */
#define MLX5_SIG_WQE_SIZE (MLX5_SEND_WQE_BB * 8)
#define MLX5_DIF_SIZE 8
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma,
Michael Guralnik, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Use query_sepcial_contexts in order to get the correct value of
terminate_scatter_list_mkey, as FW will change it for certain
configurations.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
.../net/ethernet/mellanox/mlx5/core/en_main.c | 23 +++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index db4e66c38395..de7ab0911b30 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -661,6 +661,26 @@ static void mlx5e_rq_free_shampo(struct mlx5e_rq *rq)
mlx5e_rq_shampo_hd_free(rq);
}
+static __be32 mlx5e_get_terminate_scatter_list_mkey(struct mlx5_core_dev *dev)
+{
+ u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
+ u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
+ int res;
+
+ if (!MLX5_CAP_GEN(dev, terminate_scatter_list_mkey))
+ return MLX5_TERMINATE_SCATTER_LIST_LKEY;
+
+ MLX5_SET(query_special_contexts_in, in, opcode,
+ MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
+ res = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
+ if (res)
+ return MLX5_TERMINATE_SCATTER_LIST_LKEY;
+
+ res = MLX5_GET(query_special_contexts_out, out,
+ terminate_scatter_list_mkey);
+ return cpu_to_be32(res);
+}
+
static int mlx5e_alloc_rq(struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
struct mlx5e_rq_param *rqp,
@@ -825,8 +845,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
/* check if num_frags is not a pow of two */
if (rq->wqe.info.num_frags < (1 << rq->wqe.info.log_num_frags)) {
wqe->data[f].byte_count = 0;
- wqe->data[f].lkey =
- MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ wqe->data[f].lkey = mlx5e_get_terminate_scatter_list_mkey(mdev);
wqe->data[f].addr = 0;
}
}
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH rdma-next v2 4/4] RDMA/mlx5: Use query_special_contexts for mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
` (2 preceding siblings ...)
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
@ 2023-01-17 13:14 ` Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Leon Romanovsky @ 2023-01-17 13:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, David S. Miller, Eric Dumazet, Jakub Kicinski,
linux-rdma, Michael Guralnik, netdev, Paolo Abeni,
Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Use query_sepcial_contexts to get the correct value of mkeys such as
null_mkey, terminate_scatter_list_mkey and dump_fill_mkey, as FW will
change them in certain configurations.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Reviewed-by: Michael Guralnik <michaelgur@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/infiniband/hw/mlx5/cmd.c | 45 ++++++++++++++++------------
drivers/infiniband/hw/mlx5/cmd.h | 3 +-
drivers/infiniband/hw/mlx5/main.c | 10 +++----
drivers/infiniband/hw/mlx5/mlx5_ib.h | 9 +++++-
drivers/infiniband/hw/mlx5/odp.c | 21 ++++---------
drivers/infiniband/hw/mlx5/srq.c | 2 +-
drivers/infiniband/hw/mlx5/wr.c | 2 +-
7 files changed, 48 insertions(+), 44 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/cmd.c b/drivers/infiniband/hw/mlx5/cmd.c
index ff3742b0460a..1d0c8d5e745b 100644
--- a/drivers/infiniband/hw/mlx5/cmd.c
+++ b/drivers/infiniband/hw/mlx5/cmd.c
@@ -5,34 +5,41 @@
#include "cmd.h"
-int mlx5_cmd_dump_fill_mkey(struct mlx5_core_dev *dev, u32 *mkey)
+int mlx5r_cmd_query_special_mkeys(struct mlx5_ib_dev *dev)
{
u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
+ bool is_terminate, is_dump, is_null;
int err;
- MLX5_SET(query_special_contexts_in, in, opcode,
- MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
- err = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
- if (!err)
- *mkey = MLX5_GET(query_special_contexts_out, out,
- dump_fill_mkey);
- return err;
-}
+ is_terminate = MLX5_CAP_GEN(dev->mdev, terminate_scatter_list_mkey);
+ is_dump = MLX5_CAP_GEN(dev->mdev, dump_fill_mkey);
+ is_null = MLX5_CAP_GEN(dev->mdev, null_mkey);
-int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey)
-{
- u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
- u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
- int err;
+ dev->mkeys.terminate_scatter_list_mkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ if (!is_terminate && !is_dump && !is_null)
+ return 0;
MLX5_SET(query_special_contexts_in, in, opcode,
MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
- err = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
- if (!err)
- *null_mkey = MLX5_GET(query_special_contexts_out, out,
- null_mkey);
- return err;
+ err = mlx5_cmd_exec_inout(dev->mdev, query_special_contexts, in, out);
+ if (err)
+ return err;
+
+ if (is_dump)
+ dev->mkeys.dump_fill_mkey = MLX5_GET(query_special_contexts_out,
+ out, dump_fill_mkey);
+
+ if (is_null)
+ dev->mkeys.null_mkey = cpu_to_be32(
+ MLX5_GET(query_special_contexts_out, out, null_mkey));
+
+ if (is_terminate)
+ dev->mkeys.terminate_scatter_list_mkey =
+ cpu_to_be32(MLX5_GET(query_special_contexts_out, out,
+ terminate_scatter_list_mkey));
+
+ return 0;
}
int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,
diff --git a/drivers/infiniband/hw/mlx5/cmd.h b/drivers/infiniband/hw/mlx5/cmd.h
index ee46638db5de..93a971a40d11 100644
--- a/drivers/infiniband/hw/mlx5/cmd.h
+++ b/drivers/infiniband/hw/mlx5/cmd.h
@@ -37,8 +37,7 @@
#include <linux/kernel.h>
#include <linux/mlx5/driver.h>
-int mlx5_cmd_dump_fill_mkey(struct mlx5_core_dev *dev, u32 *mkey);
-int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey);
+int mlx5r_cmd_query_special_mkeys(struct mlx5_ib_dev *dev);
int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,
void *out);
int mlx5_cmd_dealloc_pd(struct mlx5_core_dev *dev, u32 pdn, u16 uid);
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 8588f2fc0cba..fb0d97bd4074 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1756,13 +1756,9 @@ static int set_ucontext_resp(struct ib_ucontext *uctx,
struct mlx5_ib_dev *dev = to_mdev(ibdev);
struct mlx5_ib_ucontext *context = to_mucontext(uctx);
struct mlx5_bfreg_info *bfregi = &context->bfregi;
- int err;
if (MLX5_CAP_GEN(dev->mdev, dump_fill_mkey)) {
- err = mlx5_cmd_dump_fill_mkey(dev->mdev,
- &resp->dump_fill_mkey);
- if (err)
- return err;
+ resp->dump_fill_mkey = dev->mkeys.dump_fill_mkey;
resp->comp_mask |=
MLX5_IB_ALLOC_UCONTEXT_RESP_MASK_DUMP_FILL_MKEY;
}
@@ -3634,6 +3630,10 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
dev->port[i].roce.last_port_state = IB_PORT_DOWN;
}
+ err = mlx5r_cmd_query_special_mkeys(dev);
+ if (err)
+ return err;
+
err = mlx5_ib_init_multiport_master(dev);
if (err)
return err;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 7e1fb71bf7d4..ddb36c757074 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1051,6 +1051,13 @@ struct mlx5_port_caps {
u8 ext_port_cap;
};
+
+struct mlx5_special_mkeys {
+ u32 dump_fill_mkey;
+ __be32 null_mkey;
+ __be32 terminate_scatter_list_mkey;
+};
+
struct mlx5_ib_dev {
struct ib_device ib_dev;
struct mlx5_core_dev *mdev;
@@ -1081,7 +1088,6 @@ struct mlx5_ib_dev {
struct xarray odp_mkeys;
- u32 null_mkey;
struct mlx5_ib_flow_db *flow_db;
/* protect resources needed as part of reset flow */
spinlock_t reset_flow_resource_lock;
@@ -1110,6 +1116,7 @@ struct mlx5_ib_dev {
struct mlx5_port_caps port_caps[MLX5_MAX_PORTS];
u16 pkey_table_len;
u8 lag_ports;
+ struct mlx5_special_mkeys mkeys;
};
static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index b4ebeadce67c..4998eaeadcbb 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -104,7 +104,7 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries,
if (flags & MLX5_IB_UPD_XLT_ZAP) {
for (; pklm != end; pklm++, idx++) {
pklm->bcount = cpu_to_be32(MLX5_IMR_MTT_SIZE);
- pklm->key = cpu_to_be32(mr_to_mdev(imr)->null_mkey);
+ pklm->key = mr_to_mdev(imr)->mkeys.null_mkey;
pklm->va = 0;
}
return;
@@ -137,7 +137,7 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries,
pklm->key = cpu_to_be32(mtt->ibmr.lkey);
pklm->va = cpu_to_be64(idx * MLX5_IMR_MTT_SIZE);
} else {
- pklm->key = cpu_to_be32(mr_to_mdev(imr)->null_mkey);
+ pklm->key = mr_to_mdev(imr)->mkeys.null_mkey;
pklm->va = 0;
}
}
@@ -1015,7 +1015,8 @@ static int pagefault_data_segments(struct mlx5_ib_dev *dev,
/* receive WQE end of sg list. */
if (receive_queue && bcnt == 0 &&
- key == MLX5_TERMINATE_SCATTER_LIST_LKEY && io_virt == 0)
+ key == dev->mkeys.terminate_scatter_list_mkey &&
+ io_virt == 0)
break;
if (!inline_segment && total_wqe_bytes) {
@@ -1615,25 +1616,15 @@ static const struct ib_device_ops mlx5_ib_dev_odp_ops = {
int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
{
- int ret = 0;
-
internal_fill_odp_caps(dev);
if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
- return ret;
+ return 0;
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
- if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
- ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);
- if (ret) {
- mlx5_ib_err(dev, "Error getting null_mkey %d\n", ret);
- return ret;
- }
- }
-
mutex_init(&dev->odp_eq_mutex);
- return ret;
+ return 0;
}
void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *dev)
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index bcceb14a07f9..32c6643d0f7a 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -447,7 +447,7 @@ int mlx5_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
if (i < srq->msrq.max_avail_gather) {
scat[i].byte_count = 0;
- scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ scat[i].lkey = dev->mkeys.terminate_scatter_list_mkey;
scat[i].addr = 0;
}
}
diff --git a/drivers/infiniband/hw/mlx5/wr.c b/drivers/infiniband/hw/mlx5/wr.c
index bc44551493e2..df1d1b0a3ef7 100644
--- a/drivers/infiniband/hw/mlx5/wr.c
+++ b/drivers/infiniband/hw/mlx5/wr.c
@@ -1252,7 +1252,7 @@ int mlx5_ib_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
if (i < qp->rq.max_gs) {
scat[i].byte_count = 0;
- scat[i].lkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
+ scat[i].lkey = dev->mkeys.terminate_scatter_list_mkey;
scat[i].addr = 0;
}
--
2.39.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
` (3 preceding siblings ...)
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
@ 2023-02-17 20:30 ` Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2023-02-17 20:30 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Eric Dumazet, Jakub Kicinski, linux-kernel, linux-rdma,
Michael Guralnik, netdev, Or Har-Toov, Paolo Abeni,
Saeed Mahameed
On Tue, Jan 17, 2023 at 03:14:48PM +0200, Leon Romanovsky wrote:
> Changelog:
> v2:
> * Took a liberty and rewrote net/mlx5 patches
> * change logic around terminate_scatter_list_mkey
> * Added capability checks to be before executing command
> v1: https://lore.kernel.org/all/cover.1672917578.git.leonro@nvidia.com
> * Use already stored mkeys.terminate_scatter_list_mkey.
> v0: https://lore.kernel.org/all/cover.1672819469.git.leonro@nvidia.com
>
> -----------------------------------------------------------------------
> This series from Or extends mlx5 driver to rely on firmware to get
> special mkey values.
>
> Thanks
>
> Or Har-Toov (4):
> net/mlx5: Expose bits for querying special mkeys
> net/mlx5: Change define name for 0x100 lkey value
> net/mlx5e: Use query_special_contexts for mkeys
> RDMA/mlx5: Use query_special_contexts for mkeys
I pulled this into rdma, Saeed/Leon if you need these commits in the
shared branch then you can fast-forward the shared branch to:
commit 1b1e4868836a4b5b375be75fd4c9583d29500517
Author: Or Har-Toov <ohartoov@nvidia.com>
Date: Tue Jan 17 15:14:51 2023 +0200
net/mlx5e: Use query_special_contexts for mkeys
Thanks,
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-02-17 20:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-17 13:14 [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 1/4] net/mlx5: Expose bits for querying " Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 2/4] net/mlx5: Change define name for 0x100 lkey value Leon Romanovsky
2023-01-17 13:14 ` [PATCH mlx5-next v2 3/4] net/mlx5e: Use query_special_contexts for mkeys Leon Romanovsky
2023-01-17 13:14 ` [PATCH rdma-next v2 4/4] RDMA/mlx5: " Leon Romanovsky
2023-02-17 20:30 ` [PATCH rdma-next v2 0/4] Rely on firmware to get special mkeys Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).