* [PATCH mlx5-next 1/3] net/mlx5: Expose bits for enabling out-of-order by default
2023-03-19 12:59 [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
@ 2023-03-19 12:59 ` Leon Romanovsky
2023-03-19 12:59 ` [PATCH rdma-next 2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs Leon Romanovsky
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Leon Romanovsky @ 2023-03-19 12:59 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma, netdev,
Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Add needed HW bits for enabling out-of-order by default and
use go_back_n when out-of-order is not needed.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
include/linux/mlx5/mlx5_ifc.h | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 15a850f52ef2..e4306cd87cd7 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1077,7 +1077,9 @@ struct mlx5_ifc_roce_cap_bits {
u8 sw_r_roce_src_udp_port[0x1];
u8 fl_rc_qp_when_roce_disabled[0x1];
u8 fl_rc_qp_when_roce_enabled[0x1];
- u8 reserved_at_7[0x17];
+ u8 reserved_at_7[0x1];
+ u8 qp_ooo_transmit_default[0x1];
+ u8 reserved_at_9[0x15];
u8 qp_ts_format[0x2];
u8 reserved_at_20[0x60];
@@ -1493,7 +1495,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
u8 reserved_at_b0[0x1];
u8 uplink_follow[0x1];
u8 ts_cqe_to_dest_cqn[0x1];
- u8 reserved_at_b3[0x7];
+ u8 reserved_at_b3[0x6];
+ u8 go_back_n[0x1];
u8 shampo[0x1];
u8 reserved_at_bb[0x5];
@@ -3263,7 +3266,8 @@ struct mlx5_ifc_qpc_bits {
u8 log_rq_stride[0x3];
u8 no_sq[0x1];
u8 log_sq_size[0x4];
- u8 reserved_at_55[0x3];
+ u8 reserved_at_55[0x1];
+ u8 retry_mode[0x2];
u8 ts_format[0x2];
u8 reserved_at_5a[0x1];
u8 rlky[0x1];
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH rdma-next 2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs
2023-03-19 12:59 [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
2023-03-19 12:59 ` [PATCH mlx5-next 1/3] net/mlx5: Expose bits for enabling out-of-order by default Leon Romanovsky
@ 2023-03-19 12:59 ` Leon Romanovsky
2023-03-19 12:59 ` [PATCH mlx5-next 3/3] net/mlx5: Set out of order (ooo) by default Leon Romanovsky
2023-03-23 8:25 ` [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
3 siblings, 0 replies; 5+ messages in thread
From: Leon Romanovsky @ 2023-03-19 12:59 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, David S. Miller, Eric Dumazet, Jakub Kicinski,
linux-rdma, netdev, Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
Set retry_mode to GO_BACK_N when qp is created with INTEGRITY_EN flag
because out-of-order is not supported when doing HW offload of signature
operations.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/infiniband/hw/mlx5/qp.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index f04adc18e63b..d32b644885b3 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -60,6 +60,10 @@ enum raw_qp_set_mask_map {
MLX5_RAW_QP_RATE_LIMIT = 1UL << 1,
};
+enum {
+ MLX5_QP_RM_GO_BACK_N = 0x1,
+};
+
struct mlx5_modify_raw_qp_param {
u16 operation;
@@ -2521,6 +2525,10 @@ static int create_kernel_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
if (qp->flags & IB_QP_CREATE_IPOIB_UD_LSO)
MLX5_SET(qpc, qpc, ulp_stateless_offload_mode, 1);
+ if (qp->flags & IB_QP_CREATE_INTEGRITY_EN &&
+ MLX5_CAP_GEN(mdev, go_back_n))
+ MLX5_SET(qpc, qpc, retry_mode, MLX5_QP_RM_GO_BACK_N);
+
err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
kvfree(in);
if (err)
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH mlx5-next 3/3] net/mlx5: Set out of order (ooo) by default
2023-03-19 12:59 [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
2023-03-19 12:59 ` [PATCH mlx5-next 1/3] net/mlx5: Expose bits for enabling out-of-order by default Leon Romanovsky
2023-03-19 12:59 ` [PATCH rdma-next 2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs Leon Romanovsky
@ 2023-03-19 12:59 ` Leon Romanovsky
2023-03-23 8:25 ` [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
3 siblings, 0 replies; 5+ messages in thread
From: Leon Romanovsky @ 2023-03-19 12:59 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Har-Toov, Eric Dumazet, Jakub Kicinski, linux-rdma, netdev,
Paolo Abeni, Saeed Mahameed
From: Or Har-Toov <ohartoov@nvidia.com>
When FW supports ooo by default, enable the cap.
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
drivers/net/ethernet/mellanox/mlx5/core/main.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 540840e80493..564a82ac3787 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -684,6 +684,9 @@ static int handle_hca_cap_roce(struct mlx5_core_dev *dev, void *set_ctx)
MLX5_ST_SZ_BYTES(roce_cap));
MLX5_SET(roce_cap, set_hca_cap, sw_r_roce_src_udp_port, 1);
+ if (MLX5_CAP_ROCE_MAX(dev, qp_ooo_transmit_default))
+ MLX5_SET(roce_cap, set_hca_cap, qp_ooo_transmit_default, 1);
+
err = set_caps(dev, set_ctx, MLX5_SET_HCA_CAP_OP_MOD_ROCE);
return err;
}
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5
2023-03-19 12:59 [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5 Leon Romanovsky
` (2 preceding siblings ...)
2023-03-19 12:59 ` [PATCH mlx5-next 3/3] net/mlx5: Set out of order (ooo) by default Leon Romanovsky
@ 2023-03-23 8:25 ` Leon Romanovsky
3 siblings, 0 replies; 5+ messages in thread
From: Leon Romanovsky @ 2023-03-23 8:25 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky
Cc: Eric Dumazet, Jakub Kicinski, linux-kernel, linux-rdma, netdev,
Or Har-Toov, Paolo Abeni, Saeed Mahameed, Leon Romanovsky
On Sun, 19 Mar 2023 14:59:29 +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
>
> Hi,
>
> This series from Or changes default of IB out-of-order feature and
> allows to the RDMA users to decide if they need to wait for completion
> for all segments or it is enough to wait for last segment completion only.
>
> [...]
Applied, thanks!
[1/3] net/mlx5: Expose bits for enabling out-of-order by default
https://git.kernel.org/rdma/rdma/c/6e2a3a324aab9d
[2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs
https://git.kernel.org/rdma/rdma/c/742948cc02d523
[3/3] net/mlx5: Set out of order (ooo) by default
https://git.kernel.org/rdma/rdma/c/f4244e55e4c3a1
Best regards,
--
Leon Romanovsky <leon@kernel.org>
^ permalink raw reply [flat|nested] 5+ messages in thread