[PATCH rdma-next 0/3] Extend mlx5_ib software steering interface

[PATCH rdma-next 0/3] Extend mlx5_ib software steering interface

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

This series from Alex extends software steering interface to support
devices with extra capability "sw_owner_2" which will replace existing
"sw_owner".

Thanks

Alex Vesker (3):
  RDMA/mlx5: Add sw_owner_v2 bit capability
  RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices
  RDMA/mlx5: Expose TIR and QP ICM address for sw_owner_v2 devices

 drivers/infiniband/hw/mlx5/main.c | 4 +++-
 drivers/infiniband/hw/mlx5/qp.c   | 6 ++++--
 include/linux/mlx5/mlx5_ifc.h     | 3 ++-
 3 files changed, 9 insertions(+), 4 deletions(-)

--
2.26.2

[PATCH mlx5-next 1/3] RDMA/mlx5: Add sw_owner_v2 bit capability

[Leon Romanovsky]

From: Alex Vesker <valex@nvidia.com>

Added sw_owner_v2 which will be enabled for future devices,
replacing sw_owner bit.

Signed-off-by: Alex Vesker <valex@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 include/linux/mlx5/mlx5_ifc.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index aee25e4fb2cc..651591a2965d 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -420,7 +420,8 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8         reserved_at_1a[0x2];
 	u8         ipsec_encrypt[0x1];
 	u8         ipsec_decrypt[0x1];
-	u8         reserved_at_1e[0x2];
+	u8         sw_owner_v2[0x1];
+	u8         reserved_at_1f[0x1];

 	u8         termination_table_raw_traffic[0x1];
 	u8         reserved_at_21[0x1];
--
2.26.2

[PATCH rdma-next 2/3] RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices

[Leon Romanovsky]

From: Alex Vesker <valex@nvidia.com>

sw_owner_v2 will replace sw_owner for future devices, this means
that if sw_owner_v2 is set sw_owner should be ignored and DM
allocation is required for sw_owner_v2 devices to function.

Signed-off-by: Alex Vesker <valex@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/main.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 99dbef0bccbc..8963b806ad19 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -2343,7 +2343,9 @@ static inline int check_dm_type_support(struct mlx5_ib_dev *dev,
 			return -EPERM;
 
 		if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
-		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner)))
+		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner) ||
+		      MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2) ||
+		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner_v2)))
 			return -EOPNOTSUPP;
 		break;
 	}
-- 
2.26.2

[PATCH rdma-next 3/3] RDMA/mlx5: Expose TIR and QP ICM address for sw_owner_v2 devices

[Leon Romanovsky]

From: Alex Vesker <valex@nvidia.com>

Expose the ICM address to access TIR and QP, this will allow sw_owned_v2
devices to steer traffic to TIRs and QPs same as done with sw_owner
capability.

Signed-off-by: Alex Vesker <valex@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 22280ea92006..ada852fcc427 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -1477,7 +1477,8 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_RQN;
 			resp->tirn = rq->tirn;
 			resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
-			if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+			if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
+			    MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2)) {
 				resp->tir_icm_addr = MLX5_GET(
 					create_tir_out, out, icm_address_31_0);
 				resp->tir_icm_addr |=
@@ -1739,7 +1740,8 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 	if (mucontext->devx_uid) {
 		params->resp.comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
 		params->resp.tirn = qp->rss_qp.tirn;
-		if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+		if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
+		    MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2)) {
 			params->resp.tir_icm_addr =
 				MLX5_GET(create_tir_out, out, icm_address_31_0);
 			params->resp.tir_icm_addr |=
-- 
2.26.2

Re: [PATCH rdma-next 2/3] RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices

[Jason Gunthorpe]

On Thu, Sep 03, 2020 at 10:38:56AM +0300, Leon Romanovsky wrote:
> From: Alex Vesker <valex@nvidia.com>
> 
> sw_owner_v2 will replace sw_owner for future devices, this means
> that if sw_owner_v2 is set sw_owner should be ignored and DM
> allocation is required for sw_owner_v2 devices to function.
> 
> Signed-off-by: Alex Vesker <valex@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>  drivers/infiniband/hw/mlx5/main.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index 99dbef0bccbc..8963b806ad19 100644
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -2343,7 +2343,9 @@ static inline int check_dm_type_support(struct mlx5_ib_dev *dev,
>  			return -EPERM;
>  
>  		if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
> -		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner)))
> +		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner) ||
> +		      MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2) ||
> +		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner_v2)))
>  			return -EOPNOTSUPP;

Shouldn't user space ask for MLX5_IB_UAPI_DM_TYPE_STEERING_SW_V2_ICM
types too?

What happens if old user space runs on a new device?

Jason

Re: [PATCH rdma-next 2/3] RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices

[Leon Romanovsky]

On Tue, Sep 15, 2020 at 05:04:47PM -0300, Jason Gunthorpe wrote:
> On Thu, Sep 03, 2020 at 10:38:56AM +0300, Leon Romanovsky wrote:
> > From: Alex Vesker <valex@nvidia.com>
> >
> > sw_owner_v2 will replace sw_owner for future devices, this means
> > that if sw_owner_v2 is set sw_owner should be ignored and DM
> > allocation is required for sw_owner_v2 devices to function.
> >
> > Signed-off-by: Alex Vesker <valex@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> >  drivers/infiniband/hw/mlx5/main.c | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> > index 99dbef0bccbc..8963b806ad19 100644
> > +++ b/drivers/infiniband/hw/mlx5/main.c
> > @@ -2343,7 +2343,9 @@ static inline int check_dm_type_support(struct mlx5_ib_dev *dev,
> >  			return -EPERM;
> >
> >  		if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
> > -		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner)))
> > +		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner) ||
> > +		      MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2) ||
> > +		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner_v2)))
> >  			return -EOPNOTSUPP;
>
> Shouldn't user space ask for MLX5_IB_UAPI_DM_TYPE_STEERING_SW_V2_ICM
> types too?

For sw_owner_v2 devices, we will return exactly same type of DM,
so there is no need for a new one. The change is to allow creation
of such DM in addition to existing sw_owner devices.


>
> What happens if old user space runs on a new device?

Old userspace will have sw_owner_v2 cap enabled and will get DM as before.
The patch diff is hard to read but the end result will be.
	if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
	      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner) ||
	      MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner_v2) ||
	      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner_v2)))
		return -EOPNOTSUPP;

DevX users are expected to check capabilities and be ready to work with missing ones.

Thanks

>
> Jason

Re: [PATCH rdma-next 0/3] Extend mlx5_ib software steering interface

[Jason Gunthorpe]

On Thu, Sep 03, 2020 at 10:38:54AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> This series from Alex extends software steering interface to support
> devices with extra capability "sw_owner_2" which will replace existing
> "sw_owner".
> 
> Thanks
> 
> Alex Vesker (3):
>   RDMA/mlx5: Add sw_owner_v2 bit capability
>   RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices
>   RDMA/mlx5: Expose TIR and QP ICM address for sw_owner_v2 devices

Ok, can you update the shared branch with the first patch? Thanks

Jason

Re: [PATCH rdma-next 0/3] Extend mlx5_ib software steering interface

[Leon Romanovsky]

On Thu, Sep 17, 2020 at 03:10:26PM -0300, Jason Gunthorpe wrote:
> On Thu, Sep 03, 2020 at 10:38:54AM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@nvidia.com>
> >
> > This series from Alex extends software steering interface to support
> > devices with extra capability "sw_owner_2" which will replace existing
> > "sw_owner".
> >
> > Thanks
> >
> > Alex Vesker (3):
> >   RDMA/mlx5: Add sw_owner_v2 bit capability
> >   RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices
> >   RDMA/mlx5: Expose TIR and QP ICM address for sw_owner_v2 devices
>
> Ok, can you update the shared branch with the first patch? Thanks

Done, thanks
9d8feb460adb RDMA/mlx5: Add sw_owner_v2 bit capability

>
> Jason

Re: [PATCH rdma-next 0/3] Extend mlx5_ib software steering interface

[Jason Gunthorpe]

On Thu, Sep 17, 2020 at 09:13:21PM +0300, Leon Romanovsky wrote:
> On Thu, Sep 17, 2020 at 03:10:26PM -0300, Jason Gunthorpe wrote:
> > On Thu, Sep 03, 2020 at 10:38:54AM +0300, Leon Romanovsky wrote:
> > > From: Leon Romanovsky <leonro@nvidia.com>
> > >
> > > This series from Alex extends software steering interface to support
> > > devices with extra capability "sw_owner_2" which will replace existing
> > > "sw_owner".
> > >
> > > Thanks
> > >
> > > Alex Vesker (3):
> > >   RDMA/mlx5: Add sw_owner_v2 bit capability
> > >   RDMA/mlx5: Allow DM allocation for sw_owner_v2 enabled devices
> > >   RDMA/mlx5: Expose TIR and QP ICM address for sw_owner_v2 devices
> >
> > Ok, can you update the shared branch with the first patch? Thanks
> 
> Done, thanks
> 9d8feb460adb RDMA/mlx5: Add sw_owner_v2 bit capability

Applied to for-next, thanks

Jason