[PATCH rdma-next 0/8] User space steering

[PATCH rdma-next 0/8] User space steering

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

>From Ariel,

This series of patches adds user space managed steering infrastructure
to the mlx5_ib driver.

User space managed steering requires the means to access a dedicated
memory space that is used by the device to store the packet steering
and header modification tables and rules in order to manage them directly
without the device's firmware involvement. This dedicated memory is part
of the ICM memory space.

The changes are introducing the mlx5_ib API to allocate, deallocate and
register this dedicated SW ICM memory via the existing device memory API
using a private attribute which specifies the memory type.

The allocated memory itself is not IO mapped and user can only access it
using remote RDMA operations.

In addition, the series exposed the ICM address of the receive transport
interface (TIR) of Raw Packet and RSS QPs to user since they are required
to properly create and insert steering rules that direct flows to these QPs.

Thanks

Ariel Levkovich (8):
  net/mlx5: Expose SW ICM related device memory capabilities
  IB/mlx5: Support device memory type attribute
  IB/mlx5: Warn on allocated MEMIC buffers during cleanup
  IB/mlx5: Add steering SW ICM device memory type
  IB/mlx5: Device resource control for privileged DEVX user
  net/mlx5: Expose TIR ICM address in command outbox
  net/mlx5: Introduce new TIR creation core API
  IB/mlx5: Expose TIR ICM address to user space

 drivers/infiniband/hw/mlx5/Kconfig            |   2 +-
 drivers/infiniband/hw/mlx5/cmd.c              | 155 +++++++++-
 drivers/infiniband/hw/mlx5/cmd.h              |   8 +-
 drivers/infiniband/hw/mlx5/devx.c             |   4 +
 drivers/infiniband/hw/mlx5/main.c             | 264 +++++++++++++++---
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  40 ++-
 drivers/infiniband/hw/mlx5/mr.c               |  40 ++-
 drivers/infiniband/hw/mlx5/qp.c               |  46 ++-
 .../ethernet/mellanox/mlx5/core/transobj.c    |  18 +-
 include/linux/mlx5/mlx5_ifc.h                 |  51 +++-
 include/linux/mlx5/transobj.h                 |   3 +
 include/uapi/rdma/mlx5-abi.h                  |   2 +
 include/uapi/rdma/mlx5_user_ioctl_cmds.h      |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h     |   6 +
 14 files changed, 541 insertions(+), 99 deletions(-)

[PATCH rdma-next 0/8] User space steering

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

From Ariel,

This series of patches adds user space managed steering infrastructure
to the mlx5_ib driver.

User space managed steering requires the means to access a dedicated
memory space that is used by the device to store the packet steering
and header modification tables and rules in order to manage them directly
without the device's firmware involvement. This dedicated memory is part
of the ICM memory space.

The changes are introducing the mlx5_ib API to allocate, deallocate and
register this dedicated SW ICM memory via the existing device memory API
using a private attribute which specifies the memory type.

The allocated memory itself is not IO mapped and user can only access it
using remote RDMA operations.

In addition, the series exposed the ICM address of the receive transport
interface (TIR) of Raw Packet and RSS QPs to user since they are required
to properly create and insert steering rules that direct flows to these QPs.

Thanks

Ariel Levkovich (8):
  net/mlx5: Expose SW ICM related device memory capabilities
  IB/mlx5: Support device memory type attribute
  IB/mlx5: Warn on allocated MEMIC buffers during cleanup
  IB/mlx5: Add steering SW ICM device memory type
  IB/mlx5: Device resource control for privileged DEVX user
  net/mlx5: Expose TIR ICM address in command outbox
  net/mlx5: Introduce new TIR creation core API
  IB/mlx5: Expose TIR ICM address to user space

 drivers/infiniband/hw/mlx5/Kconfig            |   2 +-
 drivers/infiniband/hw/mlx5/cmd.c              | 155 +++++++++-
 drivers/infiniband/hw/mlx5/cmd.h              |   8 +-
 drivers/infiniband/hw/mlx5/devx.c             |   4 +
 drivers/infiniband/hw/mlx5/main.c             | 264 +++++++++++++++---
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  40 ++-
 drivers/infiniband/hw/mlx5/mr.c               |  40 ++-
 drivers/infiniband/hw/mlx5/qp.c               |  46 ++-
 .../ethernet/mellanox/mlx5/core/transobj.c    |  18 +-
 include/linux/mlx5/mlx5_ifc.h                 |  51 +++-
 include/linux/mlx5/transobj.h                 |   3 +
 include/uapi/rdma/mlx5-abi.h                  |   2 +
 include/uapi/rdma/mlx5_user_ioctl_cmds.h      |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h     |   6 +
 14 files changed, 541 insertions(+), 99 deletions(-)

--
2.20.1

[PATCH mlx5-next 1/8] net/mlx5: Expose SW ICM related device memory capabilities

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

Add SW ICM related fields to the device memory capabilities
structure and sw ownership capability in flow table properties.

The currently supported SW ICM types are steering and header modify
and the changes exposes the device memory capabilities for each
of these two types.

SW ICM memory can be allocated by SW and then be accessed by RDMA
operations for direct management of the HW packet handling tables.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 include/linux/mlx5/mlx5_ifc.h | 45 +++++++++++++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 20c07cb87f6c..c2a558d58067 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -80,6 +80,14 @@ enum {
 	MLX5_SHARED_RESOURCE_UID = 0xffff,
 };
 
+enum {
+	MLX5_OBJ_TYPE_SW_ICM = 0x0008,
+};
+
+enum {
+	MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM = (1ULL << MLX5_OBJ_TYPE_SW_ICM),
+};
+
 enum {
 	MLX5_CMD_OP_QUERY_HCA_CAP                 = 0x100,
 	MLX5_CMD_OP_QUERY_ADAPTER                 = 0x101,
@@ -357,7 +365,8 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8         pop_vlan_2[0x1];
 	u8         push_vlan_2[0x1];
 	u8	   reformat_and_vlan_action[0x1];
-	u8	   reserved_at_10[0x2];
+	u8	   reserved_at_10[0x1];
+	u8         sw_owner[0x1];
 	u8	   reformat_l3_tunnel_to_l2[0x1];
 	u8	   reformat_l2_to_l3_tunnel[0x1];
 	u8	   reformat_and_modify_action[0x1];
@@ -770,7 +779,19 @@ struct mlx5_ifc_device_mem_cap_bits {
 
 	u8         max_memic_size[0x20];
 
-	u8         reserved_at_c0[0x740];
+	u8         steering_sw_icm_start_address[0x40];
+
+	u8         reserved_at_100[0x8];
+	u8         log_header_modify_sw_icm_size[0x8];
+	u8         reserved_at_110[0x2];
+	u8         log_sw_icm_alloc_granularity[0x6];
+	u8         log_steering_sw_icm_size[0x8];
+
+	u8         reserved_at_120[0x20];
+
+	u8         header_modify_sw_icm_start_address[0x40];
+
+	u8         reserved_at_180[0x680];
 };
 
 enum {
@@ -919,6 +940,7 @@ enum {
 
 enum {
 	MLX5_UCTX_CAP_RAW_TX = 1UL << 0,
+	MLX5_UCTX_CAP_INTERNAL_DEV_RES = 1UL << 1,
 };
 
 struct mlx5_ifc_cmd_hca_cap_bits {
@@ -2922,6 +2944,7 @@ enum {
 	MLX5_MKC_ACCESS_MODE_MTT   = 0x1,
 	MLX5_MKC_ACCESS_MODE_KLMS  = 0x2,
 	MLX5_MKC_ACCESS_MODE_KSM   = 0x3,
+	MLX5_MKC_ACCESS_MODE_SW_ICM = 0x4,
 	MLX5_MKC_ACCESS_MODE_MEMIC = 0x5,
 };
 
@@ -9444,6 +9467,19 @@ struct mlx5_ifc_uctx_bits {
 	u8         reserved_at_20[0x160];
 };
 
+struct mlx5_ifc_sw_icm_bits {
+	u8         modify_field_select[0x40];
+
+	u8	   reserved_at_40[0x18];
+	u8         log_sw_icm_size[0x8];
+
+	u8         reserved_at_60[0x20];
+
+	u8         sw_icm_start_addr[0x40];
+
+	u8         reserved_at_c0[0x140];
+};
+
 struct mlx5_ifc_create_umem_in_bits {
 	u8         opcode[0x10];
 	u8         uid[0x10];
@@ -9481,6 +9517,11 @@ struct mlx5_ifc_destroy_uctx_in_bits {
 	u8         reserved_at_60[0x20];
 };
 
+struct mlx5_ifc_create_sw_icm_in_bits {
+	struct mlx5_ifc_general_obj_in_cmd_hdr_bits   hdr;
+	struct mlx5_ifc_sw_icm_bits		      sw_icm;
+};
+
 struct mlx5_ifc_mtrc_string_db_param_bits {
 	u8         string_db_base_address[0x20];
 
-- 
2.20.1

[PATCH rdma-next 2/8] IB/mlx5: Support device memory type attribute

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

This patch intoruduces a new mlx5_ib driver attribute
to the DM allocation method - the DM type.

In order to allow addition of new types in downstream patches
this patch also refactors the allocation, deallocation and
registration handlers to consider the requested type and
perform the necessary actions according to it.

Since not all future device memory types will be such that are mapped
to user memory, the mandatory page index output attribute is modified
to be optional.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/cmd.c          |  30 ++---
 drivers/infiniband/hw/mlx5/cmd.h          |   4 +-
 drivers/infiniband/hw/mlx5/main.c         | 130 ++++++++++++++--------
 drivers/infiniband/hw/mlx5/mlx5_ib.h      |  23 ++--
 drivers/infiniband/hw/mlx5/mr.c           |  33 +++---
 include/uapi/rdma/mlx5_user_ioctl_cmds.h  |   1 +
 include/uapi/rdma/mlx5_user_ioctl_verbs.h |   4 +
 7 files changed, 143 insertions(+), 82 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/cmd.c b/drivers/infiniband/hw/mlx5/cmd.c
index 6bcc63aaa50b..a405d81ba01a 100644
--- a/drivers/infiniband/hw/mlx5/cmd.c
+++ b/drivers/infiniband/hw/mlx5/cmd.c
@@ -82,10 +82,10 @@ int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *dev,
 	return mlx5_cmd_exec(dev, in, in_size, out, sizeof(out));
 }
 
-int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
-			  u64 length, u32 alignment)
+int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
+			 u64 length, u32 alignment)
 {
-	struct mlx5_core_dev *dev = memic->dev;
+	struct mlx5_core_dev *dev = dm->dev;
 	u64 num_memic_hw_pages = MLX5_CAP_DEV_MEM(dev, memic_bar_size)
 					>> PAGE_SHIFT;
 	u64 hw_start_addr = MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr);
@@ -115,17 +115,17 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
 		 mlx5_alignment);
 
 	while (page_idx < num_memic_hw_pages) {
-		spin_lock(&memic->memic_lock);
-		page_idx = bitmap_find_next_zero_area(memic->memic_alloc_pages,
+		spin_lock(&dm->lock);
+		page_idx = bitmap_find_next_zero_area(dm->memic_alloc_pages,
 						      num_memic_hw_pages,
 						      page_idx,
 						      num_pages, 0);
 
 		if (page_idx < num_memic_hw_pages)
-			bitmap_set(memic->memic_alloc_pages,
+			bitmap_set(dm->memic_alloc_pages,
 				   page_idx, num_pages);
 
-		spin_unlock(&memic->memic_lock);
+		spin_unlock(&dm->lock);
 
 		if (page_idx >= num_memic_hw_pages)
 			break;
@@ -135,10 +135,10 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
 
 		ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
 		if (ret) {
-			spin_lock(&memic->memic_lock);
-			bitmap_clear(memic->memic_alloc_pages,
+			spin_lock(&dm->lock);
+			bitmap_clear(dm->memic_alloc_pages,
 				     page_idx, num_pages);
-			spin_unlock(&memic->memic_lock);
+			spin_unlock(&dm->lock);
 
 			if (ret == -EAGAIN) {
 				page_idx++;
@@ -157,9 +157,9 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
 	return -ENOMEM;
 }
 
-int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length)
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, u64 addr, u64 length)
 {
-	struct mlx5_core_dev *dev = memic->dev;
+	struct mlx5_core_dev *dev = dm->dev;
 	u64 hw_start_addr = MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr);
 	u32 num_pages = DIV_ROUND_UP(length, PAGE_SIZE);
 	u32 out[MLX5_ST_SZ_DW(dealloc_memic_out)] = {0};
@@ -177,10 +177,10 @@ int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length)
 	err =  mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
 
 	if (!err) {
-		spin_lock(&memic->memic_lock);
-		bitmap_clear(memic->memic_alloc_pages,
+		spin_lock(&dm->lock);
+		bitmap_clear(dm->memic_alloc_pages,
 			     start_page_idx, num_pages);
-		spin_unlock(&memic->memic_lock);
+		spin_unlock(&dm->lock);
 	}
 
 	return err;
diff --git a/drivers/infiniband/hw/mlx5/cmd.h b/drivers/infiniband/hw/mlx5/cmd.h
index 923a7b93f507..80a644bea6c7 100644
--- a/drivers/infiniband/hw/mlx5/cmd.h
+++ b/drivers/infiniband/hw/mlx5/cmd.h
@@ -44,9 +44,9 @@ int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,
 int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out);
 int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *mdev,
 				void *in, int in_size);
-int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
+int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
 			 u64 length, u32 alignment);
-int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length);
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, u64 addr, u64 length);
 void mlx5_cmd_dealloc_pd(struct mlx5_core_dev *dev, u32 pdn, u16 uid);
 void mlx5_cmd_destroy_tir(struct mlx5_core_dev *dev, u32 tirn, u16 uid);
 void mlx5_cmd_destroy_tis(struct mlx5_core_dev *dev, u32 tisn, u16 uid);
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 4be8e5439c41..562826ecfbdd 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -2291,58 +2291,90 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm
 	return 0;
 }
 
-struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
-			       struct ib_ucontext *context,
-			       struct ib_dm_alloc_attr *attr,
-			       struct uverbs_attr_bundle *attrs)
+static int handle_alloc_dm_memic(struct ib_ucontext *ctx,
+				 struct mlx5_ib_dm *dm,
+				 struct ib_dm_alloc_attr *attr,
+				 struct uverbs_attr_bundle *attrs)
 {
-	u64 act_size = roundup(attr->length, MLX5_MEMIC_BASE_SIZE);
-	struct mlx5_memic *memic = &to_mdev(ibdev)->memic;
-	phys_addr_t memic_addr;
-	struct mlx5_ib_dm *dm;
+	struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;
 	u64 start_offset;
 	u32 page_idx;
 	int err;
 
-	dm = kzalloc(sizeof(*dm), GFP_KERNEL);
-	if (!dm)
-		return ERR_PTR(-ENOMEM);
-
-	mlx5_ib_dbg(to_mdev(ibdev), "alloc_memic req: user_length=0x%llx act_length=0x%llx log_alignment=%d\n",
-		    attr->length, act_size, attr->alignment);
+	dm->size = roundup(attr->length, MLX5_MEMIC_BASE_SIZE);
 
-	err = mlx5_cmd_alloc_memic(memic, &memic_addr,
-				   act_size, attr->alignment);
+	err = mlx5_cmd_alloc_memic(dm_db, &dm->dev_addr,
+				   dm->size, attr->alignment);
 	if (err)
-		goto err_free;
+		return err;
 
-	start_offset = memic_addr & ~PAGE_MASK;
-	page_idx = (memic_addr - pci_resource_start(memic->dev->pdev, 0) -
-		    MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >>
+	page_idx = (dm->dev_addr - pci_resource_start(dm_db->dev->pdev, 0) -
+		    MLX5_CAP64_DEV_MEM(dm_db->dev, memic_bar_start_addr)) >>
 		    PAGE_SHIFT;
 
 	err = uverbs_copy_to(attrs,
-			     MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
-			     &start_offset, sizeof(start_offset));
+			     MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
+			     &page_idx, sizeof(page_idx));
 	if (err)
 		goto err_dealloc;
 
+	start_offset = dm->dev_addr & ~PAGE_MASK;
 	err = uverbs_copy_to(attrs,
-			     MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
-			     &page_idx, sizeof(page_idx));
+			     MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
+			     &start_offset, sizeof(start_offset));
 	if (err)
 		goto err_dealloc;
 
-	bitmap_set(to_mucontext(context)->dm_pages, page_idx,
-		   DIV_ROUND_UP(act_size, PAGE_SIZE));
+	bitmap_set(to_mucontext(ctx)->dm_pages, page_idx,
+		   DIV_ROUND_UP(dm->size, PAGE_SIZE));
+
+	return 0;
+
+err_dealloc:
+	mlx5_cmd_dealloc_memic(dm_db, dm->dev_addr, dm->size);
+
+	return err;
+}
+
+struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
+			       struct ib_ucontext *context,
+			       struct ib_dm_alloc_attr *attr,
+			       struct uverbs_attr_bundle *attrs)
+{
+	struct mlx5_ib_dm *dm;
+	enum mlx5_ib_uapi_dm_type type;
+	int err;
+
+	err = uverbs_get_const_default(&type, attrs,
+				       MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
+				       MLX5_IB_UAPI_DM_TYPE_MEMIC);
+	if (err)
+		return ERR_PTR(err);
+
+	mlx5_ib_dbg(to_mdev(ibdev), "alloc_dm req: dm_type=%d user_length=0x%llx log_alignment=%d\n",
+		    type, attr->length, attr->alignment);
+
+	dm = kzalloc(sizeof(*dm), GFP_KERNEL);
+	if (!dm)
+		return ERR_PTR(-ENOMEM);
+
+	dm->type = type;
+
+	switch (type) {
+	case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+		err = handle_alloc_dm_memic(context, dm,
+					    attr,
+					    attrs);
+		break;
+	default:
+		err = -EOPNOTSUPP;
+	}
 
-	dm->dev_addr = memic_addr;
+	if (err)
+		goto err_free;
 
 	return &dm->ibdm;
 
-err_dealloc:
-	mlx5_cmd_dealloc_memic(memic, memic_addr,
-			       act_size);
 err_free:
 	kfree(dm);
 	return ERR_PTR(err);
@@ -2350,22 +2382,29 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
 
 int mlx5_ib_dealloc_dm(struct ib_dm *ibdm)
 {
-	struct mlx5_memic *memic = &to_mdev(ibdm->device)->memic;
+	struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;
 	struct mlx5_ib_dm *dm = to_mdm(ibdm);
-	u64 act_size = roundup(dm->ibdm.length, MLX5_MEMIC_BASE_SIZE);
 	u32 page_idx;
 	int ret;
 
-	ret = mlx5_cmd_dealloc_memic(memic, dm->dev_addr, act_size);
-	if (ret)
-		return ret;
+	switch (dm->type) {
+	case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+		ret = mlx5_cmd_dealloc_memic(dm_db, dm->dev_addr, dm->size);
+		if (ret)
+			return ret;
 
-	page_idx = (dm->dev_addr - pci_resource_start(memic->dev->pdev, 0) -
-		    MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >>
-		    PAGE_SHIFT;
-	bitmap_clear(to_mucontext(ibdm->uobject->context)->dm_pages,
-		     page_idx,
-		     DIV_ROUND_UP(act_size, PAGE_SIZE));
+		page_idx = (dm->dev_addr -
+			    pci_resource_start(dm_db->dev->pdev, 0) -
+			    MLX5_CAP64_DEV_MEM(dm_db->dev,
+					       memic_bar_start_addr)) >>
+			   PAGE_SHIFT;
+		bitmap_clear(to_mucontext(ibdm->uobject->context)->dm_pages,
+			     page_idx,
+			     DIV_ROUND_UP(dm->size, PAGE_SIZE));
+		break;
+	default:
+		return -EOPNOTSUPP;
+	}
 
 	kfree(dm);
 
@@ -5865,7 +5904,10 @@ ADD_UVERBS_ATTRIBUTES_SIMPLE(
 			    UA_MANDATORY),
 	UVERBS_ATTR_PTR_OUT(MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
 			    UVERBS_ATTR_TYPE(u16),
-			    UA_MANDATORY));
+			    UA_OPTIONAL),
+	UVERBS_ATTR_CONST_IN(MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
+			     enum mlx5_ib_uapi_dm_type,
+			     UA_OPTIONAL));
 
 ADD_UVERBS_ATTRIBUTES_SIMPLE(
 	mlx5_ib_flow_action,
@@ -6013,8 +6055,8 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 	INIT_LIST_HEAD(&dev->qp_list);
 	spin_lock_init(&dev->reset_flow_resource_lock);
 
-	spin_lock_init(&dev->memic.memic_lock);
-	dev->memic.dev = mdev;
+	spin_lock_init(&dev->dm.lock);
+	dev->dm.dev = mdev;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		err = init_srcu_struct(&dev->mr_srcu);
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 2510e5dd5dc1..714ff948bea7 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -48,6 +48,7 @@
 #include <rdma/mlx5-abi.h>
 #include <rdma/uverbs_ioctl.h>
 #include <rdma/mlx5_user_ioctl_cmds.h>
+#include <rdma/mlx5_user_ioctl_verbs.h>
 
 #include "srq.h"
 
@@ -558,15 +559,17 @@ enum mlx5_ib_mtt_access_flags {
 struct mlx5_ib_dm {
 	struct ib_dm		ibdm;
 	phys_addr_t		dev_addr;
+	u32			type;
+	size_t			size;
 };
 
 #define MLX5_IB_MTT_PRESENT (MLX5_IB_MTT_READ | MLX5_IB_MTT_WRITE)
 
-#define MLX5_IB_DM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE   |\
-				   IB_ACCESS_REMOTE_WRITE  |\
-				   IB_ACCESS_REMOTE_READ   |\
-				   IB_ACCESS_REMOTE_ATOMIC |\
-				   IB_ZERO_BASED)
+#define MLX5_IB_DM_MEMIC_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE   |\
+					 IB_ACCESS_REMOTE_WRITE  |\
+					 IB_ACCESS_REMOTE_READ   |\
+					 IB_ACCESS_REMOTE_ATOMIC |\
+					 IB_ZERO_BASED)
 
 struct mlx5_ib_mr {
 	struct ib_mr		ibmr;
@@ -847,9 +850,13 @@ struct mlx5_ib_flow_action {
 	};
 };
 
-struct mlx5_memic {
+struct mlx5_dm {
 	struct mlx5_core_dev *dev;
-	spinlock_t		memic_lock;
+	/* This lock is used to protect the access to the shared
+	 * allocation map when concurrent requests by different
+	 * processes are handled.
+	 */
+	spinlock_t lock;
 	DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);
 };
 
@@ -953,7 +960,7 @@ struct mlx5_ib_dev {
 	u8			umr_fence;
 	struct list_head	ib_dev_list;
 	u64			sys_image_guid;
-	struct mlx5_memic	memic;
+	struct mlx5_dm		dm;
 	u16			devx_whitelist_uid;
 	struct mlx5_srq_table   srq_table;
 	struct mlx5_async_ctx   async_ctx;
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 02119fe8338f..7c54eacd9574 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1159,8 +1159,8 @@ static void set_mr_fields(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr,
 	mr->access_flags = access_flags;
 }
 
-static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr,
-					  u64 length, int acc)
+static struct ib_mr *mlx5_ib_get_dm_mr(struct ib_pd *pd, u64 start_addr,
+				       u64 length, int acc, int mode)
 {
 	struct mlx5_ib_dev *dev = to_mdev(pd->device);
 	int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
@@ -1182,9 +1182,8 @@ static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr,
 
 	mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
 
-	MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_MEMIC & 0x3);
-	MLX5_SET(mkc, mkc, access_mode_4_2,
-		 (MLX5_MKC_ACCESS_MODE_MEMIC >> 2) & 0x7);
+	MLX5_SET(mkc, mkc, access_mode_1_0, mode & 0x3);
+	MLX5_SET(mkc, mkc, access_mode_4_2, (mode >> 2) & 0x7);
 	MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
 	MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
 	MLX5_SET(mkc, mkc, rr, !!(acc & IB_ACCESS_REMOTE_READ));
@@ -1194,8 +1193,7 @@ static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr,
 	MLX5_SET64(mkc, mkc, len, length);
 	MLX5_SET(mkc, mkc, pd, to_mpd(pd)->pdn);
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
-	MLX5_SET64(mkc, mkc, start_addr,
-		   memic_addr - pci_resource_start(dev->mdev->pdev, 0));
+	MLX5_SET64(mkc, mkc, start_addr, start_addr);
 
 	err = mlx5_core_create_mkey(mdev, &mr->mmkey, in, inlen);
 	if (err)
@@ -1237,15 +1235,24 @@ struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm,
 				struct uverbs_attr_bundle *attrs)
 {
 	struct mlx5_ib_dm *mdm = to_mdm(dm);
-	u64 memic_addr;
+	struct mlx5_core_dev *dev = to_mdev(dm->device)->mdev;
+	u64 start_addr = mdm->dev_addr + attr->offset;
+	int mode;
 
-	if (attr->access_flags & ~MLX5_IB_DM_ALLOWED_ACCESS)
-		return ERR_PTR(-EINVAL);
+	switch (mdm->type) {
+	case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+		if (attr->access_flags & ~MLX5_IB_DM_MEMIC_ALLOWED_ACCESS)
+			return ERR_PTR(-EINVAL);
 
-	memic_addr = mdm->dev_addr + attr->offset;
+		mode = MLX5_MKC_ACCESS_MODE_MEMIC;
+		start_addr -= pci_resource_start(dev->pdev, 0);
+		break;
+	default:
+		return ERR_PTR(-EINVAL);
+	}
 
-	return mlx5_ib_get_memic_mr(pd, memic_addr, attr->length,
-				    attr->access_flags);
+	return mlx5_ib_get_dm_mr(pd, start_addr, attr->length,
+				 attr->access_flags, mode);
 }
 
 struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
diff --git a/include/uapi/rdma/mlx5_user_ioctl_cmds.h b/include/uapi/rdma/mlx5_user_ioctl_cmds.h
index 0d8f564ce60b..d404c951954c 100644
--- a/include/uapi/rdma/mlx5_user_ioctl_cmds.h
+++ b/include/uapi/rdma/mlx5_user_ioctl_cmds.h
@@ -44,6 +44,7 @@ enum mlx5_ib_create_flow_action_attrs {
 enum mlx5_ib_alloc_dm_attrs {
 	MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET = (1U << UVERBS_ID_NS_SHIFT),
 	MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
+	MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
 };
 
 enum mlx5_ib_devx_methods {
diff --git a/include/uapi/rdma/mlx5_user_ioctl_verbs.h b/include/uapi/rdma/mlx5_user_ioctl_verbs.h
index 0a126a6b9337..c291fb2f8446 100644
--- a/include/uapi/rdma/mlx5_user_ioctl_verbs.h
+++ b/include/uapi/rdma/mlx5_user_ioctl_verbs.h
@@ -57,5 +57,9 @@ struct mlx5_ib_uapi_devx_async_cmd_hdr {
 	__u8		out_data[];
 };
 
+enum mlx5_ib_uapi_dm_type {
+	MLX5_IB_UAPI_DM_TYPE_MEMIC,
+};
+
 #endif
 
-- 
2.20.1

[PATCH rdma-next 3/8] IB/mlx5: Warn on allocated MEMIC buffers during cleanup

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

Adding a warning on allocated MEMIC buffers that weren't
freed prior to driver tear down.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/main.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 562826ecfbdd..23436feaa22f 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -6005,6 +6005,8 @@ static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
 		srcu_barrier(&dev->mr_srcu);
 		cleanup_srcu_struct(&dev->mr_srcu);
 	}
+
+	WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));
 }
 
 static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
-- 
2.20.1

[PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

This patch adds support for allocating, deallocating and registering
a new device memory type, STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct
rule insertion and management of the device's steering tables.

The type is provided by the user via the dedicated attribute in
the alloc_dm ioctl command.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
 drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
 drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
 drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
 drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
 drivers/infiniband/hw/mlx5/mr.c           |   7 ++
 include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
 7 files changed, 291 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
index 8d651c05de62..347d457fcb2f 100644
--- a/drivers/infiniband/hw/mlx5/Kconfig
+++ b/drivers/infiniband/hw/mlx5/Kconfig
@@ -1,6 +1,6 @@
 config MLX5_INFINIBAND
 	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
-	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
+	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
 	---help---
 	  This driver provides low-level InfiniBand support for
 	  Mellanox Connect-IB PCI Express host channel adapters (HCAs).
diff --git a/drivers/infiniband/hw/mlx5/cmd.c b/drivers/infiniband/hw/mlx5/cmd.c
index a405d81ba01a..d3d8cd670493 100644
--- a/drivers/infiniband/hw/mlx5/cmd.c
+++ b/drivers/infiniband/hw/mlx5/cmd.c
@@ -157,7 +157,7 @@ int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
 	return -ENOMEM;
 }
 
-int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, u64 addr, u64 length)
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length)
 {
 	struct mlx5_core_dev *dev = dm->dev;
 	u64 hw_start_addr = MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr);
@@ -186,6 +186,131 @@ int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, u64 addr, u64 length)
 	return err;
 }
 
+int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+			  u16 uid, phys_addr_t *addr, u32 *obj_id)
+{
+	struct mlx5_core_dev *dev = dm->dev;
+	u32 num_blocks = DIV_ROUND_UP(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
+	u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
+	unsigned long *block_map;
+	u64 icm_start_addr;
+	u32 log_icm_size;
+	u32 max_blocks;
+	u64 block_idx;
+	void *sw_icm;
+	int ret;
+
+	MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+		 MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
+	MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
+
+	switch (type) {
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+		icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
+						steering_sw_icm_start_address);
+		log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
+		block_map = dm->steering_sw_icm_alloc_blocks;
+		break;
+	case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+		icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
+					header_modify_sw_icm_start_address);
+		log_icm_size = MLX5_CAP_DEV_MEM(dev,
+						log_header_modify_sw_icm_size);
+		block_map = dm->header_modify_sw_icm_alloc_blocks;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
+	spin_lock(&dm->lock);
+	block_idx = bitmap_find_next_zero_area(block_map,
+					       max_blocks,
+					       0,
+					       num_blocks, 0);
+
+	if (block_idx < max_blocks)
+		bitmap_set(block_map,
+			   block_idx, num_blocks);
+
+	spin_unlock(&dm->lock);
+
+	if (block_idx >= max_blocks)
+		return -ENOMEM;
+
+	sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
+	icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+	MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
+		   icm_start_addr);
+	MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
+
+	ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+	if (ret) {
+		spin_lock(&dm->lock);
+		bitmap_clear(block_map,
+			     block_idx, num_blocks);
+		spin_unlock(&dm->lock);
+
+		return ret;
+	}
+
+	*addr = icm_start_addr;
+	*obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+
+	return 0;
+}
+
+int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+			    u16 uid, phys_addr_t addr, u32 obj_id)
+{
+	struct mlx5_core_dev *dev = dm->dev;
+	u32 num_blocks = DIV_ROUND_UP(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
+	u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
+	u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
+	unsigned long *block_map;
+	u64 start_idx;
+	int err;
+
+	switch (type) {
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+		start_idx =
+			(addr - MLX5_CAP64_DEV_MEM(
+					dev, steering_sw_icm_start_address)) >>
+			MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+		block_map = dm->steering_sw_icm_alloc_blocks;
+		break;
+	case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+		start_idx =
+			(addr -
+			 MLX5_CAP64_DEV_MEM(
+				 dev, header_modify_sw_icm_start_address)) >>
+			MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+		block_map = dm->header_modify_sw_icm_alloc_blocks;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+		 MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
+	MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
+	MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
+
+	err =  mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+	if (err)
+		return err;
+
+	spin_lock(&dm->lock);
+	bitmap_clear(block_map,
+		     start_idx, num_blocks);
+	spin_unlock(&dm->lock);
+
+	return 0;
+}
+
 int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out)
 {
 	u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {};
diff --git a/drivers/infiniband/hw/mlx5/cmd.h b/drivers/infiniband/hw/mlx5/cmd.h
index 80a644bea6c7..0572dcba6eae 100644
--- a/drivers/infiniband/hw/mlx5/cmd.h
+++ b/drivers/infiniband/hw/mlx5/cmd.h
@@ -46,7 +46,7 @@ int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *mdev,
 				void *in, int in_size);
 int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
 			 u64 length, u32 alignment);
-int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, u64 addr, u64 length);
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length);
 void mlx5_cmd_dealloc_pd(struct mlx5_core_dev *dev, u32 pdn, u16 uid);
 void mlx5_cmd_destroy_tir(struct mlx5_core_dev *dev, u32 tirn, u16 uid);
 void mlx5_cmd_destroy_tis(struct mlx5_core_dev *dev, u32 tisn, u16 uid);
@@ -65,4 +65,8 @@ int mlx5_cmd_alloc_q_counter(struct mlx5_core_dev *dev, u16 *counter_id,
 			     u16 uid);
 int mlx5_cmd_mad_ifc(struct mlx5_core_dev *dev, const void *inb, void *outb,
 		     u16 opmod, u8 port);
+int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+			  u16 uid, phys_addr_t *addr, u32 *obj_id);
+int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+			    u16 uid, phys_addr_t addr, u32 obj_id);
 #endif /* MLX5_IB_CMD_H */
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 23436feaa22f..3806a3e7caf5 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -2291,6 +2291,28 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm
 	return 0;
 }
 
+static inline int check_dm_type_support(struct mlx5_ib_dev *dev,
+					u32 type)
+{
+	switch (type) {
+	case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+		if (!MLX5_CAP_DEV_MEM(dev->mdev, memic))
+			return -EOPNOTSUPP;
+		break;
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+		if (!capable(CAP_SYS_RAWIO) ||
+		    !capable(CAP_NET_RAW))
+			return -EPERM;
+
+		if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
+		      MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner)))
+			return -EOPNOTSUPP;
+		break;
+	}
+
+	return 0;
+}
+
 static int handle_alloc_dm_memic(struct ib_ucontext *ctx,
 				 struct mlx5_ib_dm *dm,
 				 struct ib_dm_alloc_attr *attr,
@@ -2336,6 +2358,40 @@ static int handle_alloc_dm_memic(struct ib_ucontext *ctx,
 	return err;
 }
 
+static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx,
+				  struct mlx5_ib_dm *dm,
+				  struct ib_dm_alloc_attr *attr,
+				  struct uverbs_attr_bundle *attrs,
+				  int type)
+{
+	struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;
+	u64 act_size;
+	int err;
+
+	/* Allocation size must a multiple of the basic block size
+	 * and a power of 2.
+	 */
+	act_size = roundup(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dm_db->dev));
+	act_size = roundup_pow_of_two(act_size);
+
+	dm->size = act_size;
+	err = mlx5_cmd_alloc_sw_icm(dm_db, type, act_size,
+				    to_mucontext(ctx)->devx_uid, &dm->dev_addr,
+				    &dm->icm_dm.obj_id);
+	if (err)
+		return err;
+
+	err = uverbs_copy_to(attrs,
+			     MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
+			     &dm->dev_addr, sizeof(dm->dev_addr));
+	if (err)
+		mlx5_cmd_dealloc_sw_icm(dm_db, type, dm->size,
+					to_mucontext(ctx)->devx_uid,
+					dm->dev_addr, dm->icm_dm.obj_id);
+
+	return err;
+}
+
 struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
 			       struct ib_ucontext *context,
 			       struct ib_dm_alloc_attr *attr,
@@ -2354,6 +2410,10 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
 	mlx5_ib_dbg(to_mdev(ibdev), "alloc_dm req: dm_type=%d user_length=0x%llx log_alignment=%d\n",
 		    type, attr->length, attr->alignment);
 
+	err = check_dm_type_support(to_mdev(ibdev), type);
+	if (err)
+		return ERR_PTR(err);
+
 	dm = kzalloc(sizeof(*dm), GFP_KERNEL);
 	if (!dm)
 		return ERR_PTR(-ENOMEM);
@@ -2366,6 +2426,10 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
 					    attr,
 					    attrs);
 		break;
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+	case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+		err = handle_alloc_dm_sw_icm(context, dm, attr, attrs, type);
+		break;
 	default:
 		err = -EOPNOTSUPP;
 	}
@@ -2382,6 +2446,7 @@ struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
 
 int mlx5_ib_dealloc_dm(struct ib_dm *ibdm)
 {
+	struct mlx5_ib_ucontext *ctx = to_mucontext(ibdm->uobject->context);
 	struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;
 	struct mlx5_ib_dm *dm = to_mdm(ibdm);
 	u32 page_idx;
@@ -2398,10 +2463,18 @@ int mlx5_ib_dealloc_dm(struct ib_dm *ibdm)
 			    MLX5_CAP64_DEV_MEM(dm_db->dev,
 					       memic_bar_start_addr)) >>
 			   PAGE_SHIFT;
-		bitmap_clear(to_mucontext(ibdm->uobject->context)->dm_pages,
+		bitmap_clear(ctx->dm_pages,
 			     page_idx,
 			     DIV_ROUND_UP(dm->size, PAGE_SIZE));
 		break;
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+	case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+		ret = mlx5_cmd_dealloc_sw_icm(dm_db, dm->type, dm->size,
+					      ctx->devx_uid, dm->dev_addr,
+					      dm->icm_dm.obj_id);
+		if (ret)
+			return ret;
+		break;
 	default:
 		return -EOPNOTSUPP;
 	}
@@ -6000,6 +6073,8 @@ static struct ib_counters *mlx5_ib_create_counters(struct ib_device *device,
 
 static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
 {
+	struct mlx5_core_dev *mdev = dev->mdev;
+
 	mlx5_ib_cleanup_multiport_master(dev);
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		srcu_barrier(&dev->mr_srcu);
@@ -6007,11 +6082,29 @@ static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
 	}
 
 	WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));
+
+	WARN_ON(dev->dm.steering_sw_icm_alloc_blocks &&
+		!bitmap_empty(
+			dev->dm.steering_sw_icm_alloc_blocks,
+			BIT(MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size) -
+			    MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
+
+	kfree(dev->dm.steering_sw_icm_alloc_blocks);
+
+	WARN_ON(dev->dm.header_modify_sw_icm_alloc_blocks &&
+		!bitmap_empty(dev->dm.header_modify_sw_icm_alloc_blocks,
+			      BIT(MLX5_CAP_DEV_MEM(
+					  mdev, log_header_modify_sw_icm_size) -
+				  MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
+
+	kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
 }
 
 static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 {
 	struct mlx5_core_dev *mdev = dev->mdev;
+	u64 header_modify_icm_blocks = 0;
+	u64 steering_icm_blocks = 0;
 	int err;
 	int i;
 
@@ -6057,16 +6150,51 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 	INIT_LIST_HEAD(&dev->qp_list);
 	spin_lock_init(&dev->reset_flow_resource_lock);
 
+	if (MLX5_CAP_GEN_64(mdev, general_obj_types) &
+	    MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM) {
+		if (MLX5_CAP64_DEV_MEM(mdev, steering_sw_icm_start_address)) {
+			steering_icm_blocks =
+				BIT(MLX5_CAP_DEV_MEM(mdev,
+						     log_steering_sw_icm_size) -
+				    MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
+
+			dev->dm.steering_sw_icm_alloc_blocks =
+				kcalloc(BITS_TO_LONGS(steering_icm_blocks),
+					sizeof(unsigned long), GFP_KERNEL);
+			if (!dev->dm.steering_sw_icm_alloc_blocks)
+				goto err_mp;
+		}
+
+		if (MLX5_CAP64_DEV_MEM(mdev,
+				       header_modify_sw_icm_start_address)) {
+			header_modify_icm_blocks = BIT(
+				MLX5_CAP_DEV_MEM(
+					mdev, log_header_modify_sw_icm_size) -
+				MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
+
+			dev->dm.header_modify_sw_icm_alloc_blocks =
+				kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
+					sizeof(unsigned long), GFP_KERNEL);
+			if (!dev->dm.header_modify_sw_icm_alloc_blocks)
+				goto err_dm;
+		}
+	}
+
 	spin_lock_init(&dev->dm.lock);
 	dev->dm.dev = mdev;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
 		err = init_srcu_struct(&dev->mr_srcu);
 		if (err)
-			goto err_mp;
+			goto err_dm;
 	}
 
 	return 0;
+
+err_dm:
+	kfree(dev->dm.steering_sw_icm_alloc_blocks);
+	kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
+
 err_mp:
 	mlx5_ib_cleanup_multiport_master(dev);
 
@@ -6249,7 +6377,9 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
 		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_xrc_ops);
 	}
 
-	if (MLX5_CAP_DEV_MEM(mdev, memic))
+	if (MLX5_CAP_DEV_MEM(mdev, memic) ||
+	    MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
+	    MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM)
 		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_dm_ops);
 
 	if (mlx5_accel_ipsec_device_caps(dev->mdev) &
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 714ff948bea7..ac298f2c87a7 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -118,6 +118,10 @@ enum {
 	MLX5_MEMIC_BASE_SIZE	= 1 << MLX5_MEMIC_BASE_ALIGN,
 };
 
+#define MLX5_LOG_SW_ICM_BLOCK_SIZE(dev)                                        \
+	(MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity))
+#define MLX5_SW_ICM_BLOCK_SIZE(dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))
+
 struct mlx5_ib_ucontext {
 	struct ib_ucontext	ibucontext;
 	struct list_head	db_page_list;
@@ -561,6 +565,12 @@ struct mlx5_ib_dm {
 	phys_addr_t		dev_addr;
 	u32			type;
 	size_t			size;
+	union {
+		struct {
+			u32	obj_id;
+		} icm_dm;
+		/* other dm types specific params should be added here */
+	};
 };
 
 #define MLX5_IB_MTT_PRESENT (MLX5_IB_MTT_READ | MLX5_IB_MTT_WRITE)
@@ -571,6 +581,11 @@ struct mlx5_ib_dm {
 					 IB_ACCESS_REMOTE_ATOMIC |\
 					 IB_ZERO_BASED)
 
+#define MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE   |\
+					  IB_ACCESS_REMOTE_WRITE  |\
+					  IB_ACCESS_REMOTE_READ   |\
+					  IB_ZERO_BASED)
+
 struct mlx5_ib_mr {
 	struct ib_mr		ibmr;
 	void			*descs;
@@ -858,6 +873,8 @@ struct mlx5_dm {
 	 */
 	spinlock_t lock;
 	DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);
+	unsigned long *steering_sw_icm_alloc_blocks;
+	unsigned long *header_modify_sw_icm_alloc_blocks;
 };
 
 struct mlx5_read_counters_attr {
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 7c54eacd9574..f6ff6d6968db 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1247,6 +1247,13 @@ struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm,
 		mode = MLX5_MKC_ACCESS_MODE_MEMIC;
 		start_addr -= pci_resource_start(dev->pdev, 0);
 		break;
+	case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+	case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+		if (attr->access_flags & ~MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS)
+			return ERR_PTR(-EINVAL);
+
+		mode = MLX5_MKC_ACCESS_MODE_SW_ICM;
+		break;
 	default:
 		return ERR_PTR(-EINVAL);
 	}
diff --git a/include/uapi/rdma/mlx5_user_ioctl_verbs.h b/include/uapi/rdma/mlx5_user_ioctl_verbs.h
index c291fb2f8446..a8f34c237458 100644
--- a/include/uapi/rdma/mlx5_user_ioctl_verbs.h
+++ b/include/uapi/rdma/mlx5_user_ioctl_verbs.h
@@ -59,6 +59,8 @@ struct mlx5_ib_uapi_devx_async_cmd_hdr {
 
 enum mlx5_ib_uapi_dm_type {
 	MLX5_IB_UAPI_DM_TYPE_MEMIC,
+	MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM,
+	MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM,
 };
 
 #endif
-- 
2.20.1

[PATCH rdma-next 5/8] IB/mlx5: Device resource control for privileged DEVX user

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

For DEVX users who have SYS_RAWIO capability, we set the
internal device resources capability when creating the UCTX.
This will allow the device to restrict the allocation of internal
device resources such as SW ICM memory to privileged DEVX users
only.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/devx.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c
index d9d51a927b67..d39a3982ac51 100644
--- a/drivers/infiniband/hw/mlx5/devx.c
+++ b/drivers/infiniband/hw/mlx5/devx.c
@@ -85,6 +85,10 @@ int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user)
 	if (is_user && capable(CAP_NET_RAW) &&
 	    (MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RAW_TX))
 		cap |= MLX5_UCTX_CAP_RAW_TX;
+	if (is_user && capable(CAP_SYS_RAWIO) &&
+	    (MLX5_CAP_GEN(dev->mdev, uctx_cap) &
+	     MLX5_UCTX_CAP_INTERNAL_DEV_RES))
+		cap |= MLX5_UCTX_CAP_INTERNAL_DEV_RES;
 
 	MLX5_SET(create_uctx_in, in, opcode, MLX5_CMD_OP_CREATE_UCTX);
 	MLX5_SET(uctx, uctx, cap, cap);
-- 
2.20.1

[PATCH mlx5-next 6/8] net/mlx5: Expose TIR ICM address in command outbox

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

Adding the TIR ICM address to the create_tir command outbox
through which the device reports the ICM address of the newly
created TIR.

The TIR address can be used for direct attachment to a steering
rule in SW managed steering mode.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 include/linux/mlx5/mlx5_ifc.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index c2a558d58067..688b6172cbd5 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -6899,14 +6899,14 @@ struct mlx5_ifc_create_tis_in_bits {
 
 struct mlx5_ifc_create_tir_out_bits {
 	u8         status[0x8];
-	u8         reserved_at_8[0x18];
+	u8         icm_address_63_40[0x18];
 
 	u8         syndrome[0x20];
 
-	u8         reserved_at_40[0x8];
+	u8         icm_address_39_32[0x8];
 	u8         tirn[0x18];
 
-	u8         reserved_at_60[0x20];
+	u8         icm_address_31_0[0x20];
 };
 
 struct mlx5_ifc_create_tir_in_bits {
-- 
2.20.1

[PATCH mlx5-next 7/8] net/mlx5: Introduce new TIR creation core API

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

Introducing new TIR creation core API which allows caller
to receive back from the call the full command outbox.

This comes as a preparation for the next patch that will
retrieve the TIR ICM address from the command outbox.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Reviewed-by: Eli Cohen <eli@mellanox.com>
Reviewed-by: Mark Bloch <markb@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/transobj.c | 18 +++++++++++++-----
 include/linux/mlx5/transobj.h                  |  3 +++
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/transobj.c b/drivers/net/ethernet/mellanox/mlx5/core/transobj.c
index c4d4b76096dc..b1068500f1df 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/transobj.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/transobj.c
@@ -182,16 +182,24 @@ int mlx5_core_query_sq_state(struct mlx5_core_dev *dev, u32 sqn, u8 *state)
 }
 EXPORT_SYMBOL_GPL(mlx5_core_query_sq_state);
 
+int mlx5_core_create_tir_out(struct mlx5_core_dev *dev,
+			     u32 *in, int inlen,
+			     u32 *out, int outlen)
+{
+	MLX5_SET(create_tir_in, in, opcode, MLX5_CMD_OP_CREATE_TIR);
+
+	return mlx5_cmd_exec(dev, in, inlen, out, outlen);
+}
+EXPORT_SYMBOL(mlx5_core_create_tir_out);
+
 int mlx5_core_create_tir(struct mlx5_core_dev *dev, u32 *in, int inlen,
 			 u32 *tirn)
 {
-	u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {0};
+	u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {};
 	int err;
 
-	MLX5_SET(create_tir_in, in, opcode, MLX5_CMD_OP_CREATE_TIR);
-
-	memset(out, 0, sizeof(out));
-	err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
+	err = mlx5_core_create_tir_out(dev, in, inlen,
+				       out, sizeof(out));
 	if (!err)
 		*tirn = MLX5_GET(create_tir_out, out, tirn);
 
diff --git a/include/linux/mlx5/transobj.h b/include/linux/mlx5/transobj.h
index a261d5528ff7..dc6b1e7cb8c4 100644
--- a/include/linux/mlx5/transobj.h
+++ b/include/linux/mlx5/transobj.h
@@ -50,6 +50,9 @@ int mlx5_core_query_sq(struct mlx5_core_dev *dev, u32 sqn, u32 *out);
 int mlx5_core_query_sq_state(struct mlx5_core_dev *dev, u32 sqn, u8 *state);
 int mlx5_core_create_tir(struct mlx5_core_dev *dev, u32 *in, int inlen,
 			 u32 *tirn);
+int mlx5_core_create_tir_out(struct mlx5_core_dev *dev,
+			     u32 *in, int inlen,
+			     u32 *out, int outlen);
 int mlx5_core_modify_tir(struct mlx5_core_dev *dev, u32 tirn, u32 *in,
 			 int inlen);
 void mlx5_core_destroy_tir(struct mlx5_core_dev *dev, u32 tirn);
-- 
2.20.1

[PATCH rdma-next 8/8] IB/mlx5: Expose TIR ICM address to user space

[Leon Romanovsky]

From: Ariel Levkovich <lariel@mellanox.com>

This patch exposes the TIR ICM address of raw packet and RSS
QPs to user space.

In order to pass the new field, the patch extends the mlx5
specific QP creation response structure and fills it with
the icm address returned by the FW command, if available.

Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 46 +++++++++++++++++++++++++++++----
 include/uapi/rdma/mlx5-abi.h    |  2 ++
 2 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 3ff7c32207b9..6b332b896c57 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -1397,7 +1397,8 @@ static void destroy_raw_packet_qp_tir(struct mlx5_ib_dev *dev,
 static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,
 				    struct mlx5_ib_rq *rq, u32 tdn,
 				    u32 *qp_flags_en,
-				    struct ib_pd *pd)
+				    struct ib_pd *pd,
+				    u32 *out, int outlen)
 {
 	u8 lb_flag = 0;
 	u32 *in;
@@ -1431,8 +1432,9 @@ static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,
 
 	MLX5_SET(tirc, tirc, self_lb_block, lb_flag);
 
-	err = mlx5_core_create_tir(dev->mdev, in, inlen, &rq->tirn);
+	err = mlx5_core_create_tir_out(dev->mdev, in, inlen, out, outlen);
 
+	rq->tirn = MLX5_GET(create_tir_out, out, tirn);
 	if (!err && MLX5_GET(tirc, tirc, self_lb_block)) {
 		err = mlx5_ib_enable_lb(dev, false, true);
 
@@ -1458,6 +1460,7 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	int err;
 	u32 tdn = mucontext->tdn;
 	u16 uid = to_mpd(pd)->uid;
+	u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {};
 
 	if (qp->sq.wqe_cnt) {
 		err = create_raw_packet_qp_tis(dev, qp, sq, tdn, pd);
@@ -1490,7 +1493,9 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		if (err)
 			goto err_destroy_sq;
 
-		err = create_raw_packet_qp_tir(dev, rq, tdn, &qp->flags_en, pd);
+		err = create_raw_packet_qp_tir(
+			dev, rq, tdn, &qp->flags_en, pd, out,
+			MLX5_ST_SZ_BYTES(create_tir_out));
 		if (err)
 			goto err_destroy_rq;
 
@@ -1499,6 +1504,20 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 			resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_RQN;
 			resp->tirn = rq->tirn;
 			resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
+			if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+				resp->tir_icm_addr = MLX5_GET(
+					create_tir_out, out, icm_address_31_0);
+				resp->tir_icm_addr |=
+					(u64)MLX5_GET(create_tir_out, out,
+						      icm_address_39_32)
+					<< 32;
+				resp->tir_icm_addr |=
+					(u64)MLX5_GET(create_tir_out, out,
+						      icm_address_63_40)
+					<< 40;
+				resp->comp_mask |=
+					MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR;
+			}
 		}
 	}
 
@@ -1572,8 +1591,10 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 		udata, struct mlx5_ib_ucontext, ibucontext);
 	struct mlx5_ib_create_qp_resp resp = {};
 	int inlen;
+	int outlen;
 	int err;
 	u32 *in;
+	u32 *out;
 	void *tirc;
 	void *hfso;
 	u32 selected_fields = 0;
@@ -1653,10 +1674,12 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	}
 
 	inlen = MLX5_ST_SZ_BYTES(create_tir_in);
-	in = kvzalloc(inlen, GFP_KERNEL);
+	outlen = MLX5_ST_SZ_BYTES(create_tir_out);
+	in = kvzalloc(inlen + outlen, GFP_KERNEL);
 	if (!in)
 		return -ENOMEM;
 
+	out = in + MLX5_ST_SZ_DW(create_tir_in);
 	MLX5_SET(create_tir_in, in, uid, to_mpd(pd)->uid);
 	tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
 	MLX5_SET(tirc, tirc, disp_type,
@@ -1768,8 +1791,9 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	MLX5_SET(rx_hash_field_select, hfso, selected_fields, selected_fields);
 
 create_tir:
-	err = mlx5_core_create_tir(dev->mdev, in, inlen, &qp->rss_qp.tirn);
+	err = mlx5_core_create_tir_out(dev->mdev, in, inlen, out, outlen);
 
+	qp->rss_qp.tirn = MLX5_GET(create_tir_out, out, tirn);
 	if (!err && MLX5_GET(tirc, tirc, self_lb_block)) {
 		err = mlx5_ib_enable_lb(dev, false, true);
 
@@ -1784,6 +1808,18 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
 	if (mucontext->devx_uid) {
 		resp.comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
 		resp.tirn = qp->rss_qp.tirn;
+		if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+			resp.tir_icm_addr =
+				MLX5_GET(create_tir_out, out, icm_address_31_0);
+			resp.tir_icm_addr |= (u64)MLX5_GET(create_tir_out, out,
+							   icm_address_39_32)
+					     << 32;
+			resp.tir_icm_addr |= (u64)MLX5_GET(create_tir_out, out,
+							   icm_address_63_40)
+					     << 40;
+			resp.comp_mask |=
+				MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR;
+		}
 	}
 
 	err = ib_copy_to_udata(udata, &resp, min(udata->outlen, sizeof(resp)));
diff --git a/include/uapi/rdma/mlx5-abi.h b/include/uapi/rdma/mlx5-abi.h
index 87b3198f4b5d..0ef404f75f56 100644
--- a/include/uapi/rdma/mlx5-abi.h
+++ b/include/uapi/rdma/mlx5-abi.h
@@ -359,6 +359,7 @@ enum mlx5_ib_create_qp_resp_mask {
 	MLX5_IB_CREATE_QP_RESP_MASK_TISN = 1UL << 1,
 	MLX5_IB_CREATE_QP_RESP_MASK_RQN  = 1UL << 2,
 	MLX5_IB_CREATE_QP_RESP_MASK_SQN  = 1UL << 3,
+	MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR  = 1UL << 4,
 };
 
 struct mlx5_ib_create_qp_resp {
@@ -370,6 +371,7 @@ struct mlx5_ib_create_qp_resp {
 	__u32	rqn;
 	__u32	sqn;
 	__u32   reserved1;
+	__u64	tir_icm_addr;
 };
 
 struct mlx5_ib_alloc_mw {
-- 
2.20.1

Re: [PATCH rdma-next 0/8] User space steering

[Jakub Kicinski]

On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
> 
> From Ariel,
> 
> This series of patches adds user space managed steering infrastructure
> to the mlx5_ib driver.
> 
> User space managed steering requires the means to access a dedicated
> memory space that is used by the device to store the packet steering
> and header modification tables and rules in order to manage them directly
> without the device's firmware involvement. This dedicated memory is part
> of the ICM memory space.
> 
> The changes are introducing the mlx5_ib API to allocate, deallocate and
> register this dedicated SW ICM memory via the existing device memory API
> using a private attribute which specifies the memory type.
> 
> The allocated memory itself is not IO mapped and user can only access it
> using remote RDMA operations.
> 
> In addition, the series exposed the ICM address of the receive transport
> interface (TIR) of Raw Packet and RSS QPs to user since they are required
> to properly create and insert steering rules that direct flows to these QPs.

Hi Leon!  Out of curiosity what protocols are you guys steering here?
Is this some form of RoCE or IB?  Is there a good resource to read up on
the DEVX stuff? (I presume this series is related?)

Re: [PATCH rdma-next 0/8] User space steering

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote:
> On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@mellanox.com>
> >
> > From Ariel,
> >
> > This series of patches adds user space managed steering infrastructure
> > to the mlx5_ib driver.
> >
> > User space managed steering requires the means to access a dedicated
> > memory space that is used by the device to store the packet steering
> > and header modification tables and rules in order to manage them directly
> > without the device's firmware involvement. This dedicated memory is part
> > of the ICM memory space.
> >
> > The changes are introducing the mlx5_ib API to allocate, deallocate and
> > register this dedicated SW ICM memory via the existing device memory API
> > using a private attribute which specifies the memory type.
> >
> > The allocated memory itself is not IO mapped and user can only access it
> > using remote RDMA operations.
> >
> > In addition, the series exposed the ICM address of the receive transport
> > interface (TIR) of Raw Packet and RSS QPs to user since they are required
> > to properly create and insert steering rules that direct flows to these QPs.
>
> Hi Leon!  Out of curiosity what protocols are you guys steering here?
> Is this some form of RoCE or IB?  Is there a good resource to read up on
> the DEVX stuff? (I presume this series is related?)

mlx5 devices steering operates on packets without relation to specific
protocol. The decision to steer is based on HW capabilities and user
configuration for matching specific fields. Such configuration is done
through kernel and through DEVX.

General description of DEVX can be found in manuals of libmlx5.
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [PATCH rdma-next 0/8] User space steering

[Jakub Kicinski]

On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote:
> On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote:
> > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:  
> > > From: Leon Romanovsky <leonro@mellanox.com>
> > >
> > > From Ariel,
> > >
> > > This series of patches adds user space managed steering infrastructure
> > > to the mlx5_ib driver.
> > >
> > > User space managed steering requires the means to access a dedicated
> > > memory space that is used by the device to store the packet steering
> > > and header modification tables and rules in order to manage them directly
> > > without the device's firmware involvement. This dedicated memory is part
> > > of the ICM memory space.
> > >
> > > The changes are introducing the mlx5_ib API to allocate, deallocate and
> > > register this dedicated SW ICM memory via the existing device memory API
> > > using a private attribute which specifies the memory type.
> > >
> > > The allocated memory itself is not IO mapped and user can only access it
> > > using remote RDMA operations.
> > >
> > > In addition, the series exposed the ICM address of the receive transport
> > > interface (TIR) of Raw Packet and RSS QPs to user since they are required
> > > to properly create and insert steering rules that direct flows to these QPs.  
> >
> > Hi Leon!  Out of curiosity what protocols are you guys steering here?
> > Is this some form of RoCE or IB?  Is there a good resource to read up on
> > the DEVX stuff? (I presume this series is related?)  
> 
> mlx5 devices steering operates on packets without relation to specific
> protocol. The decision to steer is based on HW capabilities and user
> configuration for matching specific fields. Such configuration is done
> through kernel and through DEVX.
> 
> General description of DEVX can be found in manuals of libmlx5.
> https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
> https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/

Thanks for those, looks quite powerful.  What's your primary use case
here?  Is there some flow steering and encapsulation going on in RDMA
world?

Re: [PATCH rdma-next 0/8] User space steering

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2886 bytes --]

On Mon, Apr 01, 2019 at 11:32:21AM -0700, Jakub Kicinski wrote:
> On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote:
> > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote:
> > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@mellanox.com>
> > > >
> > > > From Ariel,
> > > >
> > > > This series of patches adds user space managed steering infrastructure
> > > > to the mlx5_ib driver.
> > > >
> > > > User space managed steering requires the means to access a dedicated
> > > > memory space that is used by the device to store the packet steering
> > > > and header modification tables and rules in order to manage them directly
> > > > without the device's firmware involvement. This dedicated memory is part
> > > > of the ICM memory space.
> > > >
> > > > The changes are introducing the mlx5_ib API to allocate, deallocate and
> > > > register this dedicated SW ICM memory via the existing device memory API
> > > > using a private attribute which specifies the memory type.
> > > >
> > > > The allocated memory itself is not IO mapped and user can only access it
> > > > using remote RDMA operations.
> > > >
> > > > In addition, the series exposed the ICM address of the receive transport
> > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required
> > > > to properly create and insert steering rules that direct flows to these QPs.
> > >
> > > Hi Leon!  Out of curiosity what protocols are you guys steering here?
> > > Is this some form of RoCE or IB?  Is there a good resource to read up on
> > > the DEVX stuff? (I presume this series is related?)
> >
> > mlx5 devices steering operates on packets without relation to specific
> > protocol. The decision to steer is based on HW capabilities and user
> > configuration for matching specific fields. Such configuration is done
> > through kernel and through DEVX.
> >
> > General description of DEVX can be found in manuals of libmlx5.
> > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
> > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/
>
> Thanks for those, looks quite powerful.  What's your primary use case
> here?  Is there some flow steering and encapsulation going on in RDMA
> world?

Usual use case for that is in RoCE fabric, but as I said it is not
limited to that. For example, flow steering is used in libpcap [1]
and encapsulation is used for efficient VM to VM communication over
different hosts (I assume exactly as in netdev world).

In regarding to this patch set, the combination of this steering
functionality with DEVX allows us to speedup greatly buildup of rules
by doing it in powerful host CPU instead of internal to NIC CPU.

[1]
https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [PATCH rdma-next 0/8] User space steering

[Jakub Kicinski]

On Tue, 2 Apr 2019 11:14:59 +0300, Leon Romanovsky wrote:
> On Mon, Apr 01, 2019 at 11:32:21AM -0700, Jakub Kicinski wrote:
> > On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote:  
> > > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote:  
> > > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:  
> > > > > From: Leon Romanovsky <leonro@mellanox.com>
> > > > >
> > > > > From Ariel,
> > > > >
> > > > > This series of patches adds user space managed steering infrastructure
> > > > > to the mlx5_ib driver.
> > > > >
> > > > > User space managed steering requires the means to access a dedicated
> > > > > memory space that is used by the device to store the packet steering
> > > > > and header modification tables and rules in order to manage them directly
> > > > > without the device's firmware involvement. This dedicated memory is part
> > > > > of the ICM memory space.
> > > > >
> > > > > The changes are introducing the mlx5_ib API to allocate, deallocate and
> > > > > register this dedicated SW ICM memory via the existing device memory API
> > > > > using a private attribute which specifies the memory type.
> > > > >
> > > > > The allocated memory itself is not IO mapped and user can only access it
> > > > > using remote RDMA operations.
> > > > >
> > > > > In addition, the series exposed the ICM address of the receive transport
> > > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required
> > > > > to properly create and insert steering rules that direct flows to these QPs.  
> > > >
> > > > Hi Leon!  Out of curiosity what protocols are you guys steering here?
> > > > Is this some form of RoCE or IB?  Is there a good resource to read up on
> > > > the DEVX stuff? (I presume this series is related?)  
> > >
> > > mlx5 devices steering operates on packets without relation to specific
> > > protocol. The decision to steer is based on HW capabilities and user
> > > configuration for matching specific fields. Such configuration is done
> > > through kernel and through DEVX.
> > >
> > > General description of DEVX can be found in manuals of libmlx5.
> > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
> > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/  
> >
> > Thanks for those, looks quite powerful.  What's your primary use case
> > here?  Is there some flow steering and encapsulation going on in RDMA
> > world?  
> 
> Usual use case for that is in RoCE fabric, but as I said it is not
> limited to that. For example, flow steering is used in libpcap [1]
> and encapsulation is used for efficient VM to VM communication over
> different hosts (I assume exactly as in netdev world).
> 
> In regarding to this patch set, the combination of this steering
> functionality with DEVX allows us to speedup greatly buildup of rules
> by doing it in powerful host CPU instead of internal to NIC CPU.
> 
> [1]
> https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42

Interesting, thanks for the info!

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> From: Ariel Levkovich <lariel@mellanox.com>
> 
> This patch adds support for allocating, deallocating and registering
> a new device memory type, STEERING_SW_ICM.
> This memory can be allocated and used by a privileged user for direct
> rule insertion and management of the device's steering tables.
> 
> The type is provided by the user via the dedicated attribute in
> the alloc_dm ioctl command.
> 
> Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> Reviewed-by: Eli Cohen <eli@mellanox.com>
> Reviewed-by: Mark Bloch <markb@mellanox.com>
> Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
>  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
>  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
>  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
>  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
>  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
>  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
>  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
>  7 files changed, 291 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> index 8d651c05de62..347d457fcb2f 100644
> +++ b/drivers/infiniband/hw/mlx5/Kconfig
> @@ -1,6 +1,6 @@
>  config MLX5_INFINIBAND
>  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT

Why do we need this?

Jason

Re: [PATCH rdma-next 0/8] User space steering

[Jason Gunthorpe]

On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
> 
> >From Ariel,
> 
> This series of patches adds user space managed steering infrastructure
> to the mlx5_ib driver.
> 
> User space managed steering requires the means to access a dedicated
> memory space that is used by the device to store the packet steering
> and header modification tables and rules in order to manage them directly
> without the device's firmware involvement. This dedicated memory is part
> of the ICM memory space.
> 
> The changes are introducing the mlx5_ib API to allocate, deallocate and
> register this dedicated SW ICM memory via the existing device memory API
> using a private attribute which specifies the memory type.
> 
> The allocated memory itself is not IO mapped and user can only access it
> using remote RDMA operations.
> 
> In addition, the series exposed the ICM address of the receive transport
> interface (TIR) of Raw Packet and RSS QPs to user since they are required
> to properly create and insert steering rules that direct flows to these QPs.
> 
> Thanks
> 
> Ariel Levkovich (8):
>   net/mlx5: Expose SW ICM related device memory capabilities
>   IB/mlx5: Support device memory type attribute
>   IB/mlx5: Warn on allocated MEMIC buffers during cleanup
>   IB/mlx5: Add steering SW ICM device memory type
>   IB/mlx5: Device resource control for privileged DEVX user

This doesn't apply, it conflicts with patches in mlx5-next, please
resent

>   net/mlx5: Expose TIR ICM address in command outbox
>   net/mlx5: Introduce new TIR creation core API
>   IB/mlx5: Expose TIR ICM address to user space

Is this actually two patch series? This one looks OK, lets apply it to
mlx5-next please

Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > From: Ariel Levkovich <lariel@mellanox.com>
> >
> > This patch adds support for allocating, deallocating and registering
> > a new device memory type, STEERING_SW_ICM.
> > This memory can be allocated and used by a privileged user for direct
> > rule insertion and management of the device's steering tables.
> >
> > The type is provided by the user via the dedicated attribute in
> > the alloc_dm ioctl command.
> >
> > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> >  7 files changed, 291 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > index 8d651c05de62..347d457fcb2f 100644
> > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > @@ -1,6 +1,6 @@
> >  config MLX5_INFINIBAND
> >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
>
> Why do we need this?

We are using phys_addr_t type and not u64 as an input to FW to emphasize
the caller interface, but it is different between 32 and 64 bits systems.
Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.

Thanks

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > From: Ariel Levkovich <lariel@mellanox.com>
> > >
> > > This patch adds support for allocating, deallocating and registering
> > > a new device memory type, STEERING_SW_ICM.
> > > This memory can be allocated and used by a privileged user for direct
> > > rule insertion and management of the device's steering tables.
> > >
> > > The type is provided by the user via the dedicated attribute in
> > > the alloc_dm ioctl command.
> > >
> > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > index 8d651c05de62..347d457fcb2f 100644
> > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > @@ -1,6 +1,6 @@
> > >  config MLX5_INFINIBAND
> > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> >
> > Why do we need this?
> 
> We are using phys_addr_t type and not u64 as an input to FW to emphasize
> the caller interface, but it is different between 32 and 64 bits systems.
> Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.

???

PHYS_ADDR_T should only ever be used to store a bus address - and if
you are passing a bus address to/from FW then you 0 extend it on 32
bit, and range check on 64 bit.

This is not a reason to force PHYS_ADDR_T_64BIT from a driver

Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > >
> > > > This patch adds support for allocating, deallocating and registering
> > > > a new device memory type, STEERING_SW_ICM.
> > > > This memory can be allocated and used by a privileged user for direct
> > > > rule insertion and management of the device's steering tables.
> > > >
> > > > The type is provided by the user via the dedicated attribute in
> > > > the alloc_dm ioctl command.
> > > >
> > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > index 8d651c05de62..347d457fcb2f 100644
> > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > @@ -1,6 +1,6 @@
> > > >  config MLX5_INFINIBAND
> > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > >
> > > Why do we need this?
> >
> > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > the caller interface, but it is different between 32 and 64 bits systems.
> > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
>
> ???
>
> PHYS_ADDR_T should only ever be used to store a bus address - and if
> you are passing a bus address to/from FW then you 0 extend it on 32
> bit, and range check on 64 bit.
>
> This is not a reason to force PHYS_ADDR_T_64BIT from a driver

So how will I mark that we are forwarding physical address to FW?

Thanks

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > >
> > > > > This patch adds support for allocating, deallocating and registering
> > > > > a new device memory type, STEERING_SW_ICM.
> > > > > This memory can be allocated and used by a privileged user for direct
> > > > > rule insertion and management of the device's steering tables.
> > > > >
> > > > > The type is provided by the user via the dedicated attribute in
> > > > > the alloc_dm ioctl command.
> > > > >
> > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > @@ -1,6 +1,6 @@
> > > > >  config MLX5_INFINIBAND
> > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > >
> > > > Why do we need this?
> > >
> > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > the caller interface, but it is different between 32 and 64 bits systems.
> > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> >
> > ???
> >
> > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > you are passing a bus address to/from FW then you 0 extend it on 32
> > bit, and range check on 64 bit.
> >
> > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> 
> So how will I mark that we are forwarding physical address to FW?

I don't know what this means.. phys_addr_t will transparently 0 extend
to 64 bit which is the correct thing to do when passing a phys_addr_t
to HW.

Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > >
> > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > rule insertion and management of the device's steering tables.
> > > > > >
> > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > the alloc_dm ioctl command.
> > > > > >
> > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > @@ -1,6 +1,6 @@
> > > > > >  config MLX5_INFINIBAND
> > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > >
> > > > > Why do we need this?
> > > >
> > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > >
> > > ???
> > >
> > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > bit, and range check on 64 bit.
> > >
> > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> >
> > So how will I mark that we are forwarding physical address to FW?
>
> I don't know what this means.. phys_addr_t will transparently 0 extend
> to 64 bit which is the correct thing to do when passing a phys_addr_t
> to HW.

It looks to me very similar to what I wanted: mark input as an address
and ensure that it is always u64 as our FW expects.

Thanks

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > >
> > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > rule insertion and management of the device's steering tables.
> > > > > > >
> > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > the alloc_dm ioctl command.
> > > > > > >
> > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > >
> > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > @@ -1,6 +1,6 @@
> > > > > > >  config MLX5_INFINIBAND
> > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > >
> > > > > > Why do we need this?
> > > > >
> > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > >
> > > > ???
> > > >
> > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > bit, and range check on 64 bit.
> > > >
> > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > >
> > > So how will I mark that we are forwarding physical address to FW?
> >
> > I don't know what this means.. phys_addr_t will transparently 0 extend
> > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > to HW.
> 
> It looks to me very similar to what I wanted: mark input as an address
> and ensure that it is always u64 as our FW expects.

So there is no reason for the kconfig hunk then

Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > >
> > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > >
> > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > the alloc_dm ioctl command.
> > > > > > > >
> > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > >  config MLX5_INFINIBAND
> > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > >
> > > > > > > Why do we need this?
> > > > > >
> > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > >
> > > > > ???
> > > > >
> > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > bit, and range check on 64 bit.
> > > > >
> > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > >
> > > > So how will I mark that we are forwarding physical address to FW?
> > >
> > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > to HW.
> >
> > It looks to me very similar to what I wanted: mark input as an address
> > and ensure that it is always u64 as our FW expects.
>
> So there is no reason for the kconfig hunk then

The kconfig is needed to ensure that u64 value is truncated to be u32 on
32bits systems.

Thanks

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 06:49:37PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> > On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > >
> > > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > > >
> > > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > > the alloc_dm ioctl command.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > > >  config MLX5_INFINIBAND
> > > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > > >
> > > > > > > > Why do we need this?
> > > > > > >
> > > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > > >
> > > > > > ???
> > > > > >
> > > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > > bit, and range check on 64 bit.
> > > > > >
> > > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > > >
> > > > > So how will I mark that we are forwarding physical address to FW?
> > > >
> > > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > > to HW.
> > >
> > > It looks to me very similar to what I wanted: mark input as an address
> > > and ensure that it is always u64 as our FW expects.
> >
> > So there is no reason for the kconfig hunk then
> 
> The kconfig is needed to ensure that u64 value is truncated to be u32 on
> 32bits systems.

What?

Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 12:53:13PM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 06:49:37PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > >
> > > > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > > > >
> > > > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > > > the alloc_dm ioctl command.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > > > >  config MLX5_INFINIBAND
> > > > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > > > >
> > > > > > > > > Why do we need this?
> > > > > > > >
> > > > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > > > >
> > > > > > > ???
> > > > > > >
> > > > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > > > bit, and range check on 64 bit.
> > > > > > >
> > > > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > > > >
> > > > > > So how will I mark that we are forwarding physical address to FW?
> > > > >
> > > > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > > > to HW.
> > > >
> > > > It looks to me very similar to what I wanted: mark input as an address
> > > > and ensure that it is always u64 as our FW expects.
> > >
> > > So there is no reason for the kconfig hunk then
> >
> > The kconfig is needed to ensure that u64 value is truncated to be u32 on
> > 32bits systems.
>
> What?

I wanted to say "not truncated", sorry about that.

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 07:27:36PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 12:53:13PM -0300, Jason Gunthorpe wrote:
> > On Wed, Apr 24, 2019 at 06:49:37PM +0300, Leon Romanovsky wrote:
> > > On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> > > > On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > > > > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > > > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > >
> > > > > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > > > > >
> > > > > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > > > > the alloc_dm ioctl command.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > > > > >  config MLX5_INFINIBAND
> > > > > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > > > > >
> > > > > > > > > > Why do we need this?
> > > > > > > > >
> > > > > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > > > > >
> > > > > > > > ???
> > > > > > > >
> > > > > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > > > > bit, and range check on 64 bit.
> > > > > > > >
> > > > > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > > > > >
> > > > > > > So how will I mark that we are forwarding physical address to FW?
> > > > > >
> > > > > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > > > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > > > > to HW.
> > > > >
> > > > > It looks to me very similar to what I wanted: mark input as an address
> > > > > and ensure that it is always u64 as our FW expects.
> > > >
> > > > So there is no reason for the kconfig hunk then
> > >
> > > The kconfig is needed to ensure that u64 value is truncated to be u32 on
> > > 32bits systems.
> >
> > What?
> 
> I wanted to say "not truncated", sorry about that.

Are we reading back a u64 physical value from the HW? Where?

Jason

Re: [PATCH rdma-next 0/8] User space steering

[Saeed Mahameed]

On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote:
> On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@mellanox.com>
> > 
> > > From Ariel,
> > 
> > This series of patches adds user space managed steering
> > infrastructure
> > to the mlx5_ib driver.
> > 
> > User space managed steering requires the means to access a
> > dedicated
> > memory space that is used by the device to store the packet
> > steering
> > and header modification tables and rules in order to manage them
> > directly
> > without the device's firmware involvement. This dedicated memory is
> > part
> > of the ICM memory space.
> > 
> > The changes are introducing the mlx5_ib API to allocate, deallocate
> > and
> > register this dedicated SW ICM memory via the existing device
> > memory API
> > using a private attribute which specifies the memory type.
> > 
> > The allocated memory itself is not IO mapped and user can only
> > access it
> > using remote RDMA operations.
> > 
> > In addition, the series exposed the ICM address of the receive
> > transport
> > interface (TIR) of Raw Packet and RSS QPs to user since they are
> > required
> > to properly create and insert steering rules that direct flows to
> > these QPs.
> > 
> > Thanks
> > 
> > Ariel Levkovich (8):
> >   net/mlx5: Expose SW ICM related device memory capabilities
> >   IB/mlx5: Support device memory type attribute
> >   IB/mlx5: Warn on allocated MEMIC buffers during cleanup
> >   IB/mlx5: Add steering SW ICM device memory type
> >   IB/mlx5: Device resource control for privileged DEVX user
> 
> This doesn't apply, it conflicts with patches in mlx5-next, please
> resent
> 
> >   net/mlx5: Expose TIR ICM address in command outbox
> >   net/mlx5: Introduce new TIR creation core API
> >   IB/mlx5: Expose TIR ICM address to user space
> 

it is a 3 patch series:
  net/mlx5: Expose TIR ICM address in command outbox
  net/mlx5: Introduce new TIR creation core API
  net/mlx5: Expose SW ICM related device memory capabilities

LGTM, I can apply them if you want, or do you want to wait for leon ?

Re: [PATCH rdma-next 0/8] User space steering

[Jason Gunthorpe]

On Wed, Apr 24, 2019 at 04:40:54PM +0000, Saeed Mahameed wrote:
> On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote:
> > On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote:
> > > From: Leon Romanovsky <leonro@mellanox.com>
> > > 
> > > > From Ariel,
> > > 
> > > This series of patches adds user space managed steering
> > > infrastructure
> > > to the mlx5_ib driver.
> > > 
> > > User space managed steering requires the means to access a
> > > dedicated
> > > memory space that is used by the device to store the packet
> > > steering
> > > and header modification tables and rules in order to manage them
> > > directly
> > > without the device's firmware involvement. This dedicated memory is
> > > part
> > > of the ICM memory space.
> > > 
> > > The changes are introducing the mlx5_ib API to allocate, deallocate
> > > and
> > > register this dedicated SW ICM memory via the existing device
> > > memory API
> > > using a private attribute which specifies the memory type.
> > > 
> > > The allocated memory itself is not IO mapped and user can only
> > > access it
> > > using remote RDMA operations.
> > > 
> > > In addition, the series exposed the ICM address of the receive
> > > transport
> > > interface (TIR) of Raw Packet and RSS QPs to user since they are
> > > required
> > > to properly create and insert steering rules that direct flows to
> > > these QPs.
> > > 
> > > Thanks
> > > 
> > > Ariel Levkovich (8):
> > >   net/mlx5: Expose SW ICM related device memory capabilities
> > >   IB/mlx5: Support device memory type attribute
> > >   IB/mlx5: Warn on allocated MEMIC buffers during cleanup
> > >   IB/mlx5: Add steering SW ICM device memory type
> > >   IB/mlx5: Device resource control for privileged DEVX user
> > 
> > This doesn't apply, it conflicts with patches in mlx5-next, please
> > resent
> > 
> > >   net/mlx5: Expose TIR ICM address in command outbox
> > >   net/mlx5: Introduce new TIR creation core API
> > >   IB/mlx5: Expose TIR ICM address to user space
> > 
> 
> it is a 3 patch series:
>   net/mlx5: Expose TIR ICM address in command outbox
>   net/mlx5: Introduce new TIR creation core API
>   net/mlx5: Expose SW ICM related device memory capabilities
> 
> LGTM, I can apply them if you want, or do you want to wait for leon ?

Fine by me

Jason 

Re: [PATCH rdma-next 0/8] User space steering

[Saeed Mahameed]

On Wed, 2019-04-24 at 16:03 -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 04:40:54PM +0000, Saeed Mahameed wrote:
> > On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote:
> > > On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@mellanox.com>
> > > > 
> > > > > From Ariel,
> > > > 
> > > > This series of patches adds user space managed steering
> > > > infrastructure
> > > > to the mlx5_ib driver.
> > > > 
> > > > User space managed steering requires the means to access a
> > > > dedicated
> > > > memory space that is used by the device to store the packet
> > > > steering
> > > > and header modification tables and rules in order to manage
> > > > them
> > > > directly
> > > > without the device's firmware involvement. This dedicated
> > > > memory is
> > > > part
> > > > of the ICM memory space.
> > > > 
> > > > The changes are introducing the mlx5_ib API to allocate,
> > > > deallocate
> > > > and
> > > > register this dedicated SW ICM memory via the existing device
> > > > memory API
> > > > using a private attribute which specifies the memory type.
> > > > 
> > > > The allocated memory itself is not IO mapped and user can only
> > > > access it
> > > > using remote RDMA operations.
> > > > 
> > > > In addition, the series exposed the ICM address of the receive
> > > > transport
> > > > interface (TIR) of Raw Packet and RSS QPs to user since they
> > > > are
> > > > required
> > > > to properly create and insert steering rules that direct flows
> > > > to
> > > > these QPs.
> > > > 
> > > > Thanks
> > > > 
> > > > Ariel Levkovich (8):
> > > >   net/mlx5: Expose SW ICM related device memory capabilities
> > > >   IB/mlx5: Support device memory type attribute
> > > >   IB/mlx5: Warn on allocated MEMIC buffers during cleanup
> > > >   IB/mlx5: Add steering SW ICM device memory type
> > > >   IB/mlx5: Device resource control for privileged DEVX user
> > > 
> > > This doesn't apply, it conflicts with patches in mlx5-next,
> > > please
> > > resent
> > > 
> > > >   net/mlx5: Expose TIR ICM address in command outbox
> > > >   net/mlx5: Introduce new TIR creation core API
> > > >   IB/mlx5: Expose TIR ICM address to user space
> > 
> > it is a 3 patch series:
> >   net/mlx5: Expose TIR ICM address in command outbox
> >   net/mlx5: Introduce new TIR creation core API
> >   net/mlx5: Expose SW ICM related device memory capabilities
> > 
> > LGTM, I can apply them if you want, or do you want to wait for leon
> > ?
> 
> Fine by me
> 

the above 3 mlx5-next patches applied.

Thanks,
Saeed.

> Jason 

Re: [PATCH rdma-next 0/8] User space steering

[Jason Gunthorpe]

On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote:

>   net/mlx5: Expose TIR ICM address in command outbox
>   net/mlx5: Introduce new TIR creation core API
>   IB/mlx5: Expose TIR ICM address to user space

These are applied to for-next now

Thanks,
Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Leon Romanovsky]

On Wed, Apr 24, 2019 at 01:39:26PM -0300, Jason Gunthorpe wrote:
> On Wed, Apr 24, 2019 at 07:27:36PM +0300, Leon Romanovsky wrote:
> > On Wed, Apr 24, 2019 at 12:53:13PM -0300, Jason Gunthorpe wrote:
> > > On Wed, Apr 24, 2019 at 06:49:37PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> > > > > On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > > > > > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > > > > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > > > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > > >
> > > > > > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > > > > > >
> > > > > > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > > > > > the alloc_dm ioctl command.
> > > > > > > > > > > >
> > > > > > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > > > > > >  config MLX5_INFINIBAND
> > > > > > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > > > > > >
> > > > > > > > > > > Why do we need this?
> > > > > > > > > >
> > > > > > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > > > > > >
> > > > > > > > > ???
> > > > > > > > >
> > > > > > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > > > > > bit, and range check on 64 bit.
> > > > > > > > >
> > > > > > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > > > > > >
> > > > > > > > So how will I mark that we are forwarding physical address to FW?
> > > > > > >
> > > > > > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > > > > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > > > > > to HW.
> > > > > >
> > > > > > It looks to me very similar to what I wanted: mark input as an address
> > > > > > and ensure that it is always u64 as our FW expects.
> > > > >
> > > > > So there is no reason for the kconfig hunk then
> > > >
> > > > The kconfig is needed to ensure that u64 value is truncated to be u32 on
> > > > 32bits systems.
> > >
> > > What?
> >
> > I wanted to say "not truncated", sorry about that.
>
> Are we reading back a u64 physical value from the HW? Where?

No, we are setting address only.

>
> Jason

Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type

[Jason Gunthorpe]

On Sun, Apr 28, 2019 at 02:38:48PM +0300, Leon Romanovsky wrote:
> On Wed, Apr 24, 2019 at 01:39:26PM -0300, Jason Gunthorpe wrote:
> > On Wed, Apr 24, 2019 at 07:27:36PM +0300, Leon Romanovsky wrote:
> > > On Wed, Apr 24, 2019 at 12:53:13PM -0300, Jason Gunthorpe wrote:
> > > > On Wed, Apr 24, 2019 at 06:49:37PM +0300, Leon Romanovsky wrote:
> > > > > On Wed, Apr 24, 2019 at 10:47:03AM -0300, Jason Gunthorpe wrote:
> > > > > > On Wed, Apr 24, 2019 at 04:46:04PM +0300, Leon Romanovsky wrote:
> > > > > > > On Wed, Apr 24, 2019 at 10:35:53AM -0300, Jason Gunthorpe wrote:
> > > > > > > > On Wed, Apr 24, 2019 at 04:25:16PM +0300, Leon Romanovsky wrote:
> > > > > > > > > On Wed, Apr 24, 2019 at 10:14:05AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > > On Wed, Apr 24, 2019 at 04:12:22PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > > On Wed, Apr 24, 2019 at 10:04:40AM -0300, Jason Gunthorpe wrote:
> > > > > > > > > > > > On Sun, Mar 31, 2019 at 07:44:46PM +0300, Leon Romanovsky wrote:
> > > > > > > > > > > > > From: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > > > >
> > > > > > > > > > > > > This patch adds support for allocating, deallocating and registering
> > > > > > > > > > > > > a new device memory type, STEERING_SW_ICM.
> > > > > > > > > > > > > This memory can be allocated and used by a privileged user for direct
> > > > > > > > > > > > > rule insertion and management of the device's steering tables.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The type is provided by the user via the dedicated attribute in
> > > > > > > > > > > > > the alloc_dm ioctl command.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Signed-off-by: Ariel Levkovich <lariel@mellanox.com>
> > > > > > > > > > > > > Reviewed-by: Eli Cohen <eli@mellanox.com>
> > > > > > > > > > > > > Reviewed-by: Mark Bloch <markb@mellanox.com>
> > > > > > > > > > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/Kconfig        |   2 +-
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.c          | 127 +++++++++++++++++++-
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/cmd.h          |   6 +-
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/main.c         | 136 +++++++++++++++++++++-
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/mlx5_ib.h      |  17 +++
> > > > > > > > > > > > >  drivers/infiniband/hw/mlx5/mr.c           |   7 ++
> > > > > > > > > > > > >  include/uapi/rdma/mlx5_user_ioctl_verbs.h |   2 +
> > > > > > > > > > > > >  7 files changed, 291 insertions(+), 6 deletions(-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > > > index 8d651c05de62..347d457fcb2f 100644
> > > > > > > > > > > > > +++ b/drivers/infiniband/hw/mlx5/Kconfig
> > > > > > > > > > > > > @@ -1,6 +1,6 @@
> > > > > > > > > > > > >  config MLX5_INFINIBAND
> > > > > > > > > > > > >  	tristate "Mellanox 5th generation network adapters (ConnectX series) support"
> > > > > > > > > > > > > -	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
> > > > > > > > > > > > > +	depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
> > > > > > > > > > > >
> > > > > > > > > > > > Why do we need this?
> > > > > > > > > > >
> > > > > > > > > > > We are using phys_addr_t type and not u64 as an input to FW to emphasize
> > > > > > > > > > > the caller interface, but it is different between 32 and 64 bits systems.
> > > > > > > > > > > Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
> > > > > > > > > >
> > > > > > > > > > ???
> > > > > > > > > >
> > > > > > > > > > PHYS_ADDR_T should only ever be used to store a bus address - and if
> > > > > > > > > > you are passing a bus address to/from FW then you 0 extend it on 32
> > > > > > > > > > bit, and range check on 64 bit.
> > > > > > > > > >
> > > > > > > > > > This is not a reason to force PHYS_ADDR_T_64BIT from a driver
> > > > > > > > >
> > > > > > > > > So how will I mark that we are forwarding physical address to FW?
> > > > > > > >
> > > > > > > > I don't know what this means.. phys_addr_t will transparently 0 extend
> > > > > > > > to 64 bit which is the correct thing to do when passing a phys_addr_t
> > > > > > > > to HW.
> > > > > > >
> > > > > > > It looks to me very similar to what I wanted: mark input as an address
> > > > > > > and ensure that it is always u64 as our FW expects.
> > > > > >
> > > > > > So there is no reason for the kconfig hunk then
> > > > >
> > > > > The kconfig is needed to ensure that u64 value is truncated to be u32 on
> > > > > 32bits systems.
> > > >
> > > > What?
> > >
> > > I wanted to say "not truncated", sorry about that.
> >
> > Are we reading back a u64 physical value from the HW? Where?
> 
> No, we are setting address only.

There there is no need for the kconfig, the phys_addr_t transparently
0 extends to 64 bits which is the right thing to do.

Jason