[PATCH rdma-rc 0/8] Fixes for v5.3

[PATCH rdma-rc 0/8] Fixes for v5.3

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

Hi,

This is a collection of new fixes for v5.3, everything here is fixing
kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice
to have improvement to implement rest of the series.

Thanks

Ido Kalir (1):
  IB/core: Fix NULL pointer dereference when bind QP to counter

Jason Gunthorpe (1):
  RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB

Leon Romanovsky (2):
  RDMA/counters: Properly implement PID checks
  RDMA/restrack: Rewrite PID namespace check to be reliable

Moni Shoua (4):
  IB/mlx5: Consolidate use_umr checks into single function
  IB/mlx5: Report and handle ODP support properly
  IB/mlx5: Fix MR re-registration flow to use UMR properly
  IB/mlx5: Block MR WR if UMR is not possible

 drivers/infiniband/core/counters.c   | 10 ++++------
 drivers/infiniband/core/nldev.c      |  3 +--
 drivers/infiniband/core/restrack.c   | 15 +++++++--------
 drivers/infiniband/core/umem.c       |  7 +------
 drivers/infiniband/hw/mlx5/main.c    |  6 +++---
 drivers/infiniband/hw/mlx5/mem.c     |  5 +++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++++++++++
 drivers/infiniband/hw/mlx5/mr.c      |  7 +++----
 drivers/infiniband/hw/mlx5/odp.c     | 17 +++++++++--------
 drivers/infiniband/hw/mlx5/qp.c      | 24 +++++++++++++++++++-----
 include/rdma/restrack.h              |  3 +--
 11 files changed, 65 insertions(+), 46 deletions(-)

--
2.20.1

[PATCH rdma-rc 1/8] IB/core: Fix NULL pointer dereference when bind QP to counter

[Leon Romanovsky]

From: Ido Kalir <idok@mellanox.com>

If QP is not visible to the pid, then we try to decrease its reference
count and return from the function before the QP pointer is
initialized. This lead to NULL pointer dereference.
Fix it by pass directly the res to the rdma_restract_put as arg instead of
&qp->res.

This fixes below call trace:
[ 5845.110329] BUG: kernel NULL pointer dereference, address:
00000000000000dc
[ 5845.120482] Oops: 0002 [#1] SMP PTI
[ 5845.129119] RIP: 0010:rdma_restrack_put+0x5/0x30 [ib_core]
[ 5845.169450] Call Trace:
[ 5845.170544]  rdma_counter_get_qp+0x5c/0x70 [ib_core]
[ 5845.172074]  rdma_counter_bind_qpn_alloc+0x6f/0x1a0 [ib_core]
[ 5845.173731]  nldev_stat_set_doit+0x314/0x330 [ib_core]
[ 5845.175279]  rdma_nl_rcv_msg+0xeb/0x1d0 [ib_core]
[ 5845.176772]  ? __kmalloc_node_track_caller+0x20b/0x2b0
[ 5845.178321]  rdma_nl_rcv+0xcb/0x120 [ib_core]
[ 5845.179753]  netlink_unicast+0x179/0x220
[ 5845.181066]  netlink_sendmsg+0x2d8/0x3d0
[ 5845.182338]  sock_sendmsg+0x30/0x40
[ 5845.183544]  __sys_sendto+0xdc/0x160
[ 5845.184832]  ? syscall_trace_enter+0x1f8/0x2e0
[ 5845.186209]  ? __audit_syscall_exit+0x1d9/0x280
[ 5845.187584]  __x64_sys_sendto+0x24/0x30
[ 5845.188867]  do_syscall_64+0x48/0x120
[ 5845.190097]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Fixes: 1bd8e0a9d0fd1 ("RDMA/counter: Allow manual mode configuration support")
Signed-off-by: Ido Kalir <idok@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/counters.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c
index b79890739a2c..955d061af06a 100644
--- a/drivers/infiniband/core/counters.c
+++ b/drivers/infiniband/core/counters.c
@@ -424,7 +424,7 @@ static struct ib_qp *rdma_counter_get_qp(struct ib_device *dev, u32 qp_num)
 	return qp;
 
 err:
-	rdma_restrack_put(&qp->res);
+	rdma_restrack_put(res);
 	return NULL;
 }
 
-- 
2.20.1

[PATCH rdma-rc 2/8] RDMA/counters: Properly implement PID checks

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

"Auto" configuration mode is called for visible in that PID
namespace and it ensures that all counters and QPs are coexist
in the same namespace and belong to same PID.

Fixes: 99fa331dc862 ("RDMA/counter: Add "auto" configuration mode support")
Reviewed-by: Mark Zhang <markz@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/counters.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c
index 955d061af06a..af8c85d18e62 100644
--- a/drivers/infiniband/core/counters.c
+++ b/drivers/infiniband/core/counters.c
@@ -149,13 +149,11 @@ static bool auto_mode_match(struct ib_qp *qp, struct rdma_counter *counter,
 	struct auto_mode_param *param = &counter->mode.param;
 	bool match = true;
 
-	if (rdma_is_kernel_res(&counter->res) != rdma_is_kernel_res(&qp->res))
+	if (!rdma_is_visible_in_pid_ns(&qp->res))
 		return false;
 
-	/* Ensure that counter belong to right PID */
-	if (!rdma_is_kernel_res(&counter->res) &&
-	    !rdma_is_kernel_res(&qp->res) &&
-	    (task_pid_vnr(counter->res.task) != current->pid))
+	/* Ensure that counter belongs to the right PID */
+	if (task_pid_nr(counter->res.task) != task_pid_nr(qp->res.task))
 		return false;
 
 	if (auto_mask & RDMA_COUNTER_MASK_QP_TYPE)
-- 
2.20.1

[PATCH rdma-rc 3/8] RDMA/restrack: Rewrite PID namespace check to be reliable

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

task_active_pid_ns() is wrong API to check PID namespace because it
posses some restrictions and return PID namespace where the process
was allocated. It created mismatches with current namespace, which
can be different.

Rewrite whole rdma_is_visible_in_pid_ns() logic to provide reliable
results without any relation to allocated PID namespace.

Fixes: 8be565e65fa9 ("RDMA/nldev: Factor out the PID namespace check")
Fixes: 6a6c306a09b5 ("RDMA/restrack: Make is_visible_in_pid_ns() as an API")
Reviewed-by: Mark Zhang <markz@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/nldev.c    |  3 +--
 drivers/infiniband/core/restrack.c | 15 +++++++--------
 include/rdma/restrack.h            |  3 +--
 3 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 87d40d1ecdde..020c26976558 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -382,8 +382,7 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device)
 	for (i = 0; i < RDMA_RESTRACK_MAX; i++) {
 		if (!names[i])
 			continue;
-		curr = rdma_restrack_count(device, i,
-					   task_active_pid_ns(current));
+		curr = rdma_restrack_count(device, i);
 		ret = fill_res_info_entry(msg, names[i], curr);
 		if (ret)
 			goto err;
diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c
index bddff426ee0f..a07665f7ef8c 100644
--- a/drivers/infiniband/core/restrack.c
+++ b/drivers/infiniband/core/restrack.c
@@ -107,10 +107,8 @@ void rdma_restrack_clean(struct ib_device *dev)
  * rdma_restrack_count() - the current usage of specific object
  * @dev:  IB device
  * @type: actual type of object to operate
- * @ns:   PID namespace
  */
-int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
-			struct pid_namespace *ns)
+int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type)
 {
 	struct rdma_restrack_root *rt = &dev->res[type];
 	struct rdma_restrack_entry *e;
@@ -119,10 +117,9 @@ int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
 
 	xa_lock(&rt->xa);
 	xas_for_each(&xas, e, U32_MAX) {
-		if (ns == &init_pid_ns ||
-		    (!rdma_is_kernel_res(e) &&
-		     ns == task_active_pid_ns(e->task)))
-			cnt++;
+		if (!rdma_is_visible_in_pid_ns(e))
+			continue;
+		cnt++;
 	}
 	xa_unlock(&rt->xa);
 	return cnt;
@@ -360,5 +357,7 @@ bool rdma_is_visible_in_pid_ns(struct rdma_restrack_entry *res)
 	 */
 	if (rdma_is_kernel_res(res))
 		return task_active_pid_ns(current) == &init_pid_ns;
-	return task_active_pid_ns(current) == task_active_pid_ns(res->task);
+
+	/* PID 0 means that resource is not found in current namespace */
+	return task_pid_vnr(res->task);
 }
diff --git a/include/rdma/restrack.h b/include/rdma/restrack.h
index b0fc6b26bdf5..83df1ec6664e 100644
--- a/include/rdma/restrack.h
+++ b/include/rdma/restrack.h
@@ -105,8 +105,7 @@ struct rdma_restrack_entry {
 };
 
 int rdma_restrack_count(struct ib_device *dev,
-			enum rdma_restrack_type type,
-			struct pid_namespace *ns);
+			enum rdma_restrack_type type);
 
 void rdma_restrack_kadd(struct rdma_restrack_entry *res);
 void rdma_restrack_uadd(struct rdma_restrack_entry *res);
-- 
2.20.1

[PATCH rdma-rc 4/8] RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB

[Leon Romanovsky]

From: Jason Gunthorpe <jgg@mellanox.com>

When ODP is enabled with IB_ACCESS_HUGETLB then the required pages
should be calculated based on the extent of the MR, which is rounded
to the nearest huge page alignment.

Fixes: d2183c6f1958 ("RDMA/umem: Move page_shift from ib_umem to ib_odp_umem")
Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/core/umem.c   | 7 +------
 drivers/infiniband/hw/mlx5/mem.c | 5 +++--
 2 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c
index 08da840ed7ee..56553668256f 100644
--- a/drivers/infiniband/core/umem.c
+++ b/drivers/infiniband/core/umem.c
@@ -379,14 +379,9 @@ EXPORT_SYMBOL(ib_umem_release);
 
 int ib_umem_page_count(struct ib_umem *umem)
 {
-	int i;
-	int n;
+	int i, n = 0;
 	struct scatterlist *sg;
 
-	if (umem->is_odp)
-		return ib_umem_num_pages(umem);
-
-	n = 0;
 	for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i)
 		n += sg_dma_len(sg) >> PAGE_SHIFT;
 
diff --git a/drivers/infiniband/hw/mlx5/mem.c b/drivers/infiniband/hw/mlx5/mem.c
index fe1a76d8531c..a40e0abf2338 100644
--- a/drivers/infiniband/hw/mlx5/mem.c
+++ b/drivers/infiniband/hw/mlx5/mem.c
@@ -57,9 +57,10 @@ void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr,
 	int entry;
 
 	if (umem->is_odp) {
-		unsigned int page_shift = to_ib_umem_odp(umem)->page_shift;
+		struct ib_umem_odp *odp = to_ib_umem_odp(umem);
+		unsigned int page_shift = odp->page_shift;
 
-		*ncont = ib_umem_page_count(umem);
+		*ncont = ib_umem_odp_num_pages(odp);
 		*count = *ncont << (page_shift - PAGE_SHIFT);
 		*shift = page_shift;
 		if (order)
-- 
2.20.1

[PATCH rdma-rc 5/8] IB/mlx5: Consolidate use_umr checks into single function

[Leon Romanovsky]

From: Moni Shoua <monis@mellanox.com>

Introduce helper function to unify various use_umr checks.

Signed-off-by: Moni Shoua <monis@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++++++++++
 drivers/infiniband/hw/mlx5/mr.c      |  4 +---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index f6a53455bf8b..9ae587b74b12 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1475,4 +1475,18 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev,
 			bool dyn_bfreg);
 
 int mlx5_ib_qp_set_counter(struct ib_qp *qp, struct rdma_counter *counter);
+
+static inline bool mlx5_ib_can_use_umr(struct mlx5_ib_dev *dev,
+				       bool do_modify_atomic)
+{
+	if (MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled))
+		return false;
+
+	if (do_modify_atomic &&
+	    MLX5_CAP_GEN(dev->mdev, atomic) &&
+	    MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled))
+		return false;
+
+	return true;
+}
 #endif /* MLX5_IB_H */
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index b74fad08412f..8bce65c03b84 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1293,9 +1293,7 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
 	if (err < 0)
 		return ERR_PTR(err);
 
-	use_umr = !MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled) &&
-		  (!MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled) ||
-		   !MLX5_CAP_GEN(dev->mdev, atomic));
+	use_umr = mlx5_ib_can_use_umr(dev, true);
 
 	if (order <= mr_cache_max_order(dev) && use_umr) {
 		mr = alloc_mr_from_cache(pd, umem, virt_addr, length, ncont,
-- 
2.20.1

[PATCH rdma-rc 6/8] IB/mlx5: Report and handle ODP support properly

[Leon Romanovsky]

From: Moni Shoua <monis@mellanox.com>

ODP depends on the several device capabilities, among them is the ability
to send UMR WQEs with that modify atomic and entity size of the MR.
Therefore, only if all conditions to send such a UMR WQE are met then
driver can report that ODP is supported. Use this check of conditions
in all places where driver needs to know about ODP support.

Also, implicit ODP support depends on ability of driver to send UMR WQEs
for an indirect mkey. Therefore, verify that all conditions to do so are
met when reporting support.

Fixes: c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")
Signed-off-by: Moni Shoua <monis@mellanox.com>
Reviewed-by: Guy Levi <guyle@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/main.c |  6 +++---
 drivers/infiniband/hw/mlx5/odp.c  | 17 +++++++++--------
 2 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index e12a4404096b..0569bcab02d4 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1023,7 +1023,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
-		if (MLX5_CAP_GEN(mdev, pg))
+		if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
 			props->device_cap_flags |= IB_DEVICE_ON_DEMAND_PAGING;
 		props->odp_caps = dev->odp_caps;
 	}
@@ -6139,6 +6139,8 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 		dev->port[i].roce.last_port_state = IB_PORT_DOWN;
 	}
 
+	mlx5_ib_internal_fill_odp_caps(dev);
+
 	err = mlx5_ib_init_multiport_master(dev);
 	if (err)
 		return err;
@@ -6563,8 +6565,6 @@ static void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev)
 
 static int mlx5_ib_stage_odp_init(struct mlx5_ib_dev *dev)
 {
-	mlx5_ib_internal_fill_odp_caps(dev);
-
 	return mlx5_ib_odp_init_one(dev);
 }
 
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index 1d257d1b3b0d..0a59912a4cef 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -301,7 +301,8 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
 
 	memset(caps, 0, sizeof(*caps));
 
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
+	if (!MLX5_CAP_GEN(dev->mdev, pg) ||
+	    !mlx5_ib_can_use_umr(dev, true))
 		return;
 
 	caps->general_caps = IB_ODP_SUPPORT;
@@ -355,7 +356,8 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
 
 	if (MLX5_CAP_GEN(dev->mdev, fixed_buffer_size) &&
 	    MLX5_CAP_GEN(dev->mdev, null_mkey) &&
-	    MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset))
+	    MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset) &&
+	    !MLX5_CAP_GEN(dev->mdev, umr_indirect_mkey_disabled))
 		caps->general_caps |= IB_ODP_SUPPORT_IMPLICIT;
 
 	return;
@@ -1622,8 +1624,10 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 {
 	int ret = 0;
 
-	if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
-		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
+	if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
+		return ret;
+
+	ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
 
 	if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
 		ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);
@@ -1633,9 +1637,6 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 		}
 	}
 
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
-		return ret;
-
 	ret = mlx5_ib_create_pf_eq(dev, &dev->odp_pf_eq);
 
 	return ret;
@@ -1643,7 +1644,7 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 
 void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *dev)
 {
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
+	if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
 		return;
 
 	mlx5_ib_destroy_pf_eq(dev, &dev->odp_pf_eq);
-- 
2.20.1

[PATCH rdma-rc 7/8] IB/mlx5: Fix MR re-registration flow to use UMR properly

[Leon Romanovsky]

From: Moni Shoua <monis@mellanox.com>

The UMR WQE in the MR re-registration flow requires that
modify_atomic and modify_entity_size capabilities are enabled.
Therefore, check that the these capabilities are present before going to
umr flow and go through slow path if not.

Fixes: c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")
Signed-off-by: Moni Shoua <monis@mellanox.com>
Reviewed-by: Guy Levi <guyle@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/mr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 8bce65c03b84..3401f5f6792e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1446,7 +1446,8 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
 			goto err;
 	}
 
-	if (flags & IB_MR_REREG_TRANS && !use_umr_mtt_update(mr, addr, len)) {
+	if (!mlx5_ib_can_use_umr(dev, true) ||
+	    (flags & IB_MR_REREG_TRANS && !use_umr_mtt_update(mr, addr, len))) {
 		/*
 		 * UMR can't be used - MKey needs to be replaced.
 		 */
-- 
2.20.1

[PATCH rdma-rc 8/8] IB/mlx5: Block MR WR if UMR is not possible

[Leon Romanovsky]

From: Moni Shoua <monis@mellanox.com>

Check conditions that are mandatory to post_send UMR WQEs.
1. Modifying page size.
2. Modifying remote atomic permissions if atomic access is required.

If either condition is not fulfilled then fail to post_send() flow.

Fixes: c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")
Signed-off-by: Moni Shoua <monis@mellanox.com>
Reviewed-by: Guy Levi <guyle@mellanox.com>
Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 379328b2598f..72869ff4a334 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -4162,7 +4162,7 @@ static u64 get_xlt_octo(u64 bytes)
 	       MLX5_IB_UMR_OCTOWORD;
 }

-static __be64 frwr_mkey_mask(void)
+static __be64 frwr_mkey_mask(bool atomic)
 {
 	u64 result;

@@ -4175,10 +4175,12 @@ static __be64 frwr_mkey_mask(void)
 		MLX5_MKEY_MASK_LW		|
 		MLX5_MKEY_MASK_RR		|
 		MLX5_MKEY_MASK_RW		|
-		MLX5_MKEY_MASK_A		|
 		MLX5_MKEY_MASK_SMALL_FENCE	|
 		MLX5_MKEY_MASK_FREE;

+	if (atomic)
+		result |= MLX5_MKEY_MASK_A;
+
 	return cpu_to_be64(result);
 }

@@ -4204,7 +4206,7 @@ static __be64 sig_mkey_mask(void)
 }

 static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
-			    struct mlx5_ib_mr *mr, u8 flags)
+			    struct mlx5_ib_mr *mr, u8 flags, bool atomic)
 {
 	int size = (mr->ndescs + mr->meta_ndescs) * mr->desc_size;

@@ -4212,7 +4214,7 @@ static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,

 	umr->flags = flags;
 	umr->xlt_octowords = cpu_to_be16(get_xlt_octo(size));
-	umr->mkey_mask = frwr_mkey_mask();
+	umr->mkey_mask = frwr_mkey_mask(atomic);
 }

 static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr)
@@ -4811,10 +4813,22 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 {
 	struct mlx5_ib_mr *mr = to_mmr(wr->mr);
 	struct mlx5_ib_pd *pd = to_mpd(qp->ibqp.pd);
+	struct mlx5_ib_dev *dev = to_mdev(pd->ibpd.device);
 	int mr_list_size = (mr->ndescs + mr->meta_ndescs) * mr->desc_size;
 	bool umr_inline = mr_list_size <= MLX5_IB_SQ_UMR_INLINE_THRESHOLD;
+	bool atomic = wr->access & IB_ACCESS_REMOTE_ATOMIC;
 	u8 flags = 0;

+	if (!mlx5_ib_can_use_umr(dev, atomic)) {
+		mlx5_ib_warn(to_mdev(qp->ibqp.device),
+			     "Fast update of %s for MR is disabled\n",
+			     (MLX5_CAP_GEN(dev->mdev,
+					   umr_modify_entity_size_disabled)) ?
+				     "entity size" :
+				     "atomic access");
+		return -EINVAL;
+	}
+
 	if (unlikely(wr->wr.send_flags & IB_SEND_INLINE)) {
 		mlx5_ib_warn(to_mdev(qp->ibqp.device),
 			     "Invalid IB_SEND_INLINE send flag\n");
@@ -4826,7 +4840,7 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 	if (umr_inline)
 		flags |= MLX5_UMR_INLINE;

-	set_reg_umr_seg(*seg, mr, flags);
+	set_reg_umr_seg(*seg, mr, flags, atomic);
 	*seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
 	*size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);
--
2.20.1

Re: [PATCH rdma-rc 0/8] Fixes for v5.3

[Jason Gunthorpe]

On Thu, Aug 15, 2019 at 11:38:26AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
> 
> Hi,
> 
> This is a collection of new fixes for v5.3, everything here is fixing
> kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
> Consolidate use_umr checks into single function". That patch is nice
> to have improvement to implement rest of the series.
> 
> Thanks
> 
> Ido Kalir (1):
>   IB/core: Fix NULL pointer dereference when bind QP to counter
> 
> Jason Gunthorpe (1):
>   RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
> 
> Leon Romanovsky (2):
>   RDMA/counters: Properly implement PID checks
>   RDMA/restrack: Rewrite PID namespace check to be reliable
> 
> Moni Shoua (4):
>   IB/mlx5: Consolidate use_umr checks into single function
>   IB/mlx5: Report and handle ODP support properly
>   IB/mlx5: Fix MR re-registration flow to use UMR properly
>   IB/mlx5: Block MR WR if UMR is not possible

I'm a little murky on what thes are fixing? Moni?

Jason

RE: [PATCH rdma-rc 0/8] Fixes for v5.3

[Guy Levi(SW)]

> -----Original Message-----
> From: Jason Gunthorpe <jgg@mellanox.com>
> Sent: Thursday, August 15, 2019 10:30 PM
> To: Leon Romanovsky <leon@kernel.org>; Moni Shoua
> <monis@mellanox.com>
> Cc: Doug Ledford <dledford@redhat.com>; Leon Romanovsky
> <leonro@mellanox.com>; RDMA mailing list <linux-rdma@vger.kernel.org>;
> Guy Levi(SW) <guyle@mellanox.com>; Ido Kalir <idok@mellanox.com>; Majd
> Dibbiny <majd@mellanox.com>; Mark Zhang <markz@mellanox.com>
> Subject: Re: [PATCH rdma-rc 0/8] Fixes for v5.3
> 
> On Thu, Aug 15, 2019 at 11:38:26AM +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@mellanox.com>
> >
> > Hi,
> >
> > This is a collection of new fixes for v5.3, everything here is fixing
> > kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
> > Consolidate use_umr checks into single function". That patch is nice
> > to have improvement to implement rest of the series.
> >
> > Thanks
> >
> > Ido Kalir (1):
> >   IB/core: Fix NULL pointer dereference when bind QP to counter
> >
> > Jason Gunthorpe (1):
> >   RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
> >
> > Leon Romanovsky (2):
> >   RDMA/counters: Properly implement PID checks
> >   RDMA/restrack: Rewrite PID namespace check to be reliable
> >
> > Moni Shoua (4):
> >   IB/mlx5: Consolidate use_umr checks into single function
> >   IB/mlx5: Report and handle ODP support properly
> >   IB/mlx5: Fix MR re-registration flow to use UMR properly
> >   IB/mlx5: Block MR WR if UMR is not possible
> 
> I'm a little murky on what thes are fixing? Moni?

See original fixed patch.
Sometime the HW device disables some UMR abilities and hence driver should avoid UMR usage in some cases. This HW behavior can be seen today in Linux@windows environment with "untrusted VF".

> 
> Jason

Re: [PATCH rdma-rc 0/8] Fixes for v5.3

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Thu, 2019-08-15 at 11:38 +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@mellanox.com>
> 
> Hi,
> 
> This is a collection of new fixes for v5.3, everything here is fixing
> kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
> Consolidate use_umr checks into single function". That patch is nice
> to have improvement to implement rest of the series.
> 
> Thanks
> 
> Ido Kalir (1):
>   IB/core: Fix NULL pointer dereference when bind QP to counter
> 
> Jason Gunthorpe (1):
>   RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
> 
> Leon Romanovsky (2):
>   RDMA/counters: Properly implement PID checks
>   RDMA/restrack: Rewrite PID namespace check to be reliable
> 
> Moni Shoua (4):
>   IB/mlx5: Consolidate use_umr checks into single function
>   IB/mlx5: Report and handle ODP support properly
>   IB/mlx5: Fix MR re-registration flow to use UMR properly
>   IB/mlx5: Block MR WR if UMR is not possible

Hi Leon,

I took everything except Jason's patch to for-rc.  He had tagged his
patch in patchworks as under review by himself, mainly because there are
some conflicts with other hmm patches I think, so he wanted to process
all the patches himself on a distinct branch to resolve the issues. 
Thanks.

-- 
Doug Ledford <dledford@redhat.com>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-rc 0/8] Fixes for v5.3

[Jason Gunthorpe]

On Tue, Aug 20, 2019 at 12:56:37PM -0400, Doug Ledford wrote:
> On Thu, 2019-08-15 at 11:38 +0300, Leon Romanovsky wrote:
> > From: Leon Romanovsky <leonro@mellanox.com>
> > 
> > Hi,
> > 
> > This is a collection of new fixes for v5.3, everything here is fixing
> > kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
> > Consolidate use_umr checks into single function". That patch is nice
> > to have improvement to implement rest of the series.
> > 
> > Thanks
> > 
> > Ido Kalir (1):
> >   IB/core: Fix NULL pointer dereference when bind QP to counter
> > 
> > Jason Gunthorpe (1):
> >   RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
> > 
> > Leon Romanovsky (2):
> >   RDMA/counters: Properly implement PID checks
> >   RDMA/restrack: Rewrite PID namespace check to be reliable
> > 
> > Moni Shoua (4):
> >   IB/mlx5: Consolidate use_umr checks into single function
> >   IB/mlx5: Report and handle ODP support properly
> >   IB/mlx5: Fix MR re-registration flow to use UMR properly
> >   IB/mlx5: Block MR WR if UMR is not possible
> 
> Hi Leon,
> 
> I took everything except Jason's patch to for-rc.  He had tagged his
> patch in patchworks as under review by himself, mainly because there are
> some conflicts with other hmm patches I think, so he wanted to process
> all the patches himself on a distinct branch to resolve the issues. 
> Thanks.

Ah, I thought that got grabbed alread

Can you take it but rebase it to be the first patch on top of v5.3-rc5
in the -rc branch?

Thanks,
Jason

Re: [PATCH rdma-rc 0/8] Fixes for v5.3

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2133 bytes --]

On Tue, 2019-08-20 at 14:38 -0300, Jason Gunthorpe wrote:
> On Tue, Aug 20, 2019 at 12:56:37PM -0400, Doug Ledford wrote:
> > On Thu, 2019-08-15 at 11:38 +0300, Leon Romanovsky wrote:
> > > From: Leon Romanovsky <leonro@mellanox.com>
> > > 
> > > Hi,
> > > 
> > > This is a collection of new fixes for v5.3, everything here is
> > > fixing
> > > kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
> > > Consolidate use_umr checks into single function". That patch is
> > > nice
> > > to have improvement to implement rest of the series.
> > > 
> > > Thanks
> > > 
> > > Ido Kalir (1):
> > >   IB/core: Fix NULL pointer dereference when bind QP to counter
> > > 
> > > Jason Gunthorpe (1):
> > >   RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
> > > 
> > > Leon Romanovsky (2):
> > >   RDMA/counters: Properly implement PID checks
> > >   RDMA/restrack: Rewrite PID namespace check to be reliable
> > > 
> > > Moni Shoua (4):
> > >   IB/mlx5: Consolidate use_umr checks into single function
> > >   IB/mlx5: Report and handle ODP support properly
> > >   IB/mlx5: Fix MR re-registration flow to use UMR properly
> > >   IB/mlx5: Block MR WR if UMR is not possible
> > 
> > Hi Leon,
> > 
> > I took everything except Jason's patch to for-rc.  He had tagged his
> > patch in patchworks as under review by himself, mainly because there
> > are
> > some conflicts with other hmm patches I think, so he wanted to
> > process
> > all the patches himself on a distinct branch to resolve the issues. 
> > Thanks.
> 
> Ah, I thought that got grabbed alread

I assume you meant to when you bumped for-rc to -rc5?  Evidently, it got
missed ;-)

> Can you take it but rebase it to be the first patch on top of v5.3-rc5
> in the -rc branch?

Sure.  You can get your branch head by digging this commit out from my
wip/dl-for-rc, but just for ease, I also pushed this patch as the head
out to the official for-rc branch.

-- 
Doug Ledford <dledford@redhat.com>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]