[PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Leon Romanovsky]

From: Maor Gottlieb <maorg@nvidia.com>

The cited commit moved umem into the union, hence
umem could be accessed only for user MRs. Add udata check
before access umem in the dereg flow.

Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
Tested-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
 drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
 drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 4a7a56ed740b..29d439cebd22 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -1296,7 +1296,7 @@ int mlx5_ib_update_mr_pas(struct mlx5_ib_mr *mr, unsigned int flags);
 struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,
 					     int access_flags);
 void mlx5_ib_free_implicit_mr(struct mlx5_ib_mr *mr);
-void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr);
+void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr, struct ib_udata *udata);
 struct ib_mr *mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
 				    u64 length, u64 virt_addr, int access_flags,
 				    struct ib_pd *pd, struct ib_udata *udata);
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 63e2129f1142..dc833071949f 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1977,7 +1977,7 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
 			return rc;
 	}
 
-	if (mr->umem) {
+	if (udata && mr->umem) {
 		bool is_odp = is_odp_mr(mr);
 
 		if (!is_odp)
@@ -1985,7 +1985,7 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
 				   &dev->mdev->priv.reg_pages);
 		ib_umem_release(mr->umem);
 		if (is_odp)
-			mlx5_ib_free_odp_mr(mr);
+			mlx5_ib_free_odp_mr(mr, udata);
 	}
 
 	if (mr->cache_ent) {
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index 91eb615b89ee..3928576b6696 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -530,7 +530,7 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,
 	return ERR_PTR(err);
 }
 
-void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr)
+void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr, struct ib_udata *udata)
 {
 	struct mlx5_ib_mr *mtt;
 	unsigned long idx;
@@ -541,7 +541,7 @@ void mlx5_ib_free_odp_mr(struct mlx5_ib_mr *mr)
 	 */
 	xa_for_each(&mr->implicit_children, idx, mtt) {
 		xa_erase(&mr->implicit_children, idx);
-		mlx5_ib_dereg_mr(&mtt->ibmr, NULL);
+		mlx5_ib_dereg_mr(&mtt->ibmr, udata);
 	}
 }
 
-- 
2.33.1

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Tony Lu]

On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote:
> From: Maor Gottlieb <maorg@nvidia.com>
> 
> The cited commit moved umem into the union, hence
> umem could be accessed only for user MRs. Add udata check
> before access umem in the dereg flow.
> 
> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
> Tested-by: Chuck Lever <chuck.lever@oracle.com>
> Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> ---
>  drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
>  drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
>  drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
>  3 files changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h

This patch was tested and works for me in our environment for SMC. It
wouldn't panic when release link and call ib_dereg_mr.

Tested-by: Tony Lu <tonylu@linux.alibaba.com>

Thanks,
Tony Lu

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Leon Romanovsky]

On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote:
> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote:
> > From: Maor Gottlieb <maorg@nvidia.com>
> > 
> > The cited commit moved umem into the union, hence
> > umem could be accessed only for user MRs. Add udata check
> > before access umem in the dereg flow.
> > 
> > Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
> > Tested-by: Chuck Lever <chuck.lever@oracle.com>
> > Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> >  drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
> >  drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
> >  drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
> >  3 files changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> 
> This patch was tested and works for me in our environment for SMC. It
> wouldn't panic when release link and call ib_dereg_mr.
> 
> Tested-by: Tony Lu <tonylu@linux.alibaba.com>

Thanks, unfortunately, this patch is incomplete.

> 
> Thanks,
> Tony Lu

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker speaking.

On 03.01.22 10:51, Leon Romanovsky wrote:
> On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote:
>> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote:
>>> From: Maor Gottlieb <maorg@nvidia.com>
>>>
>>> The cited commit moved umem into the union, hence
>>> umem could be accessed only for user MRs. Add udata check
>>> before access umem in the dereg flow.
>>>
>>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
>>> Tested-by: Chuck Lever <chuck.lever@oracle.com>
>>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
>>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>>> ---
>>>  drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
>>>  drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
>>>  drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
>>>  3 files changed, 5 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
>>
>> This patch was tested and works for me in our environment for SMC. It
>> wouldn't panic when release link and call ib_dereg_mr.
>>
>> Tested-by: Tony Lu <tonylu@linux.alibaba.com>
> 
> Thanks, unfortunately, this patch is incomplete.

Could you be a bit more verbose and give a status update? It's hard to
follow from the outside. But according to the "Fixes: f0ae4afe3d35"
above this was supposed to fix a regression introduced in v5.16-rc5 that
was also reported here:
https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/

Commit f0ae4afe3d35 in fact was also backported to v5.15.y and might
cause trouble there as well.

Should it maybe simply be reverted (and reapplied with all fixes later)
in mainline (5.16 will likely be released in 6 days!) and v5.15.y?

Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat)

P.S.: As a Linux kernel regression tracker I'm getting a lot of reports
on my table. I can only look briefly into most of them. Unfortunately
therefore I sometimes will get things wrong or miss something important.
I hope that's not the case here; if you think it is, don't hesitate to
tell me about it in a public reply, that's in everyone's interest.

BTW, I have no personal interest in this issue, which is tracked using
regzbot, my Linux kernel regression tracking bot
(https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting
this mail to get things rolling again and hence don't need to be CC on
all further activities wrt to this regression.

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Leon Romanovsky]

On Mon, Jan 03, 2022 at 02:15:59PM +0100, Thorsten Leemhuis wrote:
> Hi, this is your Linux kernel regression tracker speaking.
> 
> On 03.01.22 10:51, Leon Romanovsky wrote:
> > On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote:
> >> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote:
> >>> From: Maor Gottlieb <maorg@nvidia.com>
> >>>
> >>> The cited commit moved umem into the union, hence
> >>> umem could be accessed only for user MRs. Add udata check
> >>> before access umem in the dereg flow.
> >>>
> >>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
> >>> Tested-by: Chuck Lever <chuck.lever@oracle.com>
> >>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
> >>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> >>> ---
> >>>  drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
> >>>  drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
> >>>  drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
> >>>  3 files changed, 5 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> >>
> >> This patch was tested and works for me in our environment for SMC. It
> >> wouldn't panic when release link and call ib_dereg_mr.
> >>
> >> Tested-by: Tony Lu <tonylu@linux.alibaba.com>
> > 
> > Thanks, unfortunately, this patch is incomplete.
> 
> Could you be a bit more verbose and give a status update? It's hard to
> follow from the outside. But according to the "Fixes: f0ae4afe3d35"
> above this was supposed to fix a regression introduced in v5.16-rc5 that
> was also reported here:
> https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/

The problematic commit f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
should be reverted https://lore.kernel.org/all/20211222101312.1358616-1-maorg@nvidia.com
and rewritten from the beginning.

There are three possible ways to rewrite that f0ae4afe3d35 commit, while
the nutshell of the problem is: "DM MR is user created verb object that
is missing very important property (umem) which is needed to distinguish
between kernel and user created objects."

The proposals are:
1. Return back to v1, which had dummy umem, so so DM memory regions will
behave as regular user created verbs object.
2. Add extra flag to is_user/is_kernel for mlx5 mr struct and update all
paths to rely on that flag.
3. Create separate dereg MR function that will treat DM differently.

We are waiting for Jason to return from vacation and express his
preference as he didn't like my preferred option #1.

Thanks

> 
> Commit f0ae4afe3d35 in fact was also backported to v5.15.y and might
> cause trouble there as well.
> 
> Should it maybe simply be reverted (and reapplied with all fixes later)
> in mainline (5.16 will likely be released in 6 days!) and v5.15.y?
> 
> Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat)
> 
> P.S.: As a Linux kernel regression tracker I'm getting a lot of reports
> on my table. I can only look briefly into most of them. Unfortunately
> therefore I sometimes will get things wrong or miss something important.
> I hope that's not the case here; if you think it is, don't hesitate to
> tell me about it in a public reply, that's in everyone's interest.
> 
> BTW, I have no personal interest in this issue, which is tracked using
> regzbot, my Linux kernel regression tracking bot
> (https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting
> this mail to get things rolling again and hence don't need to be CC on
> all further activities wrt to this regression.
> 

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Jason Gunthorpe]

On Mon, Jan 03, 2022 at 08:26:24PM +0200, Leon Romanovsky wrote:

> The proposals are:
> 1. Return back to v1, which had dummy umem, so so DM memory regions will
> behave as regular user created verbs object.
> 2. Add extra flag to is_user/is_kernel for mlx5 mr struct and update all
> paths to rely on that flag.
> 3. Create separate dereg MR function that will treat DM differently.

It is not DM that is the problem, ti is that user and kernel has been
mixed together in this mess despite being completely different.

I've been slowly disentangling them and the series you just sent 'MR
cache enhancment' removes the last blocker from completely giving
kernel MRs their own struct.

So, the solution here is to move in the direction of making the kernel
MRs different. There is only one place that destroys a kernel MR, just
have it call a special 'destroy kernel MR' function that doesn't touch
any umem stuff at all. Remove the kernel-only parts entirely from the
current function.

After Aharon's series we can give them different types. Notice the
union is already completely disjoint except for the little bit
tracking the cache which evaporates once the cache only stores the
mkey # and not the struct memory.

Jason

Re: [PATCH rdma-rc] RDMA/mlx5: Fix dereg mr flow for kernel MRs

[Thorsten Leemhuis]

On 03.01.22 19:26, Leon Romanovsky wrote:
> On Mon, Jan 03, 2022 at 02:15:59PM +0100, Thorsten Leemhuis wrote:
>> Hi, this is your Linux kernel regression tracker speaking.
>>
>> On 03.01.22 10:51, Leon Romanovsky wrote:
>>> On Wed, Dec 22, 2021 at 10:51:58AM +0800, Tony Lu wrote:
>>>> On Tue, Dec 21, 2021 at 11:46:41AM +0200, Leon Romanovsky wrote:
>>>>> From: Maor Gottlieb <maorg@nvidia.com>
>>>>>
>>>>> The cited commit moved umem into the union, hence
>>>>> umem could be accessed only for user MRs. Add udata check
>>>>> before access umem in the dereg flow.
>>>>>
>>>>> Fixes: f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
>>>>> Tested-by: Chuck Lever <chuck.lever@oracle.com>
>>>>> Signed-off-by: Maor Gottlieb <maorg@nvidia.com>
>>>>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>>>>> ---
>>>>>  drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +-
>>>>>  drivers/infiniband/hw/mlx5/mr.c      | 4 ++--
>>>>>  drivers/infiniband/hw/mlx5/odp.c     | 4 ++--
>>>>>  3 files changed, 5 insertions(+), 5 deletions(-)
>>>>>
>>>>> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
>>>>
>>>> This patch was tested and works for me in our environment for SMC. It
>>>> wouldn't panic when release link and call ib_dereg_mr.
>>>>
>>>> Tested-by: Tony Lu <tonylu@linux.alibaba.com>
>>>
>>> Thanks, unfortunately, this patch is incomplete.
>>
>> Could you be a bit more verbose and give a status update? It's hard to
>> follow from the outside. But according to the "Fixes: f0ae4afe3d35"
>> above this was supposed to fix a regression introduced in v5.16-rc5 that
>> was also reported here:
>> https://lore.kernel.org/linux-rdma/9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com/
> 
> The problematic commit f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
> should be reverted https://lore.kernel.org/all/20211222101312.1358616-1-maorg@nvidia.com
> and rewritten from the beginning.

Thx for the clarification. Is anyone tasked for sending the revert
upstream, to make sure the revert makes it into 5.16, which is due on
Sunday night?

And someone likely should ensure the change backported to 5.15.y as
e3bc4d4b50cae7db08e50dbe43f771c906e97701 is reverted as well. CCing a
few lists and Greg to make sure everyone is in the loop.

Ciao, Thorsten