Re: [PATCH for-next] RDMA/rxe: Let rdma-core manage PDs

[Bob Pearson <rpearsonhpe@gmail.com>]

On 7/27/21 7:02 AM, Leon Romanovsky wrote:
> On Tue, Jul 27, 2021 at 06:31:59AM -0500, Bob Pearson wrote:
>> On 7/27/21 6:30 AM, Leon Romanovsky wrote:
>>> On Mon, Jul 26, 2021 at 04:58:16PM -0500, Bob Pearson wrote:
>>>> Currently several rxe objects hold references to PDs which are ref-
>>>> counted. This replicates work already done by RDMA core which takes
>>>> references to PDs in the ib objects which are contained in the rxe
>>>> objects. This patch removes struct rxe_pd from rxe objects and removes
>>>> reference counting for PDs except for PD alloc and PD dealloc. It also
>>>> adds inline extractor routines which return PDs from the PDs in the
>>>> ib objects. The names of these are made consistent including a rxe_
>>>> prefix.
>>>>
>>>> Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>
>>>> ---
>>>>  drivers/infiniband/sw/rxe/rxe_comp.c  |  4 ++--
>>>>  drivers/infiniband/sw/rxe/rxe_loc.h   |  4 ++--
>>>>  drivers/infiniband/sw/rxe/rxe_mr.c    |  8 +++----
>>>>  drivers/infiniband/sw/rxe/rxe_mw.c    | 31 +++++++++++----------------
>>>>  drivers/infiniband/sw/rxe/rxe_qp.c    |  9 +-------
>>>>  drivers/infiniband/sw/rxe/rxe_req.c   |  2 +-
>>>>  drivers/infiniband/sw/rxe/rxe_resp.c  |  4 ++--
>>>>  drivers/infiniband/sw/rxe/rxe_verbs.c | 26 ++++++----------------
>>>>  drivers/infiniband/sw/rxe/rxe_verbs.h | 24 +++++++++++++++------
>>>>  9 files changed, 48 insertions(+), 64 deletions(-)
>>>
>>> Last time when I looked on it, I came to conclusion that all RXE
>>> references can be dropped.
>>>
>>> Thanks
>>>
>> This is a step in that direction. There are more coming.
> 
> Glad to hear, thank you for your work.
> 
>> Regards,
>>
>> Bob

The other ones I can immediately get rid of are AHs, CQs and SRQs.

The ones I think may require ref counting are MRs, MWs, QPs, and XRCSRQs. Each of these
get looked up from information in RoCE packets from rkeys, qpns, and srq_nums, For reliable transports
on slow networks this can require that these objects hang around for a while and the user has no
visibility to this unless there is a completion event waiting but not for e.g. reads, writes or atomics on the target side. There can be races between destroying these objects and messages completing causing
kernel oops. Do you know another way to address these cases?

Bob