Fwd: [RFC PATCH 0/3] RDMA/rxe: Add dma-buf support

[Bob Pearson <rpearsonhpe@gmail.com>]

-------- Forwarded Message --------
Subject: Re: [RFC PATCH 0/3] RDMA/rxe: Add dma-buf support
Date: Wed, 8 Sep 2021 22:23:53 -0500
From: Bob Pearson <rpearsonhpe@gmail.com>
To: Shunsuke Mie <mie@igel.co.jp>
CC: Jason Gunthorpe <jgg@nvidia.com>

On 9/8/21 9:17 PM, Shunsuke Mie wrote:
> Hi Bob,
> 
> I agree with you. The fix has to be merged firstly. I'd like to know
> the patches.
> Is the patch for that bug fix at the following link?
> https://lore.kernel.org/linux-rdma/20210908052928.17375-1-rpearsonhpe@gmail.com/
> 
> Thanks,
> 
> 2021年9月9日(木) 7:52 Bob Pearson <rpearsonhpe@gmail.com>:
>>
>> On 9/8/21 1:16 AM, Shunsuke Mie wrote:
>>> This patch series add a support for rxe driver.
>>>
>>> A dma-buf based memory registering has beed introduced to use the memory
>>> region that lack of associated page structures (e.g. device memory and CMA
>>> managed memory) [1]. However, to use the dma-buf based memory, each rdma
>>> device drivers require add some implementation. The rxe driver has not
>>> support yet.
>>>
>>> [1] https://www.spinics.net/lists/linux-rdma/msg98592.html
>>>
>>> To enable to use the memories in rxe rdma device, add some changes and
>>> implementation in this patch series.
>>>
>>> This series consists of three patches. The first patch changes the IB core
>>> to support for rdma drivers that have not real dma device. The second
>>> patch extracts a memory mapping process of rxe as a common function to use
>>> a dma-buf support. The third patch adds the dma-buf support to rxe driver.
>>>
>>> Related user space RDMA library changes are provided as a separate
>>> patch.
>>>
>>> Shunsuke Mie (3):
>>>   RDMA/umem: Change for rdma devices has not dma device
>>>   RDMA/rxe: Extract a mapping process into a function
>>>   RDMA/rxe: Support dma-buf as memory region
>>>
>>>  drivers/infiniband/core/umem_dmabuf.c |   2 +-
>>>  drivers/infiniband/sw/rxe/rxe_loc.h   |   3 +
>>>  drivers/infiniband/sw/rxe/rxe_mr.c    | 186 +++++++++++++++++++++-----
>>>  drivers/infiniband/sw/rxe/rxe_verbs.c |  36 +++++
>>>  4 files changed, 193 insertions(+), 34 deletions(-)
>>>
>>
>> Mie,
>>
>> This looks very interesting. But, I am afraid we are going to collide. I just submitted some patches
>> to fix blktest which will very likely overlap with yours. Please take a look. It should be easy to
>> merge them. I am anxious to clear the blktest bug ASAP though.
>>
>> Bob Pearson

Mie,

I just forwarded to you the emails in the patch series. 1/5 and 2/5 are mostly elsewhere. But 2-5/5 are directly changing rxe_mr.c and especially 4/5 touches the SG to page list code. Per a recent exchange
with Jason we needed to keep two copies of the buffer list so that the kernel space API could be
used as intended and allow overlap of invalidating one MR while a new one is getting built and then
registered. In case, you can look at all the emails at https://lore.kernel.org/linux-rdma/.

Bob