[RFC] RDMA with Continuous Memory Allocator

[RFC] RDMA with Continuous Memory Allocator

[Shunsuke Mie]

Hi all,

I tried to use Continuous Memory Allocator (CMA) allocated memory for
RDMA transfer buffer in userspace, but it failed.

For more details, an ibv_reg_mr() API fails when I pass an mmaped
memory region of allocated memory by the CMA. The reason why, a CMA
mmap function, __dma_mmap_from_coherent(), sets  VM_IO and VM_PFNMAP
to vma->vm_flags, and an ib_uverbs_reg_mr(), kernel function
corresponding to the ibv_reg_mr(), tries to pin the memory region but,
it becomes to fail. because VM_IO or VM_PFNMAP regions cannot be
pined. As a result, the ibv_reg_mr() returns an error.

I think the ib_umem_get() that is called in the ib_uverbs_reg_mr() and
pins memories needs some modifications to support RDMA transfer
to/from the CMA memory.

I’d like to know your comments, ideas, and other solution.

Thanks a lot,
Shunsuke.

Re: [RFC] RDMA with Continuous Memory Allocator

[Leon Romanovsky]

On Fri, Jul 09, 2021 at 07:24:48PM +0900, Shunsuke Mie wrote:
> Hi all,
> 
> I tried to use Continuous Memory Allocator (CMA) allocated memory for
> RDMA transfer buffer in userspace, but it failed.

Sorry for my question, but why do you need it?
From that I remember, CMA memory is used for the devices that doesn't
support scatter-gather, while RDMA devices (umem) need SG.

Thanks

> 
> For more details, an ibv_reg_mr() API fails when I pass an mmaped
> memory region of allocated memory by the CMA. The reason why, a CMA
> mmap function, __dma_mmap_from_coherent(), sets  VM_IO and VM_PFNMAP
> to vma->vm_flags, and an ib_uverbs_reg_mr(), kernel function
> corresponding to the ibv_reg_mr(), tries to pin the memory region but,
> it becomes to fail. because VM_IO or VM_PFNMAP regions cannot be
> pined. As a result, the ibv_reg_mr() returns an error.
> 
> I think the ib_umem_get() that is called in the ib_uverbs_reg_mr() and
> pins memories needs some modifications to support RDMA transfer
> to/from the CMA memory.
> 
> I’d like to know your comments, ideas, and other solution.
> 
> Thanks a lot,
> Shunsuke.

Re: [RFC] RDMA with Continuous Memory Allocator

[Shunsuke Mie]

Hi Leon,
Thank you for your reply.

> Sorry for my question, but why do you need it?
I'd like to write data to the buffer, prepared by DRM (gpu) driver as
a frame buffer, using RDMA.
There is a similar project as follows.
https://www.openfabrics.org/wp-content/uploads/2020-workshop-presentations/303.-OFI-GPU-DMA-BUF-OFA2020v2.pdf
They prepare a mechanism to share a dmabuf fd between DRM driver and
RDMA driver, in order to update frame buffer using RDMA. I'm trying to
develop that in userland.

Some DRM drivers use CMA to allocate the buffer. I met the problem in
an environment that CMA used.

> From that I remember, CMA memory is used for the devices that doesn't
> support scatter-gather, while RDMA devices (umem) need SG.
Yes, the CMA memory is allocated for a device that doesn't support
scatter-gather.

Thanks,

Re: [RFC] RDMA with Continuous Memory Allocator

[Leon Romanovsky]

On Tue, Jul 13, 2021 at 03:57:53PM +0900, Shunsuke Mie wrote:
> Hi Leon,
> Thank you for your reply.
> 
> > Sorry for my question, but why do you need it?
> I'd like to write data to the buffer, prepared by DRM (gpu) driver as
> a frame buffer, using RDMA.
> There is a similar project as follows.
> https://www.openfabrics.org/wp-content/uploads/2020-workshop-presentations/303.-OFI-GPU-DMA-BUF-OFA2020v2.pdf
> They prepare a mechanism to share a dmabuf fd between DRM driver and
> RDMA driver, in order to update frame buffer using RDMA. I'm trying to
> develop that in userland.
> 
> Some DRM drivers use CMA to allocate the buffer. I met the problem in
> an environment that CMA used.

In some flows, dma-buf pins memory under the hood.

Anyway, for regular umem, you can try to use mlx5 that supports on
demand paging (ODP). That feature allows you to create umem without
need to pin pages.

Thanks