Re: [RFC] libibverbs IB Device Memory support

[Logan Gunthorpe <logang-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>]

Hi,

On 06/06/17 01:35 AM, Sagi Grimberg wrote:
> This is useful to kernel consumers too, Adding Stephen and Logan to the
> party.

Thanks for copying us.

> Personally I agree with Christoph Lamater that we are far better off
> sticking to standard mmap to expose it to user-space.
> From the thread I don't really understand the problem the API is trying
> to solve, AFAICT it boils down to a plain bar, and as such it needs to
> be treated as such.

We always get push back on IO safety (and for good reason). If the PCI
bar is exposed to userspace through mmap, users should be reading and
writing it through IO operations and if the buffers get used in any
kernel calls, the kernel also has to know that it is now dealing with IO
memory. The kernel is really not ready for this. A kernel-only API would
be able to do this safely and may make sense here.

However, without being able to do P2P DMAs with this memory (which is
what we have been working on) I don't really see the point. My intuition
is that copying from system to IO memory, or having the CPU populate
data structures in IO memory, would overall be slower than just doing it
in system memory and letting the DMA transfer be slightly slower.

Also, I have been told that, at least the two major RDMA NIC vendors
don't currently have working memory available in hardware to expose for
such uses. If there is hardware, I sure would like to get my hands on some.

In any case, this all seems slightly premature.

Logan




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html