Re: Kernel fast memory registration API proposal [RFC]

[Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]

On Jul 12, 2015, at 3:57 AM, Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:

> On 7/11/2015 1:39 PM, Christoph Hellwig wrote:
>> On Fri, Jul 10, 2015 at 12:09:37PM +0300, Sagi Grimberg wrote:
>>> And then provide helpers to populate the MR with generic kernel
>>> structures such as struct scatterlist (for scsi and other ULPs),
>>> struct page (for NFS) or struct bio_vec (for block ULPs later on).
>> 
>> Please stick to struct scatterlist for now.  Future block ULPs
>> will use that as well as the only way you can do a multi-page
>> DMA mapping is the scatterlist.
> 
> I see.
> 
>> A page is just a subset of an SGL, and we can map a page using a one element SGL trivial,
>> as we do in lots of places.
>> 
> 
> But won't that make sunrpc rdma consumers to hold an extra scatterlist
> just for memory registration?

Yes. xprtrdma’s FMR implementation already has a physaddrs array
for this purpose, and it’s FRWR implementation keeps an
ib_fast_reg_page_list array for each MR. 


> Chuck, Would a scatterlist API make life easier for you?

No benefit for me.

The NFS upper layer already slices and dices I/O until it is a
stream of contiguous single I/O requests for the server.

It passes down a vector of struct page pointers which xprtrdma’s
memory registration logic has to walk through and convert into
something the provider can deal with.

See fmr_op_map and frwr_op_map. The loop in there becomes costly
as the number of pages involved in an I/O request increases.

I suppose an s/g list wouldn’t be much different from the current
arrangement. And if NFS and SunRPC are the only users that deal
with struct page, then there’s no code sharing benefit to
providing a provider API based on struct page.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html