Re: [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, Mar 13, 2020 at 04:21:39AM -0700, Christoph Hellwig wrote:
> On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:
> > The non-page scatterlist is also a big concern for RDMA as we have
> > drivers that want the page list, so even if we did as this series
> > contemplates I'd have still have to split the drivers and create the
> > notion of a dma-only SGL.
> 
> The drivers I looked at want a list of IOVA address, aligned to the
> device "page size".  What other data do drivers want?  Execept for the
> software protocol stack drivers, which of couse need pages for the
> stack futher down.

In principle it is possible to have just an aligned page list -
however the page size is variable, following certain rules, and today
the drivers still determine the correct page size largely on their
own.  

Some progress was made recently to consolidate this, but more is
needed.

If the common code doesn't know the device page size in advance then
today's approach of sending largest possible dma mapped SGLs into the
device driver is best. The driver only has to do splitting.

> > I haven't used bio_vecs before, do they support chaining like SGL so
> > they can be very big? RDMA dma maps gigabytes of memory
> 
> bio_vecs itself don't have the chaining, but the bios build around them
> do.  But each entry can map a huge pile.  If needed we could use the
> same chaining scheme we use for scatterlists for bio_vecs as well, but
> lets see if we really end up needing that.

RDMA surely needs something to generate huge lists of dma mapped
memory. MRs are very big objects

Jason