Re: [PATCH WIP 28/43] IB/core: Introduce new fast registration API

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Sun, Jul 26, 2015 at 12:37:55PM +0300, Sagi Grimberg wrote:
> I guess I can pull DMA mapping in there, but we will need an opposite
> routine ib_umap_mr_sg() since it'll be weird if the ULP will do dma
> unmap without doing the map...

Yes, and ideally it would help ULPs to order these operations
properly. eg we shouldn't be abusing the DMA API and unmapping before
invalidate completes by default. That breaks obscure stuff in various
ways...

> >The biggest problem with checking the virtual address is
> >swiotlb. However, if swiotlb is used this API is basically broken as
> >swiotlb downgrades everything to a 2k alignment, which means we only
> >ever get 1 s/g entry.
> 
> Can you explain what do you mean by "downgrades everything to a 2k
> alignment"? If the ULP is responsible for a PAGE_SIZE alignment than
> how would this get out of alignment with swiotlb?

swiotlb copies all DMA maps to a shared buffer below 4G so it can be
used with 32 bit devices.

The shared buffer is managed in a way that copies each s/g element to
a continuous 2k aligned subsection of the buffer.

Basically, swiotlb realigns everything that passes through it.

The DMA API allows this, so ultimately, code has to check the dma
physical address when concerned about alignment.. But we should not
expect this to commonly fail.

So, something like..

  if (!ib_does_sgl_fit_in_mr(mr,sg))
     .. bounce buffer ..
     
  if (!ib_map_mr_sg(mr,sg)) // does dma mapping and checks it
     .. bounce buffer ..
     
  .. post ..
  .. send invalidate ..
  .. catch invalidate completion ...

  ib_unmap_mr(mr); // does dma unmap

?
 
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html