Re: Kernel fast memory registration API proposal [RFC]

[Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]

On 7/14/2015 8:09 PM, Jason Gunthorpe wrote:
> On Tue, Jul 14, 2015 at 07:55:39PM +0300, Sagi Grimberg wrote:
>
>> But, if people think that it's better to have an API that does implicit
>> posting always without notification, and then silently consume error or
>> flush completions. I can try and look at it as well.
>
> Can we do FMR transparently if we bundle the post? If yes, I'd call
> that a winner..

Doing FMR transparently is not possible as the unmap flow is scheduling.
Unlike NFS, iSER unmaps from a soft-IRQ context, SRP unmaps from
hard-IRQ context. Changing the context to thread context is not
acceptable. The best we can do is using FMR_POOLs transparently.
Other than polluting the API and its semantics I suspect people will
have other problems with it (leaving the MRs open).

I suggest to start with what I proposed. And in a later stage (if we
still think its needed) we can have a higher level API that hides the
post, something like:

rdma_reg_sg(struct ib_qp *qp,
             struct ib_mr *mr,
             struct scatterlist *sg,
             int sg_nents,
             u64 offset,
             u64 length,
             int access_flags)

rdma_unreg_mr(struct ib_qp *qp,
               struct ib_mr *mr)


Or incorporate that with a pool API, something like:

rdma_create_fr_pool(struct ib_qp *qp,
                     int nmrs,
                     int mr_size,
                     int create_flags)

rdma_destroy_fr_pool(struct rdma_fr_pool *pool)

rdma_fr_reg_sg(struct rdma_fr_pool *pool,
                struct scatterlist *sg,
                int sg_nents,
                u64 offset,
                u64 length,
                int access_flags)

rdma_fr_unreg_mr(struct rdma_fr_pool *pool,
                  struct ib_mr *mr)


Note that I expect problems with both approaches, but
we can look into it...

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html