Re: Kernel fast memory registration API proposal [RFC]

[Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]

On Jul 15, 2015, at 1:19 PM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote:

> On Wed, Jul 15, 2015 at 10:32:55AM -0400, Chuck Lever wrote:
> 
>> I would rather not build a non-deterministic delay into the
>> unmap interface. Using a pool or having map do an implicit
>> unmap are both solutions I’d rather avoid.
> 
> Can you explain how NFS is using FMR today? When does it unmap a FMR
> rkey and lkey?

Client side:

The content in RPC send buffers is sent with RDMA SEND using
local_dma_lkey, or with an lkey obtained from an MR allocated via
ib_dma_get_mr(). These are used to send RPC-over-RDMA headers
usually along with the content of small RPC requests.

These are left registered, essentially, during the lifetime of
the transport, and access to them is only local.

NFS READ and WRITE data payloads are mapped with ib_map_phys_mr()
just before the RPC is sent, and those payloads are unmapped
with ib_unmap_fmr() as soon as the client sees the server’s RPC
reply.

These memory regions require an rkey, which is sent in the RPC
call to the server. The server performs RDMA READ or WRITE on
these regions.

I don’t think the server ever uses FMR to register the target
memory regions for RDMA READ and WRITE.


> If NFS/etc currently have a hole on rkey invalidation when using FMR,
> and that hole simply cannot reasonably be solved, I'm actually mildly OK
> with enshrining that in a new MR API..
> 
> So, it would seem to me, the only major addition we'd need to Sagi's
> draft to support FMR, would be a way to catch the completion (the
> rdma_unreg_mr) and trigger async MR recycling async in a work queue.

As long as it is guaranteed that the unmap is scheduled as soon
as each RPC operation is complete, that might be tolerable, ie:

  rdma_unreg_mr_async()

Where the API hands the MR to a work queue to be unmapped, and
guarantees the MR cannot be reused until it knows it is unmapped.

I’m sure there’s a hole in there I’m missing.


> Sagi, how does cleanup of the temporary FRMR work in your draft
> proposal? What does the ULP do upon completion?
> 
> [Also, just mildly curious, how do we get into an unsleepable
> context anyhow? is the IB completion pending callback called in a
> sleepable context?]
> 
> Jason

--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html