Re: Kernel fast memory registration API proposal [RFC]

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Tue, Jul 14, 2015 at 07:12:01PM +0300, Sagi Grimberg wrote:
> >The ULP doesn't care if it needs to reserver the slot, and it generally
> >doesn't care about the notification either unless it needs to handle an
> >error.
> 
> That's generally correct. But the ULP may want to suppress notifications
> of other posts as well (for example a storage initiator may want to
> suppress send completions since it needs to reserve the request context
> until it receives the status anyway - which is a receive completion). It
> needs to take what was posted and what not into account to avoid
> wrapping.

I don't see how this is a concern, API wise. All callers are paring
the MR post with a real op anyhow.

> If I'm not mistaken iWARP requires to ask for a completion once every
> send-queue length and empirically, IB drivers requires it too. So again,
> I don't think implicit posting is a very good idea.

Yes, all proper use of RDMA requires counting SQEs consumed and
freeing them only on completion. Nobody should be using the EAGAIN
scheme. But it is trivial to handle, you don't start an op until you
can post the entire compound SQE, and you must always tag the last SQE
for signal if the # of SQEs left drops below the # required for the
largest compound (or another similar scheme).

But you should never have to signal on the MR post, it should always
be at the end of the compound, on the RDMA, SEND, or invalidate..

That is just basic 'how do you use a SQ properly', every ULP should
do it, wrappering posts in an API doesn't change anything fundamental.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html