From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Talpey Subject: Re: Kernel fast memory registration API proposal [RFC] Date: Tue, 14 Jul 2015 13:29:37 -0400 Message-ID: <55A54701.6080404@talpey.com> References: <559F8BD1.9080308@dev.mellanox.co.il> <20150713163015.GA23832@obsidianresearch.com> <55A4CABC.5050807@dev.mellanox.co.il> <20150714153347.GA11026@infradead.org> <55A534D1.6030008@dev.mellanox.co.il> <00ab01d0be50$78587ab0$69097010$@opengridcomputing.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <00ab01d0be50$78587ab0$69097010$@opengridcomputing.com> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Steve Wise , 'Sagi Grimberg' , 'Christoph Hellwig' Cc: 'Jason Gunthorpe' , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, 'Or Gerlitz' , 'Oren Duer' , 'Chuck Lever' , 'Bart Van Assche' , 'Liran Liss' , "'Hefty, Sean'" , 'Doug Ledford' List-Id: linux-rdma@vger.kernel.org On 7/14/2015 12:16 PM, Steve Wise wrote: > > >> -----Original Message----- >> From: Sagi Grimberg [mailto:sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org] >> Sent: Tuesday, July 14, 2015 11:12 AM >> To: Christoph Hellwig >> Cc: Jason Gunthorpe; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Steve Wise; Or Gerlitz; Oren Duer; Chuck Lever; Bart Van Assche; Liran Liss; > Hefty, >> Sean; Doug Ledford; Tom Talpey >> Subject: Re: Kernel fast memory registration API proposal [RFC] >> >> On 7/14/2015 6:33 PM, Christoph Hellwig wrote: >>> On Tue, Jul 14, 2015 at 11:39:24AM +0300, Sagi Grimberg wrote: >>>> This is exactly what I don't want to do. I don't think that implicit >>>> posting is a good idea for reasons that I mentioned earlier: >>>> >>>> "This is where I have a problem. Providing an API that may or may not >>>> post a work request on my QP is confusing, and I don't understand its >>>> semantics at all. Do I need to reserve slots on my QP? should I ask for >>>> a completion? If we suppress the completion will I see an error >>>> completion? What should I expect to find in the wr_id?" >>>> >>>> We're much better off with keeping the post interface in place but >>>> have it much simpler. >>> >>> The ULP doesn't care if it needs to reserver the slot, and it generally >>> doesn't care about the notification either unless it needs to handle an >>> error. >> >> That's generally correct. But the ULP may want to suppress notifications >> of other posts as well (for example a storage initiator may want to >> suppress send completions since it needs to reserve the request context >> until it receives the status anyway - which is a receive completion). It >> needs to take what was posted and what not into account to avoid >> wrapping. >> >> If I'm not mistaken iWARP requires to ask for a completion once every >> send-queue length and empirically, IB drivers requires it too. So again, > > Correct. Indeed, correct. But this has nothing to do with the protocol. It's necessary because without a completion, the driver can't keep the work queue and completion queue pointers properly in sync with the hardware. If the tail pointer catches up and crosses the head pointer, boom. This is a pure verbs issue, completely local behavior, and applies to all providers. >> I don't think implicit posting is a very good idea. Specifically, it will work only if completions are never suppressed, if the upper layer is aware of the additional work requests and their ordering implications, and the overhead of the interrupts is acceptable. None of these preconditions are desirable. >> >>> The completions are another mad mightmare in the RDMA stack API. Every >>> other subsystem would just allow submitter to attach a function pointer >>> that handles the completion to the posted item. >> >> This is actually a good idea. And it can be easily added I think (at >> least to mlx drivers which I better). It can help removing a switch >> statement for the ULPs when handling the completions. >> >> But no, the RDMA stack >>> instead require ID allocators and gigantic boilerplate code in the >>> consumer to untangle that gigantic mess again. >> >> I don't think the wr_id was ever intended to be an allocated tag. It's >> the ULP context, usually a pointer to a transaction structure. Even >> with passing a function pointer you need to get back your original >> context don't you? > > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html