Re: [RFC] bulk zero copy transport

[Stefan Metzmacher <metze@samba.org>]

Hi Dennis,

just as a wild idea, would be an option to use the SMB-Direct [1] protocol defined here?

It basically provides a stream/packet like transport based on IB_WR_SEND[_WITH_INV]
and in addition it allows direct memory transfers with IB_WR_RDMA_READ or IB_WR_RDMA_WRITE.

It's called SMB-Direct as it's currently used as a transport for the SMB3 protocol,
but it could also be used as transport for other things.

Over the last years I've been working on a PF_SMBDIRECT socket driver [2]
as a hobby project in order to support it for Samba. It's not yet production ready
and has known memory leaks, but the basics already work. The api [3] is based on
sendmsg/recvmsg with using MSG_OOB with msg->msg_control for direct memory transfers.
I'll actually use it with IORING_OP_SENDMSG and IORING_OP_RECVMSG, which allow msg->msg_control
starting 5.12 kernels.

metze

[1] https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-SMBD/%5bMS-SMBD%5d.pdf
[2] https://git.samba.org/?p=metze/linux/smbdirect.git;a=summary
[3] https://git.samba.org/?p=metze/linux/smbdirect.git;a=blob;f=smbdirect.h;hb=refs/heads/smbdirect-work-in-progress


Am 19.08.21 um 21:09 schrieb Dennis Dalessandro:

> Just wanted to float an idea we are thinking about. It builds on the basic idea
> of what Intel submitted as their RV module [1]. This however does things a bit
> differently and is really all about bulk zero-copy using the kernel. It is a new
> ULP.
> 
> The major differences are that there will be no new cdev needed. We will make
> use of the existing HFI1 cdev where an FD is needed. We also propose to make use
> of IO-Uring (hence needing FD) to get requests into the kernel. The idea will be
> to not share Uverbs objects with the kernel. The kernel will maintain
> ownership of the qp, pd, mr, cq, etc.
> 
> Connections we envision to be maintained by the kernel using RDMA CM. Similar in
> fashion to how RDS or IPoIB works. This of course means an RC QP which allows
> our TID RDMA feature to work under the hood.
> 
> We have looked into RDS and RTRS and both seem to be the wrong interface. RDS
> provides a lot of what we are looking for but it seems to be a bit overkill and
> has higher overhead than we hope to achieve. Performance results show it to be
> less performant than direct to verbs.
> 
> After reviewing the RV submission, I don't think there is any reason to try to
> revamp that submission. It seems to be very tightly tied to PSM3 whereas this is
> meant to be more generic.
> 
> At this point we are interested in what questions you would have or opinions. We
> would like to get some feedback early in the process. As we develop the code
> we'll continue to post, similar to how we did rdmavt and welcome anyone that
> wants to collaborate.
> 
> [1] https://lore.kernel.org/linux-rdma/20210319125635.34492-1-kaike.wan@intel.com/
> 
> -Denny
>