From: Stefan Metzmacher <firstname.lastname@example.org>
To: Dennis Dalessandro <email@example.com>,
Linux RDMA <firstname.lastname@example.org>,
Jason Gunthorpe <email@example.com>,
Doug Ledford <firstname.lastname@example.org>
Subject: Re: [RFC] bulk zero copy transport
Date: Fri, 20 Aug 2021 10:18:39 +0200 [thread overview]
Message-ID: <email@example.com> (raw)
just as a wild idea, would be an option to use the SMB-Direct  protocol defined here?
It basically provides a stream/packet like transport based on IB_WR_SEND[_WITH_INV]
and in addition it allows direct memory transfers with IB_WR_RDMA_READ or IB_WR_RDMA_WRITE.
It's called SMB-Direct as it's currently used as a transport for the SMB3 protocol,
but it could also be used as transport for other things.
Over the last years I've been working on a PF_SMBDIRECT socket driver 
as a hobby project in order to support it for Samba. It's not yet production ready
and has known memory leaks, but the basics already work. The api  is based on
sendmsg/recvmsg with using MSG_OOB with msg->msg_control for direct memory transfers.
I'll actually use it with IORING_OP_SENDMSG and IORING_OP_RECVMSG, which allow msg->msg_control
starting 5.12 kernels.
Am 19.08.21 um 21:09 schrieb Dennis Dalessandro:
> Just wanted to float an idea we are thinking about. It builds on the basic idea
> of what Intel submitted as their RV module . This however does things a bit
> differently and is really all about bulk zero-copy using the kernel. It is a new
> The major differences are that there will be no new cdev needed. We will make
> use of the existing HFI1 cdev where an FD is needed. We also propose to make use
> of IO-Uring (hence needing FD) to get requests into the kernel. The idea will be
> to not share Uverbs objects with the kernel. The kernel will maintain
> ownership of the qp, pd, mr, cq, etc.
> Connections we envision to be maintained by the kernel using RDMA CM. Similar in
> fashion to how RDS or IPoIB works. This of course means an RC QP which allows
> our TID RDMA feature to work under the hood.
> We have looked into RDS and RTRS and both seem to be the wrong interface. RDS
> provides a lot of what we are looking for but it seems to be a bit overkill and
> has higher overhead than we hope to achieve. Performance results show it to be
> less performant than direct to verbs.
> After reviewing the RV submission, I don't think there is any reason to try to
> revamp that submission. It seems to be very tightly tied to PSM3 whereas this is
> meant to be more generic.
> At this point we are interested in what questions you would have or opinions. We
> would like to get some feedback early in the process. As we develop the code
> we'll continue to post, similar to how we did rdmavt and welcome anyone that
> wants to collaborate.
>  https://firstname.lastname@example.org/
next prev parent reply other threads:[~2021-08-20 8:18 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-19 19:09 [RFC] bulk zero copy transport Dennis Dalessandro
2021-08-19 23:01 ` Jason Gunthorpe
2021-08-20 12:55 ` Dennis Dalessandro
2021-08-20 8:18 ` Stefan Metzmacher [this message]
2021-08-20 12:37 ` Dennis Dalessandro
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).