archive mirror
 help / color / mirror / Atom feed
From: Dennis Dalessandro <>
To: Linux RDMA <>,
	Jason Gunthorpe <>,
	Doug Ledford <>
Subject: [RFC] bulk zero copy transport
Date: Thu, 19 Aug 2021 15:09:02 -0400	[thread overview]
Message-ID: <> (raw)

Just wanted to float an idea we are thinking about. It builds on the basic idea
of what Intel submitted as their RV module [1]. This however does things a bit
differently and is really all about bulk zero-copy using the kernel. It is a new

The major differences are that there will be no new cdev needed. We will make
use of the existing HFI1 cdev where an FD is needed. We also propose to make use
of IO-Uring (hence needing FD) to get requests into the kernel. The idea will be
to not share Uverbs objects with the kernel. The kernel will maintain
ownership of the qp, pd, mr, cq, etc.

Connections we envision to be maintained by the kernel using RDMA CM. Similar in
fashion to how RDS or IPoIB works. This of course means an RC QP which allows
our TID RDMA feature to work under the hood.

We have looked into RDS and RTRS and both seem to be the wrong interface. RDS
provides a lot of what we are looking for but it seems to be a bit overkill and
has higher overhead than we hope to achieve. Performance results show it to be
less performant than direct to verbs.

After reviewing the RV submission, I don't think there is any reason to try to
revamp that submission. It seems to be very tightly tied to PSM3 whereas this is
meant to be more generic.

At this point we are interested in what questions you would have or opinions. We
would like to get some feedback early in the process. As we develop the code
we'll continue to post, similar to how we did rdmavt and welcome anyone that
wants to collaborate.



             reply	other threads:[~2021-08-19 19:09 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-19 19:09 Dennis Dalessandro [this message]
2021-08-19 23:01 ` [RFC] bulk zero copy transport Jason Gunthorpe
2021-08-20 12:55   ` Dennis Dalessandro
2021-08-20  8:18 ` Stefan Metzmacher
2021-08-20 12:37   ` Dennis Dalessandro

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).