archive mirror
 help / color / mirror / Atom feed
	Kaike Wan <>
Subject: [PATCH RFC 0/9] A rendezvous module
Date: Fri, 19 Mar 2021 08:56:26 -0400	[thread overview]
Message-ID: <> (raw)

From: Kaike Wan <>

RDMA transactions on RC QPs have a high demand for memory for HPC
applications such as MPI, especially on nodes with high cpu core
count, where a process is normally dispatched to each core and an RC QP
is created for each remote process. For a 100-node fabric,
about 10 GB - 100 GB memory is required for WQEs/Buffers/QP states on
each server node. This high demand imposes a severe restriction
on the scalability of HPC fabric.

A number of solutions have been implemented over the years. UD based
solutions solve the scalability problem by requiring only one UD
QP per process that can send a message to or receive a message from any
other process. However, it does not have the performance of an
RC QP and the application has to manage the segmentation for large

SRQ reduces the memory demand by sharing a receive queue among multiple
QPs. However, it still requires an RC QP for each remote
process and each RC QP still requires a send queue. In addition, it is
an optional feature.

XRC further reduces the memory demand by allowing a single QP per
process to communicate with any process on a remote node. In this
mode, each process requires only one QP for each remote node. Again,
this is optional and not all vendors support it.

Dynamically Connected transport minimizes the memory usage, but requires
vendor specific hardware and changes in applications.
Addtionally, all of these mechanisms can induce latency jitter due to
use of more QPs, more QP state, and hence additional
PCIe transactions at scale where NIC QP/CQ caches overflow.

Based on this, we are here proposing a generic, vendor-agnostic approach
to address the RC scalalibity and potentially improve RDMA
performance for larger messages which uses RDMA Write as part of a
rendezvous protocol.

Here are the key points of the proposal:
- For each device used by a job on a given node, there are a fixed
  number of RC connections (default: 4) between this node and any
  other remote node, no matter how many processes are running on each
  node for this job. That is, for a given job and device, all the
  processes on a given node will communicate with all the processes of
  the same job on a remote node through the same fixed number of
  connections. This eliminates the increased memory demand caused by
  core count increase and reduces the overall RC connection setup
- Memory region cache is added to reduce the MR
  registration/deregistration costs, as the same buffers tend to be used
  repeatedly by applications. The mr cache is per process and can be
  accessed only by processes in the same job.

Here is how we are proposing to implement this application enabling
rendezvous module that take advantage of the various features
of the RDMA subsystem:
- Create a rendezvous module (rv) under drivers/infiniband/ulp/rv/. This
  can be changed if a better location is recommended.
- The rv modules adds a char device file /dev/rv that user
  applications/middlewares can open to communicate with the rv module
  (PSM3 OFI provider in libfabric 1.12.0 being the primary use case).
  The file interface is crucial for associating the job/process
  with the mr, the connection endpoints, and RDMA requests.
  rv_file_open - Open a handle to the rv module for the current user
  rv_file_close - Close the handle and clean up.
  rv_file_ioctl - The main communication methods: to attach to a device,
                  to register/deregister mr, to create connection
                  endpoint, to connect, to query, to get stats, to do
                  RDMA transactions, etc.
  rv_file_mmap - Mmap an event ring into user space for the current user
- Basic mode of operations (PSM3 is used as an example for user
  - A middleware (like MPI) has out-of-band communication channels
    between any two nodes, which are used to establish high performance
    communications for providers such as PSM3.
  - On each node, the PSM3 provider opens the rv module, and issues an
    attach request to bind to a specific local device and specifies
    information specific to the job. This associates the user process
    with the RDMA device and the job.
  - On each node, the PSM3 provider will establish user space low
    latency eager and control message communications, typically via user
    space UD QPs.
  - On each node, the PSM3 provider will mmap a ring buffer for events
    from the rv module.
  - On each node, the PSM3 provider creates a set of connection
    endpoints by passing the destination info to the rv module. If the
    local node is chosen as the listener for the connection (based on
    address comparison between the two nodes), it will start to listen
    for any incoming IB CM connection requests and accept them when whey
    arrive. This step associates the shared connection endpoints with
    the job and device.
  - On each node, the PSM3 provider will request connection
    establishment through the rv module. On the client node, rv sends an
    IB CM request to the remote node. On the listener node, nothing
    additional needs to be done for the connect request from the PSM3
  - On each node, the PSM3 provider will wait until the RC connection is
    established. The PSM3 provider will query the rv module about the
    connection status periodically.
  - As large MPI messages are requested, the PSM3 provider will request
    rv to register MRs for the MPI application’s send and receive
    buffers.  The rv module makes use of its MR cache to limit when such
    requests need to interact with verbs MR calls for the NIC.
    The PSM3 provider control message mechanisms are used to exchange IO
    virtual addresses and rkeys for such buffers and MRs.
  - For RDMA transactions, the RDMA WRITE WITH IMMED request will be
    used. The immediate data will contain info about the target
    process on the remote node  and which outstanding rendezvous message
    this RDMA is for. For send completion and receive completion,
    an event will be posted into the event ring buffer and the PSM3
    provider can poll for completion.
  - As RDMA transactions complete, the PSM3 provider will indicate
    completion of the corresponding MPI send/receive to the MPI
    middleware/application. The PSM3 provider will also inform the rv
    module it may deregister the MR.  The deregistration allows rv
    to cache the MR for use in future requests by the PSM3 provider to
    register the same memory for use in another MPI message. The
    cached MR will be removed if the user buffer is freed and a MMU
    notifier event is received by the rv module.

Please comment,

Thank you,


Kaike Wan (9):
  RDMA/rv: Public interferce for the RDMA Rendezvous module
  RDMA/rv: Add the internal header files
  RDMA/rv: Add the rv module
  RDMA/rv: Add functions for memory region cache
  RDMA/rv: Add function to register/deregister memory region
  RDMA/rv: Add connection management functions
  RDMA/rv: Add functions for RDMA transactions
  RDMA/rv: Add functions for file operations
  RDMA/rv: Integrate the file operations into the rv module

 MAINTAINERS                                |    6 +
 drivers/infiniband/Kconfig                 |    1 +
 drivers/infiniband/core/core_priv.h        |    4 +
 drivers/infiniband/core/rdma_core.c        |  237 ++
 drivers/infiniband/core/uverbs_cmd.c       |   54 +-
 drivers/infiniband/ulp/Makefile            |    1 +
 drivers/infiniband/ulp/rv/Kconfig          |   11 +
 drivers/infiniband/ulp/rv/Makefile         |    9 +
 drivers/infiniband/ulp/rv/rv.h             |  892 ++++++
 drivers/infiniband/ulp/rv/rv_conn.c        | 3037 ++++++++++++++++++++
 drivers/infiniband/ulp/rv/rv_file.c        | 1196 ++++++++
 drivers/infiniband/ulp/rv/rv_main.c        |  271 ++
 drivers/infiniband/ulp/rv/rv_mr.c          |  393 +++
 drivers/infiniband/ulp/rv/rv_mr_cache.c    |  428 +++
 drivers/infiniband/ulp/rv/rv_mr_cache.h    |  152 +
 drivers/infiniband/ulp/rv/rv_rdma.c        |  416 +++
 drivers/infiniband/ulp/rv/trace.c          |    7 +
 drivers/infiniband/ulp/rv/trace.h          |   10 +
 drivers/infiniband/ulp/rv/trace_conn.h     |  547 ++++
 drivers/infiniband/ulp/rv/trace_dev.h      |   82 +
 drivers/infiniband/ulp/rv/trace_mr.h       |  109 +
 drivers/infiniband/ulp/rv/trace_mr_cache.h |  181 ++
 drivers/infiniband/ulp/rv/trace_rdma.h     |  254 ++
 drivers/infiniband/ulp/rv/trace_user.h     |  321 +++
 include/rdma/uverbs_types.h                |   10 +
 include/uapi/rdma/rv_user_ioctls.h         |  558 ++++
 26 files changed, 9139 insertions(+), 48 deletions(-)
 create mode 100644 drivers/infiniband/ulp/rv/Kconfig
 create mode 100644 drivers/infiniband/ulp/rv/Makefile
 create mode 100644 drivers/infiniband/ulp/rv/rv.h
 create mode 100644 drivers/infiniband/ulp/rv/rv_conn.c
 create mode 100644 drivers/infiniband/ulp/rv/rv_file.c
 create mode 100644 drivers/infiniband/ulp/rv/rv_main.c
 create mode 100644 drivers/infiniband/ulp/rv/rv_mr.c
 create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.c
 create mode 100644 drivers/infiniband/ulp/rv/rv_mr_cache.h
 create mode 100644 drivers/infiniband/ulp/rv/rv_rdma.c
 create mode 100644 drivers/infiniband/ulp/rv/trace.c
 create mode 100644 drivers/infiniband/ulp/rv/trace.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_conn.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_dev.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_mr.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_mr_cache.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_rdma.h
 create mode 100644 drivers/infiniband/ulp/rv/trace_user.h
 create mode 100644 include/uapi/rdma/rv_user_ioctls.h


             reply	other threads:[~2021-03-19 12:57 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-19 12:56 kaike.wan [this message]
2021-03-19 12:56 ` [PATCH RFC 1/9] RDMA/rv: Public interferce for the RDMA Rendezvous module kaike.wan
2021-03-19 16:00   ` Jason Gunthorpe
2021-03-19 12:56 ` [PATCH RFC 2/9] RDMA/rv: Add the internal header files kaike.wan
2021-03-19 16:02   ` Jason Gunthorpe
2021-03-19 12:56 ` [PATCH RFC 3/9] RDMA/rv: Add the rv module kaike.wan
2021-03-19 12:56 ` [PATCH RFC 4/9] RDMA/rv: Add functions for memory region cache kaike.wan
2021-03-19 12:56 ` [PATCH RFC 5/9] RDMA/rv: Add function to register/deregister memory region kaike.wan
2021-03-19 12:56 ` [PATCH RFC 6/9] RDMA/rv: Add connection management functions kaike.wan
2021-03-19 12:56 ` [PATCH RFC 7/9] RDMA/rv: Add functions for RDMA transactions kaike.wan
2021-03-19 12:56 ` [PATCH RFC 8/9] RDMA/rv: Add functions for file operations kaike.wan
2021-03-19 12:56 ` [PATCH RFC 9/9] RDMA/rv: Integrate the file operations into the rv module kaike.wan
2021-03-19 13:53 ` [PATCH RFC 0/9] A rendezvous module Jason Gunthorpe
2021-03-19 14:49   ` Wan, Kaike
2021-03-19 15:48     ` Jason Gunthorpe
2021-03-19 19:22       ` Dennis Dalessandro
2021-03-19 19:44         ` Jason Gunthorpe
2021-03-19 20:12           ` Rimmer, Todd
2021-03-19 20:26             ` Jason Gunthorpe
2021-03-19 20:46               ` Rimmer, Todd
2021-03-19 20:54                 ` Jason Gunthorpe
2021-03-19 20:59                   ` Wan, Kaike
2021-03-19 21:28                     ` Dennis Dalessandro
2021-03-19 21:58                       ` Wan, Kaike
2021-03-19 22:35                         ` Jason Gunthorpe
2021-03-19 22:57                       ` Rimmer, Todd
2021-03-19 23:06                         ` Jason Gunthorpe
2021-03-20 16:39                         ` Dennis Dalessandro
2021-03-21  8:56                           ` Leon Romanovsky
2021-03-21 16:24                             ` Dennis Dalessandro
2021-03-21 16:45                               ` Jason Gunthorpe
2021-03-21 17:21                                 ` Dennis Dalessandro
2021-03-21 18:08                                   ` Jason Gunthorpe
2021-03-22 15:17                                     ` Rimmer, Todd
2021-03-22 16:47                                       ` Jason Gunthorpe
2021-03-22 17:31                                     ` Hefty, Sean
2021-03-23 22:56                                       ` Jason Gunthorpe
2021-03-23 23:29                                         ` Rimmer, Todd
2021-03-21 19:19                                   ` Wan, Kaike
2021-03-23 15:36                                   ` Christoph Hellwig
2021-03-23 15:35                                 ` Christoph Hellwig
2021-03-23 15:33                               ` Christoph Hellwig
2021-03-23 15:30                         ` Christoph Hellwig
2021-03-23 15:46                           ` Jason Gunthorpe
2021-03-23 16:07                             ` Christoph Hellwig
2021-03-23 17:25                               ` Rimmer, Todd
2021-03-23 17:44                                 ` Jason Gunthorpe
2021-03-19 20:18           ` Dennis Dalessandro
2021-03-19 20:30             ` Jason Gunthorpe
2021-03-19 20:34       ` Hefty, Sean
2021-03-21 12:08         ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).