RE: [PATCH RFC 0/9] A rendezvous module

From: "Wan, Kaike" <kaike.wan@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "dledford@redhat.com" <dledford@redhat.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"Rimmer, Todd" <todd.rimmer@intel.com>
Subject: RE: [PATCH RFC 0/9] A rendezvous module
Date: Fri, 19 Mar 2021 14:49:29 +0000	[thread overview]
Message-ID: <SN6PR11MB33115FD9F1F1D6122A9522C0F4689@SN6PR11MB3311.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20210319135302.GS2356281@nvidia.com>

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, March 19, 2021 9:53 AM
> To: Wan, Kaike <kaike.wan@intel.com>
> Cc: dledford@redhat.com; linux-rdma@vger.kernel.org; Rimmer, Todd
> <todd.rimmer@intel.com>
> Subject: Re: [PATCH RFC 0/9] A rendezvous module
> 
> On Fri, Mar 19, 2021 at 08:56:26AM -0400, kaike.wan@intel.com wrote:
> 
> > - Basic mode of operations (PSM3 is used as an example for user
> >   applications):
> >   - A middleware (like MPI) has out-of-band communication channels
> >     between any two nodes, which are used to establish high performance
> >     communications for providers such as PSM3.
> 
> Huh? Doesn't PSM3 already use it's own special non-verbs char devices that
> already have memory caches and other stuff? Now you want to throw that
> all away and do yet another char dev just for HFI? Why?
[Wan, Kaike] I think that you are referring to PSM2, which uses the OPA hfi1 driver that is specific to the OPA hardware.
PSM3 uses standard verbs drivers and supports standard RoCE.  A focus is the Intel RDMA Ethernet NICs. As such it cannot use the hfi1 driver through the special PSM2 interface. Rather it works with the hfi1 driver through standard verbs interface. The rv module was a new design to bring these concepts to standard transports and hardware.

> 
> I also don't know why you picked the name rv, this looks like it has little to do
> with the usual MPI rendezvous protocol. This is all about bulk transfers. It is
> actually a lot like RDS. Maybe you should be using RDS?
[Wan, Kaike] While there are similarities in concepts, details are different.  Quite frankly this could be viewed as an application accelerator much like RDS served that purpose for Oracle, which continues to be its main use case. The rv module is currently targeting to enables the MPI/OFI/PSM3 application.

The name "rv" is chosen simply because this module is designed to enable the rendezvous protocol of the MPI/OFI/PSM3 application stack for large messages. Short messages are handled by eager transfer through UDP in PSM3.

FYI, there is an OFA presentation from Thurs reviewing PSM3 and RV and covering much of the architecture and rationale.
> 
> Jason