Re: [PATCH RFC 0/9] A rendezvous module

[Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>]

On 3/19/2021 11:48 AM, Jason Gunthorpe wrote:
> On Fri, Mar 19, 2021 at 02:49:29PM +0000, Wan, Kaike wrote:
>>> From: Jason Gunthorpe <jgg@nvidia.com>
>>> Sent: Friday, March 19, 2021 9:53 AM
>>> To: Wan, Kaike <kaike.wan@intel.com>
>>> Cc: dledford@redhat.com; linux-rdma@vger.kernel.org; Rimmer, Todd
>>> <todd.rimmer@intel.com>
>>> Subject: Re: [PATCH RFC 0/9] A rendezvous module
>>>
>>> On Fri, Mar 19, 2021 at 08:56:26AM -0400, kaike.wan@intel.com wrote:
>>>
>>>> - Basic mode of operations (PSM3 is used as an example for user
>>>>    applications):
>>>>    - A middleware (like MPI) has out-of-band communication channels
>>>>      between any two nodes, which are used to establish high performance
>>>>      communications for providers such as PSM3.
>>>
>>> Huh? Doesn't PSM3 already use it's own special non-verbs char devices that
>>> already have memory caches and other stuff? Now you want to throw that
>>> all away and do yet another char dev just for HFI? Why?
> 
>> [Wan, Kaike] I think that you are referring to PSM2, which uses the
>> OPA hfi1 driver that is specific to the OPA hardware.  PSM3 uses
>> standard verbs drivers and supports standard RoCE.
> 
> Uhhh.. "PSM" has always been about the ipath special char device, and
> if I recall properly the library was semi-discontinued and merged into
> libfabric.

This driver is intended to work with a fork of the PSM2 library. The 
PSM2 library which is for Omni-Path is now maintained by Cornelis 
Networks on our GitHub. PSM3 is something from Intel for Ethernet. I 
know it's a bit confusing.

> So here you are talking about a libfabric verbs provider that doesn't
> use the ipath style char interface but uses verbs and this rv thing so
> we call it a libfabric PSM3 provider because thats not confusing to
> anyone at all..
> 
>> A focus is the Intel RDMA Ethernet NICs. As such it cannot use the
>> hfi1 driver through the special PSM2 interface.
> 
> These are the drivers that aren't merged yet, I see. So why are you
> sending this now? I'm not interested to look at even more Intel code
> when their driver saga is still ongoing for years.
> 
>> Rather it works with the hfi1 driver through standard verbs
>> interface.
> 
> But nobody would do that right? You'd get better results using the
> hif1 native interfaces instead of their slow fake verbs stuff.

I can't imagine why. I'm not sure what you mean by our slow fake verbs 
stuff? We support verbs just fine. It's certainly not fake.

>>> I also don't know why you picked the name rv, this looks like it has little to do
>>> with the usual MPI rendezvous protocol. This is all about bulk transfers. It is
>>> actually a lot like RDS. Maybe you should be using RDS?
.
.
>> The name "rv" is chosen simply because this module is designed to
>> enable the rendezvous protocol of the MPI/OFI/PSM3 application stack
>> for large messages. Short messages are handled by eager transfer
>> through UDP in PSM3.
> 
> A bad name seems like it will further limit potential re-use of this
> code.
  As to the name, I started this driver when I was still at Intel before 
handing the reins to Kaike when we spun out. I plucked the name rv out 
of the air without much consideration. While the code seems to have 
changed a lot, the name seems to have stuck. It was as Kaike mentions 
because it helps enable the rendezvous side of PSM3.

-Denny