Re: [PATCH RFC 0/9] A rendezvous module

[Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>]

On 3/19/2021 6:57 PM, Rimmer, Todd wrote:
>>> [Wan, Kaike] Incorrect. The rv module works with hfi1.
>>
>> Interesting. I was thinking the opposite. So what's the benefit? When would
>> someone want to do that?
> The more interesting scenario is for customers who would like to run libfabric and other Open Fabrics Alliance software over various verbs capable hardware.

Ah ok that makes sense. Not that running it over hfi1 is the goal but 
being able to run over verbs devices. Makes sense to me now.

> Today PSM2 is a good choice for OPA hardware.  However for some other devices without existing libfabric providers, rxm and rxd are the best choices.
> As was presented in Open Fabrics workshop today by James Erwin, PSM3 offers noticeable benefits over existing libfabric rxm and rxd providers
> and the rv module offers noticeable performance benefits when using PSM3.

For those that haven't seen it the talks will be posted to YouTube 
and/or OpenFabrics.org web page. There are actually two talks on this 
stuff. The first of which is by Todd is available now [1], James' talk 
will be up soon I'm sure.

>> I haven't had a chance to look beyond the cover letter in depth at how things
>> have changed. I really hope it's not that bad.
> While a few stylistic elements got carried forward, as you noticed.  This is much different from hfi1 as it doesn't directly access hardware and is hence smaller.
> We carefully looked at overlap with features in ib_core and the patch set contains a couple minor API additions to ib_core to simplify some operations
> which others may find useful.

Right, so if there is common functionality between hfi1 and rv then it 
might belong in the core. Especially considering if it's something 
that's common between a ULP and a HW driver.

>> I also don't know why you picked the name rv, this looks like it has little to do with the usual MPI rendezvous protocol.
> The focus of the design was to support the bulk transfer part of the MPI rendezvous protocol, hence the name rv.
> We'd welcome other name suggestions, wanted to keep the name simple and brief.

Like I said previously you can place the blame for the name on me. Kaike 
and Todd just carried it forward. I think Sean had an idea in one of the 
other replies. Let's hear some other suggestions too.

>> No pre-adding reserved stuff
>> Lots of alignment holes, don't do that either.
> We'd like advise on a challenging situation.  Some customers desire NICs to support nVidia GPUs in some environments.
> Unfortunately the nVidia GPU drivers are not upstream, and have not been for years.  So we are forced to have both out of tree
> and upstream versions of the code.  We need the same applications to be able to work over both, so we would like the
> GPU enabled versions of the code to have the same ABI as the upstream code as this greatly simplifies things.
> We have removed all GPU specific code from the upstream submission, but used both the "alignment holes" and the "reserved"
> mechanisms to hold places for GPU specific fields which can't be upstreamed.

This problem extends to other drivers as well. I'm also interested in 
advice on the situation. I don't particularly like this either, but we 
need a way to accomplish the goal. We owe it to users to be flexible. 
Please offer suggestions.

[1] https://www.youtube.com/watch?v=iOvt_Iqz0uU

-Denny