Re: [PATCH RFC 0/9] A rendezvous module

[Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>]

On 3/19/2021 3:44 PM, Jason Gunthorpe wrote:
> On Fri, Mar 19, 2021 at 03:22:45PM -0400, Dennis Dalessandro wrote:
> 
>>>> [Wan, Kaike] I think that you are referring to PSM2, which uses the
>>>> OPA hfi1 driver that is specific to the OPA hardware.  PSM3 uses
>>>> standard verbs drivers and supports standard RoCE.
>>>
>>> Uhhh.. "PSM" has always been about the ipath special char device, and
>>> if I recall properly the library was semi-discontinued and merged into
>>> libfabric.
>>
>> This driver is intended to work with a fork of the PSM2 library. The PSM2
>> library which is for Omni-Path is now maintained by Cornelis Networks on our
>> GitHub. PSM3 is something from Intel for Ethernet. I know it's a bit
>> confusing.
> 
> "a bit" huh?

Just a bit. :)

> 
>>> So here you are talking about a libfabric verbs provider that doesn't
>>> use the ipath style char interface but uses verbs and this rv thing so
>>> we call it a libfabric PSM3 provider because thats not confusing to
>>> anyone at all..
>>>
>>>> A focus is the Intel RDMA Ethernet NICs. As such it cannot use the
>>>> hfi1 driver through the special PSM2 interface.
>>>
>>> These are the drivers that aren't merged yet, I see. So why are you
>>> sending this now? I'm not interested to look at even more Intel code
>>> when their driver saga is still ongoing for years.
>>>
>>>> Rather it works with the hfi1 driver through standard verbs
>>>> interface.
>>>
>>> But nobody would do that right? You'd get better results using the
>>> hif1 native interfaces instead of their slow fake verbs stuff.
>>
>> I can't imagine why. I'm not sure what you mean by our slow fake verbs
>> stuff? We support verbs just fine. It's certainly not fake.
> 
> hfi1 calls to the kernel for data path operations - that is "fake" in
> my book. Verbs was always about avoiding that kernel transition, to
> put it back in betrays the spirit. So a kernel call for rv, or the hfi
> cdev, or the verbs post-send is really all a wash.

Probably better to argue that in another thread I guess.

> I didn't understand your answer, do you see using this with hfi1 or
> not?

I don't see how this could ever use hfi1. So no.

> It looks a lot copy&pasted from the hfi1 driver, so now we are on our
> third copy of this code :(

I haven't had a chance to look beyond the cover letter in depth at how 
things have changed. I really hope it's not that bad.

> And why did it suddenly become a ULP that somehow shares uverbs
> resources?? I'm pretty skeptical that can be made to work correctly..

I see your point.

I was just providing some background/clarification. Bottom line this has 
nothing to do with hfi1 or OPA or PSM2.

-Denny