Re: [PATCH RFC 0/9] A rendezvous module

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Mar 19, 2021 at 08:46:56PM +0000, Rimmer, Todd wrote:
> > I'm suprirsed to hear someone advocate that is a good thing when we were
> > all told that the hfi1 cdev *must* exist because the kernel transition through
> > verbs was far to expensive.
>
> It depends on the goal vendors have with verbs vs other APIs such as
> libfabric.  hfi1's verbs goal was focused on storage bandwidth and
> the cdev was focused on HPC latency and bandwidth for MPI via PSM2
> and libfabric.  I'm unclear why we are debating hfi1 here, seems it
> should be in another thread.

Because this is copied from hfi1?

> > What is a UD-X?
> UD-X is a vendor specific set of HW interfaces and wire protocols
> implemented in UCX for nVidia Connect-X series of network devices.
> Many of it's concepts are very similar to those which ipath and hfi1
> HW and software implemented.

Oh, there is lots of stuff in UCX, I'm not surprised you similarities
to what PSM did since psm/libfabric/ucx are all solving the same
problems.

> > rv seems to completely destroy alot of the HPC performance offloads that
> > vendors are layering on RC QPs

> Different vendors have different approaches to performance and chose
> different design trade-offs.

That isn't my point, by limiting the usability you also restrict the
drivers where this would meaningfully be useful.

So far we now know that it is not useful for mlx5 or hfi1, that leaves
only hns unknown and still in the HPC arena.

Jason