Re: RFC: ipath ioctls and their replacements

[Roland Dreier <rdreier@cisco.com>]

    Eric> Roland you know the RDMA model best, are things so tied to
    Eric> the current crop of infiniband protocols that what the ipath
    Eric> code wants to do is not covered?

    Eric> They clearly need subsystem support and what they are trying
    Eric> to do either isn't covered or they don't see how to use what
    Eric> is there.  Do the infiniband verbs not allow dealing with a
    Eric> unreliable datagram protocol?

I think this has been answered already but the issue is really that
the PathScale hardware does not implement RDMA or even any of the
other connection-oriented abstractions that the RDMA layer is designed
for.  The hardware has only much lower level capabilities, which
basically can send and receive packets on an IB link.

With those capabilites it is possible to implement IB transports in
software -- so for example RDMA read operations are simulated by
having the CPU on the receiver copy data to send the response.
However that implementation is not going to make good use of the IB
midlayer, which really operates at the abstraction level above the IB
transport.

It's also possible to use the PathScale hardware to directly implement
MPI on top of a protocol optimized specifically for MPI, without using
IB verbs semantics or an IB transport on the wire.  But clearly the
userspace interface needed for doing this is not going to match up
very well with a userspace interface for IB verbs (which is at a
different abstraction level).

 - R.