RE: [PATCH v3 for-next 01/33] IB/core: Add RoCE GID cache

["Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>]

> > RoCE v2 is really Infiniband over UDP over IP.  Why don't we just call
> it IBoUDP like it is?
> RoCEv2 is the name in the IBTA spec (Annex 17)

We call RoCE IBoE in the kernel, because that's what it is.  RoCE is an IBTA marketing name.

Looking through the Annex, I don't see where Ethernet is even a requirement for this technology to work.  The IB transport is layered over a standard UDP header.  I do see where the spec calls out updating the IP header, but that's it.

Regardless of what it's called, it replaces the underlying network and transport protocols, versus IB-classic or IBoE/RoCE.  That should be captured properly, not by saying there's a new GID type.  RoCE v2 doesn't even use GIDs as part of its protocol.  It uses UDP/IP addresses.


> > IBoUDP changes the Ethertype, replaces the network header, adds a new
> transport protocol header, and layers IB over that.  This change should be
> exposed properly and not as just a new GID type.
> I don't understand what do you suggest here. Can you give an example?

I don't have a solution here.  Please look at Michael Wang's patch series and see how this would fit into that model.  The introduction of iWarp required defining a new 'transport' type.  IBoE added a new link layer.  Based on those changes, this would warrant introducing a new network layer, so that it can be distinguished properly from the other options.  Maybe that's the right approach?

Cisco's NIC reports a transport layer of 'USNIC_UDP', which should really just be 'UDP'.  That NIC supports UDP/IP/Ethernet, based on the rdma stack's model.  RoCE v2 is also UDP/IP/Ethernet, it only layers IB over that.  (This makes the use of the term 'transport' confusing.  Maybe there should also be a 'session' protocol?)  It seems completely reasonable that a device which does IB/UDP/IP/Ethernet to someday expose the UDP/IP/Ethernet portion (if it doesn't already), and from the same port at the same time.

Rather than continuing to try to make everything look like an IB-classic device because it's convenient, the stack needs to start exposing things properly.  I don't know what the right solution should be, but trying to capture this level of detail as a different GID type definitely looks like a step in the wrong direction.

- Sean