RE: [RFC] XRC upstream merge reboot

["Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>]

> Great that you are taking this on!  I will review this next week.

Hopefully I'll have some early patches sometime next week.  See below for my current thoughts based on how the implementation is progressing.  My thoughts change hourly.

> > From an architecture viewpoint, XRC adds 4 new XRC specific objects:
> domains, INI QPs, TGT QPs, and SRQs.  For the purposes of the libibverbs
> API only, I'm suggesting the following mappings:
> >
> > XRC domains - Hidden under a PD, dynamically allocated when needed.  An
> extended ops call allows the xrcd to be shared between processes.  This
> minimizes changes to existing structures and APIs which only take a struct
> ibv_pd.
> >
> > INI QPs - Exposed through a new IBV_QPT_XRC_SQ qp type.  This is a send-
> only QP with minimal differences from an RC QP from a user's perspective.
> >
> > TGT QPs - Not exposed to user space.  XRC TGT QP creation and setup is
> handled by the kernel.
> >
> > XRC SRQs - Exposed through a new IBV_QPT_XRC_RQ qp type.  This is an SRQ
> that is tracked using a struct ibv_qp.  This minimizes API changes to both
> libibverbs and librdmacm.

The librdmacm wants IBV_QPT_XRC_RQ to be defined, which it uses for SRQN resolution.  XRC_SQ results in using the IB CM connection protocol.  XRC_RQ results in using CM SIDR.

The actual XRC SRQ is allocated through a new call like the existing patches.  XRC_RQ is not used by ibv_create_qp().

> > Need new uverbs commands to support alloc/dealloc xrcd and create xrc
> srq.  Create QP must handle XRC INI QPs.  XRC TGT QPs are not exposed;
> ***all XRC INI->TGT QP setup is done in band***.
> >
> > Somewhere, an xrc sub-module listens on a SID and accepts incoming XRC
> connection requests.  This requires associating the xrcd and SID, the
> details of which I'm not clear on.  The xrcd is most readily known to
> uverbs, but a SID is usually formed by the rdma_cm.  Even how the latter
> is done is unclear.

I'm not sure if xrcd needs to be explicitly exposed to user space.  I think user space will end up needing to be involved in some degree when INI->TGT QPs are established from the perspective of the rdma_cm.  For example, initiating the listen and telling the kernel whether to accept or reject a connection.  I still don't know that XRC TGT QPs need to be exposed through libibverbs similar to how they are being done.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html