Re: RDMA subsystem namespace related questions (was Re: Finding the namespace of a struct ib_device)

[Jason Gunthorpe <jgg@ziepe.ca>]

On Wed, Sep 30, 2020 at 12:57:48AM +0800, Ka-Cheong Poon wrote:
> On 9/7/20 9:48 PM, Ka-Cheong Poon wrote:
> 
> > This may require a number of changes and the way a client interacts with
> > the current RDMA framework.  For example, currently a client registers
> > once using one struct ib_client and gets device notifications for all
> > namespaces and devices.  Suppose there is rdma_[un]register_net_client(),
> > it may need to require a client to use a different struct ib_client to
> > register for each net namespace.  And struct ib_client probably needs to
> > have a field to store the net namespace.  Probably all those client
> > interaction functions will need to be modified.  Since the clients xarray
> > is global, more clients may mean performance implication, such as it takes
> > longer to go through the whole clients xarray.
> > 
> > There are probably many other subtle changes required.  It may turn out to
> > be not so straight forward.  Is this community willing the take such changes?
> > I can take a stab at it if the community really thinks that this is preferred.
> 
> 
> Attached is a diff of a prototype for the above.  This exercise is
> to see what needs to be done to have a more network namespace aware
> interface for RDMA client registration.

An RDMA device is either in all namespaces or in a single
namespace. If a client has some interest in only some namespaces then
it should check the namespace during client registration and not
register if it isn't interested. No need to change anything in the
core code.

> Is the RDMA shared namespace mode the preferred mode to use as it is the
> default mode?  

Shared is the legacy mode, modern systems should switch to namespace
mode at early boot

> Is it expected that a client knows the running mode before
> interacting with the RDMA subsystem?  

Why would a client care?

> Is a client not supposed to differentiate different namespaces?

None do today.

> A new connection comes in and the event handler is called for an
> RDMA_CM_EVENT_CONNECT_REQUEST event.  There is no obvious namespace info regarding
> the event.  It seems that the only way to find out the namespace info is to
> use the context of struct rdma_cm_id.  

The rdma_cm_id has only a single namespace, the ULP knows what it is
because it created it. A listening ID can't spawn new IDs in different
namespaces.

> (*) Note that in __rdma_create_id(), it does a get_net(net) to put a
>     reference on a namespace.  Suppose a kernel module calls rdma_create_id()
>     in its namespace .init function to create an RDMA listener and calls
>     rdma_destroy_id() in its namespace .exit function to destroy it.  

Yes, namespaces remain until all objects touching them are deleted.

It seems like a ULP error to drive cm_id lifetime entirely from the
per-net stuff.

This would be similar to creating a socket in the kernel.

>     __rdma_create_id() adds a reference to a namespace, when a sys admin
>     deletes a namespace (say `ip netns del ...`), the namespace won't be
>     deleted because of this reference.  And the module will not release this
>     reference until its .exit function is called only when the namespace is
>     deleted.  To resolve this issue, in the diff (in __rdma_create_id()), I
>     did something similar to the kern check in sk_alloc().

What you are running into is there is no kernel user of net
namespaces, all current ULPs exclusively use the init_net.

Without an example of what that is supposed to be like it is hard to
really have a discussion. You should reference other TCP in the kernel
to see if someone has figured out how to make this work for TCP. It
should be basically the same.

Jason