Re: Finding the namespace of a struct ib_device

[Ka-Cheong Poon <ka-cheong.poon@oracle.com>]

On 9/7/20 6:22 PM, Leon Romanovsky wrote:
> On Mon, Sep 07, 2020 at 05:28:23PM +0800, Ka-Cheong Poon wrote:
>> On 9/7/20 5:04 PM, Leon Romanovsky wrote:
>>> On Mon, Sep 07, 2020 at 04:24:26PM +0800, Ka-Cheong Poon wrote:
>>>> On 9/7/20 3:18 PM, Leon Romanovsky wrote:
>>>>> On Mon, Sep 07, 2020 at 11:33:38AM +0800, Ka-Cheong Poon wrote:
>>>>>> On 9/6/20 3:44 PM, Leon Romanovsky wrote:
>>>>>>> On Fri, Sep 04, 2020 at 10:02:10PM +0800, Ka-Cheong Poon wrote:
>>>>>>>> On 9/4/20 7:32 PM, Jason Gunthorpe wrote:
>>>>>>>>> On Fri, Sep 04, 2020 at 12:01:12PM +0800, Ka-Cheong Poon wrote:
>>>>>>>>>> On 9/4/20 1:39 AM, Jason Gunthorpe wrote:
>>>>>>>>>>> On Thu, Sep 03, 2020 at 10:02:01PM +0800, Ka-Cheong Poon wrote:
>>>>>>>>>>>> When a struct ib_client's add() function is called. is there a
>>>>>>>>>>>> supported method to find out the namespace of the passed in
>>>>>>>>>>>> struct ib_device?  There is rdma_dev_access_netns() but it does
>>>>>>>>>>>> not return the namespace.  It seems that it needs to have
>>>>>>>>>>>> something like the following.
>>>>>>>>>>>>
>>>>>>>>>>>> struct net *rdma_dev_to_netns(struct ib_device *ib_dev)
>>>>>>>>>>>> {
>>>>>>>>>>>>              return read_pnet(&ib_dev->coredev.rdma_net);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Comments?
>>>>>>>>>>>
>>>>>>>>>>> I suppose, but why would something need this?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If the client needs to allocate stuff for the namespace
>>>>>>>>>> related to that device, it needs to know the namespace of
>>>>>>>>>> that device.  Then when that namespace is deleted, the
>>>>>>>>>> client can clean up those related stuff as the client's
>>>>>>>>>> namespace exit function can be called before the remove()
>>>>>>>>>> function is triggered in rdma_dev_exit_net().  Without
>>>>>>>>>> knowing the namespace of that device, coordination cannot
>>>>>>>>>> be done.
>>>>>>>>>
>>>>>>>>> Since each device can only be in one namespace, why would a client
>>>>>>>>> ever need to allocate at a level more granular than a device?
>>>>>>>>
>>>>>>>>
>>>>>>>> A client wants to have namespace specific info.  If the
>>>>>>>> device belongs to a namespace, it wants to associate those
>>>>>>>> info with that device.  When a namespace is deleted, the
>>>>>>>> info will need to be deleted.  You can consider the info
>>>>>>>> as associated with both a namespace and a device.
>>>>>>>
>>>>>>> Can you be more specific about which info you are talking about?
>>>>>>
>>>>>>
>>>>>> Actually, a lot of info can be both namespace and device specific.
>>>>>> For example, a client wants to have a different PD allocation policy
>>>>>> with a device when used in different namespaces.
>>>>>>
>>>>>>
>>>>>>> And what is the client that is net namespace-aware from one side,
>>>>>>> but from another separate data between them "manually"?
>>>>>>
>>>>>>
>>>>>> Could you please elaborate what is meant by "namespace aware from
>>>>>> one side but from another separate data between them manually"?
>>>>>> I understand what namespace aware means.  But it is not clear what
>>>>>> is meant by "separating data manually".  Do you mean having different
>>>>>> behavior in different namespaces?  If this is the case, there is
>>>>>> nothing special here.  An admin may choose to have different behavior
>>>>>> in different namespaces.  There is nothing manual going on in the
>>>>>> client code.
>>>>>
>>>>> We are talking about net-namespaces, and as we wrote above, the ib_device
>>>>> that supports such namespace can exist only in a single one
>>>>>
>>>>> The client that implemented such support can check its namespace while
>>>>> "client->add" is called. It should be equal to be seen by ib_device.
>>>>>
>>>>> See:
>>>>>     rdma_dev_change_netns ->
>>>>>     	enable_device_and_get ->
>>>>> 		add_client_context ->
>>>>> 			client->add(device)
>>>>
>>>>
>>>> This is the original question.  How does the client's add() function
>>>> know the namespace of device?  What is your suggestion in finding
>>>> the net namespace of device at add() time?
>>>
>>> As I wrote above, "It should be equal to be seen by ib_device.", check net
>>> namespace of your client.
>>
>>
>> Could you please be more specific?  A client calls ib_register_client() to
>> register with the RDMA framework.  Then when a device is added, the client's
>> add() function is called with the struct ib_device.  How does the client
>> find out the namespace "seen by the ib_device"?  Do you mean that there is
>> a variant of ib_register_client() which can take a net namespace as parameter?
>> Or is there a variant of struct ib_client which has a net namespace field?
>> Or?  Thanks.
> 
> "Do you mean that there is a variant of ib_register_client()
> which can take a net namespace as parameter?"
> 
> No, it doesn't exist but it is easy to extend and IMHO the right
> thing to do.


This may require a number of changes and the way a client interacts with
the current RDMA framework.  For example, currently a client registers
once using one struct ib_client and gets device notifications for all
namespaces and devices.  Suppose there is rdma_[un]register_net_client(),
it may need to require a client to use a different struct ib_client to
register for each net namespace.  And struct ib_client probably needs to
have a field to store the net namespace.  Probably all those client
interaction functions will need to be modified.  Since the clients xarray
is global, more clients may mean performance implication, such as it takes
longer to go through the whole clients xarray.

There are probably many other subtle changes required.  It may turn out to
be not so straight forward.  Is this community willing the take such changes?
I can take a stab at it if the community really thinks that this is preferred.

Thanks.


-- 
K. Poon
ka-cheong.poon@oracle.com