Re: [PATCH] rdma/core: add __module_get()/module_put() to cma_[de]ref_dev()

[Stefan Metzmacher <metze@samba.org>]

[-- Attachment #1.1: Type: text/plain, Size: 2713 bytes --]

Am 30.09.19 um 14:41 schrieb Jason Gunthorpe:
> On Mon, Sep 30, 2019 at 11:04:55AM +0200, Stefan Metzmacher wrote:
>> Currently there seems to be a problem when an RDMA listener or connection
>> is active on an ib_device.
>>
>> 'rmmod rdma_rxe' (the same for 'siw' and most likely all
>> others) just hangs like this until shutdown the listeners and
>> connections:
>>
>>   [<0>] remove_client_context+0x97/0xe0 [ib_core]
>>   [<0>] disable_device+0x90/0x120 [ib_core]
>>   [<0>] __ib_unregister_device+0x41/0xa0 [ib_core]
>>   [<0>] ib_unregister_driver+0xbb/0x100 [ib_core]
>>   [<0>] rxe_module_exit+0x1a/0x8aa [rdma_rxe]
>>   [<0>] __x64_sys_delete_module+0x147/0x290
>>   [<0>] do_syscall_64+0x5a/0x130
>>   [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>
>> The following would be expected:
>>
>>   rmmod: ERROR: Module rdma_rxe is in use
>>
>> And this change provides that.
>>
>> Once all add listeners and connections are gone
>> the module can be removed again.
>>
>> Signed-off-by: Stefan Metzmacher <metze@samba.org>
>> Cc: stable@vger.kernel.org
>> ---
>>  drivers/infiniband/core/cma.c | 2 ++
>>  1 file changed, 2 insertions(+)
> 
> How do you even get here? Are you using in-kernel modules to access
> rxe?

Yes, I'm testing my smbdirect driver with Samba.
https://git.samba.org/?p=metze/linux/smbdirect.git;a=summary

Samba's 'smbd' listens for incoming connections.

> Drivers are supposed to declare a DEVICE_FATAL error when their module
> is removed and then progress toward cleaning up. It would seem this is
> missing in rxe.

It's a bit hard for me to follow the code path from other drivers
and how an unload of them would trigger IB_EVENT_DEVICE_FATAL.

Can you give me an explicit example of what the backtrace would
look like for the code that triggers IB_EVENT_DEVICE_FATAL?

> Globally blocking module unload would break the existing dis-associate
> flows, and blocking until listeners are removed seems like all rdma
> drivers will instantly become permanetly blocked when things like SRP
> or IPoIB CM mode are running?

So the design is to allow drivers to be unloaded while there are
active connections?

If so is this specific to RDMA drivers?

For filesystems it typically not allowed that the driver is unloaded
while it's mounted.

> I think the proper thing is to fix rxe (and probably siw) to signal
> the DEVICE_FATAL so the CMA listeners can cleanly disconnect

I just found that drivers/nvme/host/rdma.c and
drivers/nvme/target/rdma.c both use ib_register_client();
in order to get notified that a device is going to be removed.

Maybe I should also use ib_register_client()?

Thanks!
metze


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]