Re: [PATCH] rdma/core: add __module_get()/module_put() to cma_[de]ref_dev()

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Sep 30, 2019 at 11:04:55AM +0200, Stefan Metzmacher wrote:
> Currently there seems to be a problem when an RDMA listener or connection
> is active on an ib_device.
> 
> 'rmmod rdma_rxe' (the same for 'siw' and most likely all
> others) just hangs like this until shutdown the listeners and
> connections:
> 
>   [<0>] remove_client_context+0x97/0xe0 [ib_core]
>   [<0>] disable_device+0x90/0x120 [ib_core]
>   [<0>] __ib_unregister_device+0x41/0xa0 [ib_core]
>   [<0>] ib_unregister_driver+0xbb/0x100 [ib_core]
>   [<0>] rxe_module_exit+0x1a/0x8aa [rdma_rxe]
>   [<0>] __x64_sys_delete_module+0x147/0x290
>   [<0>] do_syscall_64+0x5a/0x130
>   [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> The following would be expected:
> 
>   rmmod: ERROR: Module rdma_rxe is in use
> 
> And this change provides that.
> 
> Once all add listeners and connections are gone
> the module can be removed again.
> 
> Signed-off-by: Stefan Metzmacher <metze@samba.org>
> Cc: stable@vger.kernel.org
> ---
>  drivers/infiniband/core/cma.c | 2 ++
>  1 file changed, 2 insertions(+)

How do you even get here? Are you using in-kernel modules to access
rxe?

Drivers are supposed to declare a DEVICE_FATAL error when their module
is removed and then progress toward cleaning up. It would seem this is
missing in rxe.

Globally blocking module unload would break the existing dis-associate
flows, and blocking until listeners are removed seems like all rdma
drivers will instantly become permanetly blocked when things like SRP
or IPoIB CM mode are running?

I think the proper thing is to fix rxe (and probably siw) to signal
the DEVICE_FATAL so the CMA listeners can cleanly disconnect

Jason