librdmacm: cleanup/reinit API?

librdmacm: cleanup/reinit API?

[Manikarnike, Vasuki]

Hello,

I notice that librdmacm does not have a reinit/deinit/cleanup API.
I'm looking for ucma_cleanup(), sort of the opposite of ucma_init().

The first call to rdma_get_devices() from the application calls ucma_init() if required.
ucma_init() builds the  list of devices, and subsequent library calls use this list.

We'd like to issue a full chip reset on a Mellanox CX-5 adapter by using the 'mlxfwreset' tool,
and we'd like to do this without requiring our application to be restarted.

Our application uses SPDK, and I notice that trying to probe for new RDMA connections fails after the reset.
The probe fails because RDMA address resolution fails.

The reset destroys the Ethernet interfaces and adds them back to the system.
The list built by ucma_init() is never updated, and library calls continue to use a stale list.

When the system is in this state, I was able to run "rping" and "ibv_rc_pingpong" to the client.
If the application is restarted, the RDMA address is successfully resolved.

This points to the fact that the list of devices maintained by librdmacm is stale and needs to be updated somehow.

I added a new API ucma_cleanup() which walks through the list of devices and frees them.
I called this API from our application after issuing the chip reset.
After this all the RDMA address resolution calls succeeded.

I see that ucma_cleanup() existed as a destructor function at some point, but was removed.
Could you please throw some light on why this was removed?

Thanks!
Vasuki

System details:
OS: Debian 10 (buster) based
Kernel: based on RHEL 3.10.0-1062.18.1
Hardware: Mellanox CX-5 Adapter
Library version: librdmacm 24.0-2~bpo10+1
SPDK 20.07/DPDK 20.05

Re: librdmacm: cleanup/reinit API?

[Leon Romanovsky]

On Mon, Oct 12, 2020 at 10:38:50PM +0000, Manikarnike, Vasuki wrote:
> Hello,
>
> I notice that librdmacm does not have a reinit/deinit/cleanup API.
> I'm looking for ucma_cleanup(), sort of the opposite of ucma_init().
>
> The first call to rdma_get_devices() from the application calls ucma_init() if required.
> ucma_init() builds the  list of devices, and subsequent library calls use this list.
>
> We'd like to issue a full chip reset on a Mellanox CX-5 adapter by using the 'mlxfwreset' tool,
> and we'd like to do this without requiring our application to be restarted.

It is supported with this already merged PR.
https://github.com/linux-rdma/rdma-core/pull/750

Thanks

Re: librdmacm: cleanup/reinit API?

[Manikarnike, Vasuki]

Thanks.

On 11/15/20, 3:24 AM, "Leon Romanovsky" <leon@kernel.org> wrote:

    On Mon, Oct 12, 2020 at 10:38:50PM +0000, Manikarnike, Vasuki wrote:
    > Hello,
    >
    > I notice that librdmacm does not have a reinit/deinit/cleanup API.
    > I'm looking for ucma_cleanup(), sort of the opposite of ucma_init().
    >
    > The first call to rdma_get_devices() from the application calls ucma_init() if required.
    > ucma_init() builds the  list of devices, and subsequent library calls use this list.
    >
    > We'd like to issue a full chip reset on a Mellanox CX-5 adapter by using the 'mlxfwreset' tool,
    > and we'd like to do this without requiring our application to be restarted.

    It is supported with this already merged PR.
    https://github.com/linux-rdma/rdma-core/pull/750

    Thanks