[PATCH] Delay the initialization of rnbd_server module to late_initcall level

[PATCH] Delay the initialization of rnbd_server module to late_initcall level

[haris.iqbal]

From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>

Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>

The rnbd_server module's communication manager initialization depends on the
registration of the "network namespace subsystem" of the RDMA CM agent module.
As such, when the kernel is configured to load the rnbd_server and the RDMA
cma module during initialization; and if the rnbd_server module is initialized
before RDMA cma module, a null ptr dereference occurs during the RDMA bind
operation.
This patch delays the initialization of the rnbd_server module to the
late_initcall level, since RDMA cma module uses module_init which puts it into
the device_initcall level.
---
 drivers/block/rnbd/rnbd-srv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
index 86e61523907b..213df05e5994 100644
--- a/drivers/block/rnbd/rnbd-srv.c
+++ b/drivers/block/rnbd/rnbd-srv.c
@@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
 	rnbd_srv_destroy_sysfs_files();
 }
 
-module_init(rnbd_srv_init_module);
+late_initcall(rnbd_srv_init_module);
 module_exit(rnbd_srv_cleanup_module);
-- 
2.25.1

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jinpu Wang]

Hi Haris,

Thanks for patch, a few comments inline
On Wed, Jun 17, 2020 at 12:37 PM <haris.iqbal@cloud.ionos.com> wrote:
>
> From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
>
> Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
This part should be the last after commit message.
The subject sounds better maybe: block/rnbd: Delay the initialization
of rnbd_server module to late_initcall level
>
> The rnbd_server module's communication manager initialization depends on the
> registration of the "network namespace subsystem" of the RDMA CM agent module.
> As such, when the kernel is configured to load the rnbd_server and the RDMA
> cma module during initialization; and if the rnbd_server module is initialized
> before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> operation.
would be better to include the call trace here.
> This patch delays the initialization of the rnbd_server module to the
> late_initcall level, since RDMA cma module uses module_init which puts it into
> the device_initcall level.

With the comments addressed:
Acked-by: Jack Wang <jinpu.wang@cloud.ionos.com>
> ---
>  drivers/block/rnbd/rnbd-srv.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> index 86e61523907b..213df05e5994 100644
> --- a/drivers/block/rnbd/rnbd-srv.c
> +++ b/drivers/block/rnbd/rnbd-srv.c
> @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
>         rnbd_srv_destroy_sysfs_files();
>  }
>
> -module_init(rnbd_srv_init_module);
> +late_initcall(rnbd_srv_init_module);
>  module_exit(rnbd_srv_cleanup_module);
> --
> 2.25.1
>

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Leon Romanovsky]

On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
>
> Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
>
> The rnbd_server module's communication manager initialization depends on the
> registration of the "network namespace subsystem" of the RDMA CM agent module.
> As such, when the kernel is configured to load the rnbd_server and the RDMA
> cma module during initialization; and if the rnbd_server module is initialized
> before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> operation.
> This patch delays the initialization of the rnbd_server module to the
> late_initcall level, since RDMA cma module uses module_init which puts it into
> the device_initcall level.
> ---
>  drivers/block/rnbd/rnbd-srv.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> index 86e61523907b..213df05e5994 100644
> --- a/drivers/block/rnbd/rnbd-srv.c
> +++ b/drivers/block/rnbd/rnbd-srv.c
> @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
>  	rnbd_srv_destroy_sysfs_files();
>  }
>
> -module_init(rnbd_srv_init_module);
> +late_initcall(rnbd_srv_init_module);

I don't think that this is correct change. Somehow nvme-rdma works:
module_init(nvme_rdma_init_module);
-> nvme_rdma_init_module
 -> nvmf_register_transport(&nvme_rdma_transport);
  -> nvme_rdma_create_ctrl
   -> nvme_rdma_setup_ctrl
    -> nvme_rdma_configure_admin_queue
     -> nvme_rdma_alloc_queue
      -> rdma_create_id

>  module_exit(rnbd_srv_cleanup_module);
> --
> 2.25.1
>

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jason Gunthorpe]

On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> >
> > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> >
> > The rnbd_server module's communication manager initialization depends on the
> > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > cma module during initialization; and if the rnbd_server module is initialized
> > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > operation.
> > This patch delays the initialization of the rnbd_server module to the
> > late_initcall level, since RDMA cma module uses module_init which puts it into
> > the device_initcall level.
> >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > index 86e61523907b..213df05e5994 100644
> > +++ b/drivers/block/rnbd/rnbd-srv.c
> > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> >  	rnbd_srv_destroy_sysfs_files();
> >  }
> >
> > -module_init(rnbd_srv_init_module);
> > +late_initcall(rnbd_srv_init_module);
> 
> I don't think that this is correct change. Somehow nvme-rdma works:
> module_init(nvme_rdma_init_module);
> -> nvme_rdma_init_module
>  -> nvmf_register_transport(&nvme_rdma_transport);
>   -> nvme_rdma_create_ctrl
>    -> nvme_rdma_setup_ctrl
>     -> nvme_rdma_configure_admin_queue
>      -> nvme_rdma_alloc_queue
>       -> rdma_create_id

If it does work, it is by luck.

Keep in mind all this only matters for kernels without modules.

Maybe cma should be upgraded to subsystem_init ? That is a bit tricky
too as it needs the ib_client stuff working

Jason

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Leon Romanovsky]

On Wed, Jun 17, 2020 at 03:20:46PM -0300, Jason Gunthorpe wrote:
> On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> > On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > >
> > > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > >
> > > The rnbd_server module's communication manager initialization depends on the
> > > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > > cma module during initialization; and if the rnbd_server module is initialized
> > > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > > operation.
> > > This patch delays the initialization of the rnbd_server module to the
> > > late_initcall level, since RDMA cma module uses module_init which puts it into
> > > the device_initcall level.
> > >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > > index 86e61523907b..213df05e5994 100644
> > > +++ b/drivers/block/rnbd/rnbd-srv.c
> > > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> > >  	rnbd_srv_destroy_sysfs_files();
> > >  }
> > >
> > > -module_init(rnbd_srv_init_module);
> > > +late_initcall(rnbd_srv_init_module);
> >
> > I don't think that this is correct change. Somehow nvme-rdma works:
> > module_init(nvme_rdma_init_module);
> > -> nvme_rdma_init_module
> >  -> nvmf_register_transport(&nvme_rdma_transport);
> >   -> nvme_rdma_create_ctrl
> >    -> nvme_rdma_setup_ctrl
> >     -> nvme_rdma_configure_admin_queue
> >      -> nvme_rdma_alloc_queue
> >       -> rdma_create_id
>
> If it does work, it is by luck.

I didn't check every ULP, but it seems that all ULPs use the same
module_init.

>
> Keep in mind all this only matters for kernels without modules.

Can it be related to the fact that other ULPs call to ib_register_client()
before calling to rdma-cm? RNBD does not have such call.

>
> Maybe cma should be upgraded to subsystem_init ? That is a bit tricky
> too as it needs the ib_client stuff working.

It can work, but makes me wonder if it is last change in this area,
every time we touch *_initcall(), we break something else.

I'm not proposing this, but just loudly wondering, do we really need rdma-cm
as a separate module? Can we bring it to be part of ib_core?

Thanks

>
> Jason

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jason Gunthorpe]

On Wed, Jun 17, 2020 at 10:07:56PM +0300, Leon Romanovsky wrote:
> On Wed, Jun 17, 2020 at 03:20:46PM -0300, Jason Gunthorpe wrote:
> > On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > > > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > >
> > > > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > >
> > > > The rnbd_server module's communication manager initialization depends on the
> > > > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > > > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > > > cma module during initialization; and if the rnbd_server module is initialized
> > > > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > > > operation.
> > > > This patch delays the initialization of the rnbd_server module to the
> > > > late_initcall level, since RDMA cma module uses module_init which puts it into
> > > > the device_initcall level.
> > > >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > > > index 86e61523907b..213df05e5994 100644
> > > > +++ b/drivers/block/rnbd/rnbd-srv.c
> > > > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> > > >  	rnbd_srv_destroy_sysfs_files();
> > > >  }
> > > >
> > > > -module_init(rnbd_srv_init_module);
> > > > +late_initcall(rnbd_srv_init_module);
> > >
> > > I don't think that this is correct change. Somehow nvme-rdma works:
> > > module_init(nvme_rdma_init_module);
> > > -> nvme_rdma_init_module
> > >  -> nvmf_register_transport(&nvme_rdma_transport);
> > >   -> nvme_rdma_create_ctrl
> > >    -> nvme_rdma_setup_ctrl
> > >     -> nvme_rdma_configure_admin_queue
> > >      -> nvme_rdma_alloc_queue
> > >       -> rdma_create_id
> >
> > If it does work, it is by luck.
> 
> I didn't check every ULP, but it seems that all ULPs use the same
> module_init.
> 
> >
> > Keep in mind all this only matters for kernels without modules.
> 
> Can it be related to the fact that other ULPs call to ib_register_client()
> before calling to rdma-cm? RNBD does not have such call.

If the rdma_create_id() is not on a callchain from module_init then
you don't have a problem.

nvme has a bug here, IIRC. It is not OK to create RDMA CM IDs outside
a client - CM IDs are supposed to be cleaned up when the client is
removed.

Similarly they are supposed to be created from the client attachment.

Though listening CM IDs unbound to any device may change that
slightly, I think it is probably best practice to create the listening
ID only if a client is bound.

Most probably that is the best way to fix rnbd

> I'm not proposing this, but just loudly wondering, do we really need rdma-cm
> as a separate module? Can we bring it to be part of ib_core?

No idea.. It doesn't help this situation at least

Jason

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Haris Iqbal]

(Apologies for multiple emails. Was having trouble with an extension,
cause of which emails did not get delivered to the mailing list.
Resolved now.)

> Somehow nvme-rdma works:

I think that's because the callchain during the nvme_rdma_init_module
initialization stops at "nvmf_register_transport()". Here only the
"struct nvmf_transport_ops nvme_rdma_transport" is registered, which
contains the function "nvme_rdma_create_ctrl()". I tested this in my
local setup and during kernel boot, that's the extent of the
callchain.
The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
called later from "nvmf_dev_write()". I am not sure when this is
called, probably when the "discover" happens from the client side or
during the server config. I am trying to test this to confirm, will
send more details once I am done.
Am I missing something here?


> If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.

I am a little confused. I thought the problem occurs from a call to
either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
or a direct call to "rdma_bind_addr()" as in rtrs case.
In both the cases, a call to "rdma_create_id()" is needed before this.


> Similarly they are supposed to be created from the client attachment.
I am a beginner in terms of concepts here. Did you mean when a client
tries to establish the first connection to an rdma server?


On Thu, Jun 18, 2020 at 12:56 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Wed, Jun 17, 2020 at 10:07:56PM +0300, Leon Romanovsky wrote:
> > On Wed, Jun 17, 2020 at 03:20:46PM -0300, Jason Gunthorpe wrote:
> > > On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > > > > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > >
> > > > > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > >
> > > > > The rnbd_server module's communication manager initialization depends on the
> > > > > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > > > > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > > > > cma module during initialization; and if the rnbd_server module is initialized
> > > > > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > > > > operation.
> > > > > This patch delays the initialization of the rnbd_server module to the
> > > > > late_initcall level, since RDMA cma module uses module_init which puts it into
> > > > > the device_initcall level.
> > > > >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > > > > index 86e61523907b..213df05e5994 100644
> > > > > +++ b/drivers/block/rnbd/rnbd-srv.c
> > > > > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> > > > >         rnbd_srv_destroy_sysfs_files();
> > > > >  }
> > > > >
> > > > > -module_init(rnbd_srv_init_module);
> > > > > +late_initcall(rnbd_srv_init_module);
> > > >
> > > > I don't think that this is correct change. Somehow nvme-rdma works:
> > > > module_init(nvme_rdma_init_module);
> > > > -> nvme_rdma_init_module
> > > >  -> nvmf_register_transport(&nvme_rdma_transport);
> > > >   -> nvme_rdma_create_ctrl
> > > >    -> nvme_rdma_setup_ctrl
> > > >     -> nvme_rdma_configure_admin_queue
> > > >      -> nvme_rdma_alloc_queue
> > > >       -> rdma_create_id
> > >
> > > If it does work, it is by luck.
> >
> > I didn't check every ULP, but it seems that all ULPs use the same
> > module_init.
> >
> > >
> > > Keep in mind all this only matters for kernels without modules.
> >
> > Can it be related to the fact that other ULPs call to ib_register_client()
> > before calling to rdma-cm? RNBD does not have such call.
>
> If the rdma_create_id() is not on a callchain from module_init then
> you don't have a problem.
>
> nvme has a bug here, IIRC. It is not OK to create RDMA CM IDs outside
> a client - CM IDs are supposed to be cleaned up when the client is
> removed.
>
> Similarly they are supposed to be created from the client attachment.
>
> Though listening CM IDs unbound to any device may change that
> slightly, I think it is probably best practice to create the listening
> ID only if a client is bound.
>
> Most probably that is the best way to fix rnbd
>
> > I'm not proposing this, but just loudly wondering, do we really need rdma-cm
> > as a separate module? Can we bring it to be part of ib_core?
>
> No idea.. It doesn't help this situation at least
>
> Jason



-- 

Regards
-Haris

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Haris Iqbal]

It seems that the "rdma_bind_addr()" is called by the nvme rdma
module; but during the following events
1) When a discover happens from the client side. Call trace for that looks like,
[ 1098.409398] nvmf_dev_write
[ 1098.409403] nvmf_create_ctrl
[ 1098.414568] nvme_rdma_create_ctrl
[ 1098.415009] nvme_rdma_setup_ctrl
[ 1098.415010] nvme_rdma_configure_admin_queue
[ 1098.415010] nvme_rdma_alloc_queue
                             -->(rdma_create_id)
[ 1098.415032] rdma_resolve_addr
[ 1098.415032] cma_bind_addr
[ 1098.415033] rdma_bind_addr

2) When a connect happens from the client side. Call trace is the same
as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
n being the number of IO queues being created.


On the server side, when an nvmf port is enabled, that also triggers a
call to "rdma_bind_addr()", but that is not from the nvme rdma module.
may be nvme target rdma? (not sure).

On Thu, Jun 18, 2020 at 7:15 AM Haris Iqbal <haris.iqbal@cloud.ionos.com> wrote:
>
> (Apologies for multiple emails. Was having trouble with an extension,
> cause of which emails did not get delivered to the mailing list.
> Resolved now.)
>
> > Somehow nvme-rdma works:
>
> I think that's because the callchain during the nvme_rdma_init_module
> initialization stops at "nvmf_register_transport()". Here only the
> "struct nvmf_transport_ops nvme_rdma_transport" is registered, which
> contains the function "nvme_rdma_create_ctrl()". I tested this in my
> local setup and during kernel boot, that's the extent of the
> callchain.
> The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
> called later from "nvmf_dev_write()". I am not sure when this is
> called, probably when the "discover" happens from the client side or
> during the server config. I am trying to test this to confirm, will
> send more details once I am done.
> Am I missing something here?
>
>
> > If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.
>
> I am a little confused. I thought the problem occurs from a call to
> either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
> or a direct call to "rdma_bind_addr()" as in rtrs case.
> In both the cases, a call to "rdma_create_id()" is needed before this.
>
>
> > Similarly they are supposed to be created from the client attachment.
> I am a beginner in terms of concepts here. Did you mean when a client
> tries to establish the first connection to an rdma server?
>
>
> On Thu, Jun 18, 2020 at 12:56 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Wed, Jun 17, 2020 at 10:07:56PM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 17, 2020 at 03:20:46PM -0300, Jason Gunthorpe wrote:
> > > > On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> > > > > On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > > > > > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > >
> > > > > > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > >
> > > > > > The rnbd_server module's communication manager initialization depends on the
> > > > > > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > > > > > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > > > > > cma module during initialization; and if the rnbd_server module is initialized
> > > > > > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > > > > > operation.
> > > > > > This patch delays the initialization of the rnbd_server module to the
> > > > > > late_initcall level, since RDMA cma module uses module_init which puts it into
> > > > > > the device_initcall level.
> > > > > >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > > > > > index 86e61523907b..213df05e5994 100644
> > > > > > +++ b/drivers/block/rnbd/rnbd-srv.c
> > > > > > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> > > > > >         rnbd_srv_destroy_sysfs_files();
> > > > > >  }
> > > > > >
> > > > > > -module_init(rnbd_srv_init_module);
> > > > > > +late_initcall(rnbd_srv_init_module);
> > > > >
> > > > > I don't think that this is correct change. Somehow nvme-rdma works:
> > > > > module_init(nvme_rdma_init_module);
> > > > > -> nvme_rdma_init_module
> > > > >  -> nvmf_register_transport(&nvme_rdma_transport);
> > > > >   -> nvme_rdma_create_ctrl
> > > > >    -> nvme_rdma_setup_ctrl
> > > > >     -> nvme_rdma_configure_admin_queue
> > > > >      -> nvme_rdma_alloc_queue
> > > > >       -> rdma_create_id
> > > >
> > > > If it does work, it is by luck.
> > >
> > > I didn't check every ULP, but it seems that all ULPs use the same
> > > module_init.
> > >
> > > >
> > > > Keep in mind all this only matters for kernels without modules.
> > >
> > > Can it be related to the fact that other ULPs call to ib_register_client()
> > > before calling to rdma-cm? RNBD does not have such call.
> >
> > If the rdma_create_id() is not on a callchain from module_init then
> > you don't have a problem.
> >
> > nvme has a bug here, IIRC. It is not OK to create RDMA CM IDs outside
> > a client - CM IDs are supposed to be cleaned up when the client is
> > removed.
> >
> > Similarly they are supposed to be created from the client attachment.
> >
> > Though listening CM IDs unbound to any device may change that
> > slightly, I think it is probably best practice to create the listening
> > ID only if a client is bound.
> >
> > Most probably that is the best way to fix rnbd
> >
> > > I'm not proposing this, but just loudly wondering, do we really need rdma-cm
> > > as a separate module? Can we bring it to be part of ib_core?
> >
> > No idea.. It doesn't help this situation at least
> >
> > Jason
>
>
>
> --
>
> Regards
> -Haris



-- 

Regards
-Haris

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Haris Iqbal]

Hi Jason and Leon,

Did you get a chance to look into my previous email?

On Thu, Jun 18, 2020 at 2:44 PM Haris Iqbal <haris.iqbal@cloud.ionos.com> wrote:
>
> It seems that the "rdma_bind_addr()" is called by the nvme rdma
> module; but during the following events
> 1) When a discover happens from the client side. Call trace for that looks like,
> [ 1098.409398] nvmf_dev_write
> [ 1098.409403] nvmf_create_ctrl
> [ 1098.414568] nvme_rdma_create_ctrl
> [ 1098.415009] nvme_rdma_setup_ctrl
> [ 1098.415010] nvme_rdma_configure_admin_queue
> [ 1098.415010] nvme_rdma_alloc_queue
>                              -->(rdma_create_id)
> [ 1098.415032] rdma_resolve_addr
> [ 1098.415032] cma_bind_addr
> [ 1098.415033] rdma_bind_addr
>
> 2) When a connect happens from the client side. Call trace is the same
> as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
> n being the number of IO queues being created.
>
>
> On the server side, when an nvmf port is enabled, that also triggers a
> call to "rdma_bind_addr()", but that is not from the nvme rdma module.
> may be nvme target rdma? (not sure).
>
> On Thu, Jun 18, 2020 at 7:15 AM Haris Iqbal <haris.iqbal@cloud.ionos.com> wrote:
> >
> > (Apologies for multiple emails. Was having trouble with an extension,
> > cause of which emails did not get delivered to the mailing list.
> > Resolved now.)
> >
> > > Somehow nvme-rdma works:
> >
> > I think that's because the callchain during the nvme_rdma_init_module
> > initialization stops at "nvmf_register_transport()". Here only the
> > "struct nvmf_transport_ops nvme_rdma_transport" is registered, which
> > contains the function "nvme_rdma_create_ctrl()". I tested this in my
> > local setup and during kernel boot, that's the extent of the
> > callchain.
> > The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
> > called later from "nvmf_dev_write()". I am not sure when this is
> > called, probably when the "discover" happens from the client side or
> > during the server config. I am trying to test this to confirm, will
> > send more details once I am done.
> > Am I missing something here?
> >
> >
> > > If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.
> >
> > I am a little confused. I thought the problem occurs from a call to
> > either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
> > or a direct call to "rdma_bind_addr()" as in rtrs case.
> > In both the cases, a call to "rdma_create_id()" is needed before this.
> >
> >
> > > Similarly they are supposed to be created from the client attachment.
> > I am a beginner in terms of concepts here. Did you mean when a client
> > tries to establish the first connection to an rdma server?
> >
> >
> > On Thu, Jun 18, 2020 at 12:56 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Wed, Jun 17, 2020 at 10:07:56PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Jun 17, 2020 at 03:20:46PM -0300, Jason Gunthorpe wrote:
> > > > > On Wed, Jun 17, 2020 at 02:28:11PM +0300, Leon Romanovsky wrote:
> > > > > > On Wed, Jun 17, 2020 at 04:07:32PM +0530, haris.iqbal@cloud.ionos.com wrote:
> > > > > > > From: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > >
> > > > > > > Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
> > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > >
> > > > > > > The rnbd_server module's communication manager initialization depends on the
> > > > > > > registration of the "network namespace subsystem" of the RDMA CM agent module.
> > > > > > > As such, when the kernel is configured to load the rnbd_server and the RDMA
> > > > > > > cma module during initialization; and if the rnbd_server module is initialized
> > > > > > > before RDMA cma module, a null ptr dereference occurs during the RDMA bind
> > > > > > > operation.
> > > > > > > This patch delays the initialization of the rnbd_server module to the
> > > > > > > late_initcall level, since RDMA cma module uses module_init which puts it into
> > > > > > > the device_initcall level.
> > > > > > >  drivers/block/rnbd/rnbd-srv.c | 2 +-
> > > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > > >
> > > > > > > diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
> > > > > > > index 86e61523907b..213df05e5994 100644
> > > > > > > +++ b/drivers/block/rnbd/rnbd-srv.c
> > > > > > > @@ -840,5 +840,5 @@ static void __exit rnbd_srv_cleanup_module(void)
> > > > > > >         rnbd_srv_destroy_sysfs_files();
> > > > > > >  }
> > > > > > >
> > > > > > > -module_init(rnbd_srv_init_module);
> > > > > > > +late_initcall(rnbd_srv_init_module);
> > > > > >
> > > > > > I don't think that this is correct change. Somehow nvme-rdma works:
> > > > > > module_init(nvme_rdma_init_module);
> > > > > > -> nvme_rdma_init_module
> > > > > >  -> nvmf_register_transport(&nvme_rdma_transport);
> > > > > >   -> nvme_rdma_create_ctrl
> > > > > >    -> nvme_rdma_setup_ctrl
> > > > > >     -> nvme_rdma_configure_admin_queue
> > > > > >      -> nvme_rdma_alloc_queue
> > > > > >       -> rdma_create_id
> > > > >
> > > > > If it does work, it is by luck.
> > > >
> > > > I didn't check every ULP, but it seems that all ULPs use the same
> > > > module_init.
> > > >
> > > > >
> > > > > Keep in mind all this only matters for kernels without modules.
> > > >
> > > > Can it be related to the fact that other ULPs call to ib_register_client()
> > > > before calling to rdma-cm? RNBD does not have such call.
> > >
> > > If the rdma_create_id() is not on a callchain from module_init then
> > > you don't have a problem.
> > >
> > > nvme has a bug here, IIRC. It is not OK to create RDMA CM IDs outside
> > > a client - CM IDs are supposed to be cleaned up when the client is
> > > removed.
> > >
> > > Similarly they are supposed to be created from the client attachment.
> > >
> > > Though listening CM IDs unbound to any device may change that
> > > slightly, I think it is probably best practice to create the listening
> > > ID only if a client is bound.
> > >
> > > Most probably that is the best way to fix rnbd
> > >
> > > > I'm not proposing this, but just loudly wondering, do we really need rdma-cm
> > > > as a separate module? Can we bring it to be part of ib_core?
> > >
> > > No idea.. It doesn't help this situation at least
> > >
> > > Jason
> >
> >
> >
> > --
> >
> > Regards
> > -Haris
>
>
>
> --
>
> Regards
> -Haris



-- 

Regards
-Haris

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jason Gunthorpe]

On Tue, Jun 23, 2020 at 03:20:27PM +0530, Haris Iqbal wrote:
> Hi Jason and Leon,
> 
> Did you get a chance to look into my previous email?

Was there a question?

Jason

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Haris Iqbal]

On Tue, Jun 23, 2020 at 5:47 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Jun 23, 2020 at 03:20:27PM +0530, Haris Iqbal wrote:
> > Hi Jason and Leon,
> >
> > Did you get a chance to look into my previous email?
>
> Was there a question?

Multiple actually :)

>
> Jason

In response to your emails,

> Somehow nvme-rdma works:

I think that's because the callchain during the nvme_rdma_init_module
initialization stops at "nvmf_register_transport()". Here only the
"struct nvmf_transport_ops nvme_rdma_transport" is registered, which
contains the function "nvme_rdma_create_ctrl()". I tested this in my
local setup and during kernel boot, that's the extent of the
callchain.
The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
called later from "nvmf_dev_write()". I am not sure when this is
called, probably when the "discover" happens from the client side or
during the server config.

It seems that the "rdma_bind_addr()" is called by the nvme rdma
module; but during the following events
1) When a discover happens from the client side. Call trace for that looks like,
[ 1098.409398] nvmf_dev_write
[ 1098.409403] nvmf_create_ctrl
[ 1098.414568] nvme_rdma_create_ctrl
[ 1098.415009] nvme_rdma_setup_ctrl
[ 1098.415010] nvme_rdma_configure_admin_queue
[ 1098.415010] nvme_rdma_alloc_queue
                             -->(rdma_create_id)
[ 1098.415032] rdma_resolve_addr
[ 1098.415032] cma_bind_addr
[ 1098.415033] rdma_bind_addr

2) When a connect happens from the client side. Call trace is the same
as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
n being the number of IO queues being created.

On the server side, when an nvmf port is enabled, that also triggers a
call to "rdma_bind_addr()", but that is not from the nvme rdma module.
may be nvme target rdma? (not sure).

Does this make sense or am I missing something here?


> If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.

I am a little confused. I thought the problem occurs from a call to
either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
or a direct call to "rdma_bind_addr()" as in rtrs case.
In both the cases, a call to "rdma_create_id()" is needed before this.


> Similarly they are supposed to be created from the client attachment.
I am a beginner in terms of concepts here. Did you mean when a client
tries to establish the first connection to an rdma server?


-- 

Regards
-Haris

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jason Gunthorpe]

On Tue, Jun 23, 2020 at 07:15:03PM +0530, Haris Iqbal wrote:
> On Tue, Jun 23, 2020 at 5:47 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Tue, Jun 23, 2020 at 03:20:27PM +0530, Haris Iqbal wrote:
> > > Hi Jason and Leon,
> > >
> > > Did you get a chance to look into my previous email?
> >
> > Was there a question?
> 
> Multiple actually :)
> 
> >
> > Jason
> 
> In response to your emails,
> 
> > Somehow nvme-rdma works:
> 
> I think that's because the callchain during the nvme_rdma_init_module
> initialization stops at "nvmf_register_transport()". Here only the
> "struct nvmf_transport_ops nvme_rdma_transport" is registered, which
> contains the function "nvme_rdma_create_ctrl()". I tested this in my
> local setup and during kernel boot, that's the extent of the
> callchain.
> The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
> called later from "nvmf_dev_write()". I am not sure when this is
> called, probably when the "discover" happens from the client side or
> during the server config.
> 
> It seems that the "rdma_bind_addr()" is called by the nvme rdma
> module; but during the following events
> 1) When a discover happens from the client side. Call trace for that looks like,
> [ 1098.409398] nvmf_dev_write
> [ 1098.409403] nvmf_create_ctrl
> [ 1098.414568] nvme_rdma_create_ctrl
> [ 1098.415009] nvme_rdma_setup_ctrl
> [ 1098.415010] nvme_rdma_configure_admin_queue
> [ 1098.415010] nvme_rdma_alloc_queue
> [ 1098.415032] rdma_resolve_addr
> [ 1098.415032] cma_bind_addr
> [ 1098.415033] rdma_bind_addr
> 
> 2) When a connect happens from the client side. Call trace is the same
> as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
> n being the number of IO queues being created.
> 
> On the server side, when an nvmf port is enabled, that also triggers a
> call to "rdma_bind_addr()", but that is not from the nvme rdma module.
> may be nvme target rdma? (not sure).
> 
> Does this make sense or am I missing something here?

It make sense, delaying creating and CM ID's until user space starts
will solve this init time problme

> 
> > If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.
> 
> I am a little confused. I thought the problem occurs from a call to
> either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
> or a direct call to "rdma_bind_addr()" as in rtrs case.
> In both the cases, a call to "rdma_create_id()" is needed before this.

Right rdma_create_id() must precede anything that has problems, and it
should not be done from module_init.

Jason

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Haris Iqbal]

On Tue, Jun 23, 2020 at 7:54 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Jun 23, 2020 at 07:15:03PM +0530, Haris Iqbal wrote:
> > On Tue, Jun 23, 2020 at 5:47 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Tue, Jun 23, 2020 at 03:20:27PM +0530, Haris Iqbal wrote:
> > > > Hi Jason and Leon,
> > > >
> > > > Did you get a chance to look into my previous email?
> > >
> > > Was there a question?
> >
> > Multiple actually :)
> >
> > >
> > > Jason
> >
> > In response to your emails,
> >
> > > Somehow nvme-rdma works:
> >
> > I think that's because the callchain during the nvme_rdma_init_module
> > initialization stops at "nvmf_register_transport()". Here only the
> > "struct nvmf_transport_ops nvme_rdma_transport" is registered, which
> > contains the function "nvme_rdma_create_ctrl()". I tested this in my
> > local setup and during kernel boot, that's the extent of the
> > callchain.
> > The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
> > called later from "nvmf_dev_write()". I am not sure when this is
> > called, probably when the "discover" happens from the client side or
> > during the server config.
> >
> > It seems that the "rdma_bind_addr()" is called by the nvme rdma
> > module; but during the following events
> > 1) When a discover happens from the client side. Call trace for that looks like,
> > [ 1098.409398] nvmf_dev_write
> > [ 1098.409403] nvmf_create_ctrl
> > [ 1098.414568] nvme_rdma_create_ctrl
> > [ 1098.415009] nvme_rdma_setup_ctrl
> > [ 1098.415010] nvme_rdma_configure_admin_queue
> > [ 1098.415010] nvme_rdma_alloc_queue
> > [ 1098.415032] rdma_resolve_addr
> > [ 1098.415032] cma_bind_addr
> > [ 1098.415033] rdma_bind_addr
> >
> > 2) When a connect happens from the client side. Call trace is the same
> > as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
> > n being the number of IO queues being created.
> >
> > On the server side, when an nvmf port is enabled, that also triggers a
> > call to "rdma_bind_addr()", but that is not from the nvme rdma module.
> > may be nvme target rdma? (not sure).
> >
> > Does this make sense or am I missing something here?
>
> It make sense, delaying creating and CM ID's until user space starts
> will solve this init time problme

Right, and the patch is trying to achieve the delay by changing the
init level to "late_initcall()"

>
> >
> > > If the rdma_create_id() is not on a callchain from module_init then you don't have a problem.
> >
> > I am a little confused. I thought the problem occurs from a call to
> > either "rdma_resolve_addr()" which calls "rdma_bind_addr()",
> > or a direct call to "rdma_bind_addr()" as in rtrs case.
> > In both the cases, a call to "rdma_create_id()" is needed before this.
>
> Right rdma_create_id() must precede anything that has problems, and it
> should not be done from module_init.

I understand this, but I am not sure why that is; as in why it should
not be done from module_init?

Also, about one of your previous statements,

> It is not OK to create RDMA CM IDs outside
> a client - CM IDs are supposed to be cleaned up when the client is
> removed.
>
> Similarly they are supposed to be created from the client attachment.

This again is a little confusing to me, since what I've observed in
nvmt is, when a server port is created, the "rdma_bind_addr()"
function is called.
And this goes well with the server/target and client/initiator model,
where the server has to get ready and start listening before a client
can initiate a connection.
What am I missing here?

>
> Jason

-- 

Regards
-Haris

Re: [PATCH] Delay the initialization of rnbd_server module to late_initcall level

[Jason Gunthorpe]

On Tue, Jun 23, 2020 at 05:05:51PM +0530, Haris Iqbal wrote:
> On Tue, Jun 23, 2020 at 7:54 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Tue, Jun 23, 2020 at 07:15:03PM +0530, Haris Iqbal wrote:
> > > On Tue, Jun 23, 2020 at 5:47 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > > >
> > > > On Tue, Jun 23, 2020 at 03:20:27PM +0530, Haris Iqbal wrote:
> > > > > Hi Jason and Leon,
> > > > >
> > > > > Did you get a chance to look into my previous email?
> > > >
> > > > Was there a question?
> > >
> > > Multiple actually :)
> > >
> > > >
> > > > Jason
> > >
> > > In response to your emails,
> > >
> > > > Somehow nvme-rdma works:
> > >
> > > I think that's because the callchain during the nvme_rdma_init_module
> > > initialization stops at "nvmf_register_transport()". Here only the
> > > "struct nvmf_transport_ops nvme_rdma_transport" is registered, which
> > > contains the function "nvme_rdma_create_ctrl()". I tested this in my
> > > local setup and during kernel boot, that's the extent of the
> > > callchain.
> > > The ".create_ctrl"; which now points to "nvme_rdma_create_ctrl()" is
> > > called later from "nvmf_dev_write()". I am not sure when this is
> > > called, probably when the "discover" happens from the client side or
> > > during the server config.
> > >
> > > It seems that the "rdma_bind_addr()" is called by the nvme rdma
> > > module; but during the following events
> > > 1) When a discover happens from the client side. Call trace for that looks like,
> > > [ 1098.409398] nvmf_dev_write
> > > [ 1098.409403] nvmf_create_ctrl
> > > [ 1098.414568] nvme_rdma_create_ctrl
> > > [ 1098.415009] nvme_rdma_setup_ctrl
> > > [ 1098.415010] nvme_rdma_configure_admin_queue
> > > [ 1098.415010] nvme_rdma_alloc_queue
> > > [ 1098.415032] rdma_resolve_addr
> > > [ 1098.415032] cma_bind_addr
> > > [ 1098.415033] rdma_bind_addr
> > >
> > > 2) When a connect happens from the client side. Call trace is the same
> > > as above, plus "nvme_rdma_alloc_queue()" is called n number of times;
> > > n being the number of IO queues being created.
> > >
> > > On the server side, when an nvmf port is enabled, that also triggers a
> > > call to "rdma_bind_addr()", but that is not from the nvme rdma module.
> > > may be nvme target rdma? (not sure).
> > >
> > > Does this make sense or am I missing something here?
> >
> > It make sense, delaying creating and CM ID's until user space starts
> > will solve this init time problme
> 
> Right, and the patch is trying to achieve the delay by changing the
> init level to "late_initcall()"

It should not be done with initcall levels

> > Right rdma_create_id() must precede anything that has problems, and it
> > should not be done from module_init.
> 
> I understand this, but I am not sure why that is; as in why it should
> not be done from module_init?

Because that is how our module ordering scheme works

> > It is not OK to create RDMA CM IDs outside
> > a client - CM IDs are supposed to be cleaned up when the client is
> > removed.
> >
> > Similarly they are supposed to be created from the client attachment.
> 
> This again is a little confusing to me, since what I've observed in
> nvmt is, when a server port is created, the "rdma_bind_addr()"
> function is called.
> And this goes well with the server/target and client/initiator model,
> where the server has to get ready and start listening before a client
> can initiate a connection.
> What am I missing here?

client means a struct ib_client

Jason

> 
> >
> > Jason
> 

[PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Md Haris Iqbal]

The rnbd_server module's communication manager (cm) initialization depends
on the registration of the "network namespace subsystem" of the RDMA CM
agent module. As such, when the kernel is configured to load the
rnbd_server and the RDMA cma module during initialization; and if the
rnbd_server module is initialized before RDMA cma module, a null ptr
dereference occurs during the RDMA bind operation.

Call trace below,

[    1.904782] Call Trace:
[    1.904782]  ? xas_load+0xd/0x80
[    1.904782]  xa_load+0x47/0x80
[    1.904782]  cma_ps_find+0x44/0x70
[    1.904782]  rdma_bind_addr+0x782/0x8b0
[    1.904782]  ? get_random_bytes+0x35/0x40
[    1.904782]  rtrs_srv_cm_init+0x50/0x80
[    1.904782]  rtrs_srv_open+0x102/0x180
[    1.904782]  ? rnbd_client_init+0x6e/0x6e
[    1.904782]  rnbd_srv_init_module+0x34/0x84
[    1.904782]  ? rnbd_client_init+0x6e/0x6e
[    1.904782]  do_one_initcall+0x4a/0x200
[    1.904782]  kernel_init_freeable+0x1f1/0x26e
[    1.904782]  ? rest_init+0xb0/0xb0
[    1.904782]  kernel_init+0xe/0x100
[    1.904782]  ret_from_fork+0x22/0x30
[    1.904782] Modules linked in:
[    1.904782] CR2: 0000000000000015
[    1.904782] ---[ end trace c42df88d6c7b0a48 ]---

All this happens cause the cm init is in the call chain of the module init,
which is not a preferred practice.

So remove the call to rdma_create_id() from the module init call chain.
Instead register rtrs-srv as an ib client, which makes sure that the
rdma_create_id() is called only when an ib device is added.

Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
---
 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
 drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
 2 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
index 0d9241f5d9e6..916f99464d09 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
@@ -16,6 +16,7 @@
 #include "rtrs-srv.h"
 #include "rtrs-log.h"
 #include <rdma/ib_cm.h>
+#include <rdma/ib_verbs.h>
 
 MODULE_DESCRIPTION("RDMA Transport Server");
 MODULE_LICENSE("GPL");
@@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
 static struct rtrs_rdma_dev_pd dev_pd;
 static mempool_t *chunk_pool;
 struct class *rtrs_dev_class;
+static struct rtrs_srv_ib_ctx ib_ctx;
 
 static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
 static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
@@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
 	kfree(ctx);
 }
 
+static int rtrs_srv_add_one(struct ib_device *device)
+{
+	struct rtrs_srv_ctx *ctx;
+	int ret;
+
+	/*
+	 * Keep a track on the number of ib devices added
+	 */
+	ib_ctx.ib_dev_count++;
+
+	if (!ib_ctx.rdma_init) {
+		/*
+		 * Since our CM IDs are NOT bound to any ib device we will create them
+		 * only once
+		 */
+		ctx = ib_ctx.srv_ctx;
+		ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
+		if (ret) {
+			/*
+			 * We errored out here.
+			 * According to the ib code, if we encounter an error here then the
+			 * error code is ignored, and no more calls to our ops are made.
+			 */
+			pr_err("Failed to initialize RDMA connection");
+			return ret;
+		}
+		ib_ctx.rdma_init = true;
+	}
+
+	return 0;
+}
+
+static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
+{
+	struct rtrs_srv_ctx *ctx;
+
+	ib_ctx.ib_dev_count--;
+
+	if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
+		/*
+		 * Since our CM IDs are NOT bound to any ib device we will remove them
+		 * only once, when the last device is removed
+		 */
+		ctx = ib_ctx.srv_ctx;
+		rdma_destroy_id(ctx->cm_id_ip);
+		rdma_destroy_id(ctx->cm_id_ib);
+		ib_ctx.rdma_init = false;
+	}
+}
+
+static struct ib_client rtrs_srv_client = {
+	.name	= "rtrs_server",
+	.add	= rtrs_srv_add_one,
+	.remove	= rtrs_srv_remove_one
+};
+
 /**
  * rtrs_srv_open() - open RTRS server context
  * @ops:		callback functions
@@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
 
-	err = rtrs_srv_rdma_init(ctx, port);
+	ib_ctx = (struct rtrs_srv_ib_ctx) {
+		.srv_ctx	= ctx,
+		.port		= port,
+	};
+
+	err = ib_register_client(&rtrs_srv_client);
 	if (err) {
 		free_srv_ctx(ctx);
 		return ERR_PTR(err);
 	}
 
+	/*
+	 * Since ib_register_client does not propagate the device add error
+	 * we check if the RDMA connection init was successful or not
+	 */
+	if (!ib_ctx.rdma_init) {
+		free_srv_ctx(ctx);
+		return NULL;
+	}
+
 	return ctx;
 }
 EXPORT_SYMBOL(rtrs_srv_open);
@@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
  */
 void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
 {
-	rdma_destroy_id(ctx->cm_id_ip);
-	rdma_destroy_id(ctx->cm_id_ib);
+	ib_unregister_client(&rtrs_srv_client);
 	close_ctx(ctx);
 	free_srv_ctx(ctx);
 }
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
index dc95b0932f0d..6e9d9000cd8d 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
@@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
 	struct list_head srv_list;
 };
 
+struct rtrs_srv_ib_ctx {
+	struct rtrs_srv_ctx	*srv_ctx;
+	u16			port;
+	int			ib_dev_count;
+	bool			rdma_init;
+};
+
 extern struct class *rtrs_dev_class;
 
 void close_sess(struct rtrs_srv_sess *sess);
-- 
2.25.1

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> The rnbd_server module's communication manager (cm) initialization depends
> on the registration of the "network namespace subsystem" of the RDMA CM
> agent module. As such, when the kernel is configured to load the
> rnbd_server and the RDMA cma module during initialization; and if the
> rnbd_server module is initialized before RDMA cma module, a null ptr
> dereference occurs during the RDMA bind operation.
>
> Call trace below,
>
> [    1.904782] Call Trace:
> [    1.904782]  ? xas_load+0xd/0x80
> [    1.904782]  xa_load+0x47/0x80
> [    1.904782]  cma_ps_find+0x44/0x70
> [    1.904782]  rdma_bind_addr+0x782/0x8b0
> [    1.904782]  ? get_random_bytes+0x35/0x40
> [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> [    1.904782]  rtrs_srv_open+0x102/0x180
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  rnbd_srv_init_module+0x34/0x84
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  do_one_initcall+0x4a/0x200
> [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> [    1.904782]  ? rest_init+0xb0/0xb0
> [    1.904782]  kernel_init+0xe/0x100
> [    1.904782]  ret_from_fork+0x22/0x30
> [    1.904782] Modules linked in:
> [    1.904782] CR2: 0000000000000015
> [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
>
> All this happens cause the cm init is in the call chain of the module init,
> which is not a preferred practice.
>
> So remove the call to rdma_create_id() from the module init call chain.
> Instead register rtrs-srv as an ib client, which makes sure that the
> rdma_create_id() is called only when an ib device is added.
>
> Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> ---
>  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
>  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
>  2 files changed, 81 insertions(+), 3 deletions(-)

Please don't send vX patches as reply-to in "git send-email" command.

>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> index 0d9241f5d9e6..916f99464d09 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> @@ -16,6 +16,7 @@
>  #include "rtrs-srv.h"
>  #include "rtrs-log.h"
>  #include <rdma/ib_cm.h>
> +#include <rdma/ib_verbs.h>
>
>  MODULE_DESCRIPTION("RDMA Transport Server");
>  MODULE_LICENSE("GPL");
> @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
>  static struct rtrs_rdma_dev_pd dev_pd;
>  static mempool_t *chunk_pool;
>  struct class *rtrs_dev_class;
> +static struct rtrs_srv_ib_ctx ib_ctx;
>
>  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
>  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
>  	kfree(ctx);
>  }
>
> +static int rtrs_srv_add_one(struct ib_device *device)
> +{
> +	struct rtrs_srv_ctx *ctx;
> +	int ret;
> +
> +	/*
> +	 * Keep a track on the number of ib devices added
> +	 */
> +	ib_ctx.ib_dev_count++;
> +
> +	if (!ib_ctx.rdma_init) {
> +		/*
> +		 * Since our CM IDs are NOT bound to any ib device we will create them
> +		 * only once
> +		 */
> +		ctx = ib_ctx.srv_ctx;
> +		ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> +		if (ret) {
> +			/*
> +			 * We errored out here.
> +			 * According to the ib code, if we encounter an error here then the
> +			 * error code is ignored, and no more calls to our ops are made.
> +			 */
> +			pr_err("Failed to initialize RDMA connection");
> +			return ret;
> +		}
> +		ib_ctx.rdma_init = true;

This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
logic can be simplified.

if (ib_ctx.ib_dev_count)
	return 0;

ctx = ib_ctx.srv_ctx;
ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
if (ret)
	return ret;
ib_ctx.ib_dev_count++;
return 0;

> +	}
> +
> +	return 0;
> +}
> +
> +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> +{
> +	struct rtrs_srv_ctx *ctx;
> +
> +	ib_ctx.ib_dev_count--;
> +
> +	if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {

It is not kernel coding style.
if (ib_ctx.ib_dev_count)
	return;

ctx = ib_ctx.srv_ctx;
rdma_destroy_id(ctx->cm_id_ip);
rdma_destroy_id(ctx->cm_id_ib);

Thanks

> +		/*
> +		 * Since our CM IDs are NOT bound to any ib device we will remove them
> +		 * only once, when the last device is removed
> +		 */
> +		ctx = ib_ctx.srv_ctx;
> +		rdma_destroy_id(ctx->cm_id_ip);
> +		rdma_destroy_id(ctx->cm_id_ib);
> +		ib_ctx.rdma_init = false;
> +	}
> +}
> +
> +static struct ib_client rtrs_srv_client = {
> +	.name	= "rtrs_server",
> +	.add	= rtrs_srv_add_one,
> +	.remove	= rtrs_srv_remove_one
> +};
> +
>  /**
>   * rtrs_srv_open() - open RTRS server context
>   * @ops:		callback functions
> @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
>  	if (!ctx)
>  		return ERR_PTR(-ENOMEM);
>
> -	err = rtrs_srv_rdma_init(ctx, port);
> +	ib_ctx = (struct rtrs_srv_ib_ctx) {
> +		.srv_ctx	= ctx,
> +		.port		= port,
> +	};
> +
> +	err = ib_register_client(&rtrs_srv_client);
>  	if (err) {
>  		free_srv_ctx(ctx);
>  		return ERR_PTR(err);
>  	}
>
> +	/*
> +	 * Since ib_register_client does not propagate the device add error
> +	 * we check if the RDMA connection init was successful or not
> +	 */
> +	if (!ib_ctx.rdma_init) {
> +		free_srv_ctx(ctx);
> +		return NULL;
> +	}
> +
>  	return ctx;
>  }
>  EXPORT_SYMBOL(rtrs_srv_open);
> @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
>   */
>  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
>  {
> -	rdma_destroy_id(ctx->cm_id_ip);
> -	rdma_destroy_id(ctx->cm_id_ib);
> +	ib_unregister_client(&rtrs_srv_client);
>  	close_ctx(ctx);
>  	free_srv_ctx(ctx);
>  }
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> index dc95b0932f0d..6e9d9000cd8d 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
>  	struct list_head srv_list;
>  };
>
> +struct rtrs_srv_ib_ctx {
> +	struct rtrs_srv_ctx	*srv_ctx;
> +	u16			port;
> +	int			ib_dev_count;
> +	bool			rdma_init;
> +};
> +
>  extern struct class *rtrs_dev_class;
>
>  void close_sess(struct rtrs_srv_sess *sess);
> --
> 2.25.1
>

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > The rnbd_server module's communication manager (cm) initialization depends
> > on the registration of the "network namespace subsystem" of the RDMA CM
> > agent module. As such, when the kernel is configured to load the
> > rnbd_server and the RDMA cma module during initialization; and if the
> > rnbd_server module is initialized before RDMA cma module, a null ptr
> > dereference occurs during the RDMA bind operation.
> >
> > Call trace below,
> >
> > [    1.904782] Call Trace:
> > [    1.904782]  ? xas_load+0xd/0x80
> > [    1.904782]  xa_load+0x47/0x80
> > [    1.904782]  cma_ps_find+0x44/0x70
> > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > [    1.904782]  ? get_random_bytes+0x35/0x40
> > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > [    1.904782]  rtrs_srv_open+0x102/0x180
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  do_one_initcall+0x4a/0x200
> > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > [    1.904782]  ? rest_init+0xb0/0xb0
> > [    1.904782]  kernel_init+0xe/0x100
> > [    1.904782]  ret_from_fork+0x22/0x30
> > [    1.904782] Modules linked in:
> > [    1.904782] CR2: 0000000000000015
> > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> >
> > All this happens cause the cm init is in the call chain of the module init,
> > which is not a preferred practice.
> >
> > So remove the call to rdma_create_id() from the module init call chain.
> > Instead register rtrs-srv as an ib client, which makes sure that the
> > rdma_create_id() is called only when an ib device is added.
> >
> > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > ---
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> >  2 files changed, 81 insertions(+), 3 deletions(-)
>
> Please don't send vX patches as reply-to in "git send-email" command.
>
> >
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > index 0d9241f5d9e6..916f99464d09 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > @@ -16,6 +16,7 @@
> >  #include "rtrs-srv.h"
> >  #include "rtrs-log.h"
> >  #include <rdma/ib_cm.h>
> > +#include <rdma/ib_verbs.h>
> >
> >  MODULE_DESCRIPTION("RDMA Transport Server");
> >  MODULE_LICENSE("GPL");
> > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> >  static struct rtrs_rdma_dev_pd dev_pd;
> >  static mempool_t *chunk_pool;
> >  struct class *rtrs_dev_class;
> > +static struct rtrs_srv_ib_ctx ib_ctx;
> >
> >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> >       kfree(ctx);
> >  }
> >
> > +static int rtrs_srv_add_one(struct ib_device *device)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +     int ret;
> > +
> > +     /*
> > +      * Keep a track on the number of ib devices added
> > +      */
> > +     ib_ctx.ib_dev_count++;
> > +
> > +     if (!ib_ctx.rdma_init) {
> > +             /*
> > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > +              * only once
> > +              */
> > +             ctx = ib_ctx.srv_ctx;
> > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > +             if (ret) {
> > +                     /*
> > +                      * We errored out here.
> > +                      * According to the ib code, if we encounter an error here then the
> > +                      * error code is ignored, and no more calls to our ops are made.
> > +                      */
> > +                     pr_err("Failed to initialize RDMA connection");
> > +                     return ret;
> > +             }
> > +             ib_ctx.rdma_init = true;
>
> This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> logic can be simplified.

Yes, this was the first logic in my head. But I have few thoughts,
The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
successful execution of rtrs_srv_rdma_init() and not really an IB
device count. Meaning if we have multiple calls to add, due to
multiple devices, our count would stay 1. And while removal we might
end up calling rdma_destroy_id() on our first remove call even though
another device is still remaining.

If we increment "ib_ctx.ib_dev_count" every time add is called, even
before we call rtrs_srv_rdma_init() and irrespective of whether
rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
IB devices added. However, when remove is called, we now know the
number of devices added, but not whether rtrs_srv_rdma_init() was
successful or not. We may end up calling rdma_destroy_id() on NULL
cm_ids

Does this make sense or am I missing something?


>
> if (ib_ctx.ib_dev_count)
>         return 0;
>
> ctx = ib_ctx.srv_ctx;
> ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> if (ret)
>         return ret;
> ib_ctx.ib_dev_count++;
> return 0;
>
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +
> > +     ib_ctx.ib_dev_count--;
> > +
> > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
>
> It is not kernel coding style.
> if (ib_ctx.ib_dev_count)
>         return;
>
> ctx = ib_ctx.srv_ctx;
> rdma_destroy_id(ctx->cm_id_ip);
> rdma_destroy_id(ctx->cm_id_ib);
>
> Thanks
>
> > +             /*
> > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > +              * only once, when the last device is removed
> > +              */
> > +             ctx = ib_ctx.srv_ctx;
> > +             rdma_destroy_id(ctx->cm_id_ip);
> > +             rdma_destroy_id(ctx->cm_id_ib);
> > +             ib_ctx.rdma_init = false;
> > +     }
> > +}
> > +
> > +static struct ib_client rtrs_srv_client = {
> > +     .name   = "rtrs_server",
> > +     .add    = rtrs_srv_add_one,
> > +     .remove = rtrs_srv_remove_one
> > +};
> > +
> >  /**
> >   * rtrs_srv_open() - open RTRS server context
> >   * @ops:             callback functions
> > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> >       if (!ctx)
> >               return ERR_PTR(-ENOMEM);
> >
> > -     err = rtrs_srv_rdma_init(ctx, port);
> > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > +             .srv_ctx        = ctx,
> > +             .port           = port,
> > +     };
> > +
> > +     err = ib_register_client(&rtrs_srv_client);
> >       if (err) {
> >               free_srv_ctx(ctx);
> >               return ERR_PTR(err);
> >       }
> >
> > +     /*
> > +      * Since ib_register_client does not propagate the device add error
> > +      * we check if the RDMA connection init was successful or not
> > +      */
> > +     if (!ib_ctx.rdma_init) {
> > +             free_srv_ctx(ctx);
> > +             return NULL;
> > +     }
> > +
> >       return ctx;
> >  }
> >  EXPORT_SYMBOL(rtrs_srv_open);
> > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> >   */
> >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> >  {
> > -     rdma_destroy_id(ctx->cm_id_ip);
> > -     rdma_destroy_id(ctx->cm_id_ib);
> > +     ib_unregister_client(&rtrs_srv_client);
> >       close_ctx(ctx);
> >       free_srv_ctx(ctx);
> >  }
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > index dc95b0932f0d..6e9d9000cd8d 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> >       struct list_head srv_list;
> >  };
> >
> > +struct rtrs_srv_ib_ctx {
> > +     struct rtrs_srv_ctx     *srv_ctx;
> > +     u16                     port;
> > +     int                     ib_dev_count;
> > +     bool                    rdma_init;
> > +};
> > +
> >  extern struct class *rtrs_dev_class;
> >
> >  void close_sess(struct rtrs_srv_sess *sess);
> > --
> > 2.25.1
> >



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > The rnbd_server module's communication manager (cm) initialization depends
> > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > agent module. As such, when the kernel is configured to load the
> > > rnbd_server and the RDMA cma module during initialization; and if the
> > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > dereference occurs during the RDMA bind operation.
> > >
> > > Call trace below,
> > >
> > > [    1.904782] Call Trace:
> > > [    1.904782]  ? xas_load+0xd/0x80
> > > [    1.904782]  xa_load+0x47/0x80
> > > [    1.904782]  cma_ps_find+0x44/0x70
> > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > [    1.904782]  kernel_init+0xe/0x100
> > > [    1.904782]  ret_from_fork+0x22/0x30
> > > [    1.904782] Modules linked in:
> > > [    1.904782] CR2: 0000000000000015
> > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > >
> > > All this happens cause the cm init is in the call chain of the module init,
> > > which is not a preferred practice.
> > >
> > > So remove the call to rdma_create_id() from the module init call chain.
> > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > rdma_create_id() is called only when an ib device is added.
> > >
> > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > ---
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > >  2 files changed, 81 insertions(+), 3 deletions(-)
> >
> > Please don't send vX patches as reply-to in "git send-email" command.
> >
> > >
> > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > index 0d9241f5d9e6..916f99464d09 100644
> > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > @@ -16,6 +16,7 @@
> > >  #include "rtrs-srv.h"
> > >  #include "rtrs-log.h"
> > >  #include <rdma/ib_cm.h>
> > > +#include <rdma/ib_verbs.h>
> > >
> > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > >  MODULE_LICENSE("GPL");
> > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > >  static struct rtrs_rdma_dev_pd dev_pd;
> > >  static mempool_t *chunk_pool;
> > >  struct class *rtrs_dev_class;
> > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > >
> > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > >       kfree(ctx);
> > >  }
> > >
> > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > +{
> > > +     struct rtrs_srv_ctx *ctx;
> > > +     int ret;
> > > +
> > > +     /*
> > > +      * Keep a track on the number of ib devices added
> > > +      */
> > > +     ib_ctx.ib_dev_count++;
> > > +
> > > +     if (!ib_ctx.rdma_init) {
> > > +             /*
> > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > +              * only once
> > > +              */
> > > +             ctx = ib_ctx.srv_ctx;
> > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > +             if (ret) {
> > > +                     /*
> > > +                      * We errored out here.
> > > +                      * According to the ib code, if we encounter an error here then the
> > > +                      * error code is ignored, and no more calls to our ops are made.
> > > +                      */
> > > +                     pr_err("Failed to initialize RDMA connection");
> > > +                     return ret;
> > > +             }
> > > +             ib_ctx.rdma_init = true;
> >
> > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > logic can be simplified.
>
> Yes, this was the first logic in my head. But I have few thoughts,
> The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> successful execution of rtrs_srv_rdma_init() and not really an IB
> device count. Meaning if we have multiple calls to add, due to
> multiple devices, our count would stay 1. And while removal we might
> end up calling rdma_destroy_id() on our first remove call even though
> another device is still remaining.
>
> If we increment "ib_ctx.ib_dev_count" every time add is called, even
> before we call rtrs_srv_rdma_init() and irrespective of whether
> rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> IB devices added. However, when remove is called, we now know the
> number of devices added, but not whether rtrs_srv_rdma_init() was
> successful or not. We may end up calling rdma_destroy_id() on NULL
> cm_ids
>
> Does this make sense or am I missing something?
>
>
> >
> > if (ib_ctx.ib_dev_count)
> >         return 0;
> >
> > ctx = ib_ctx.srv_ctx;
> > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > if (ret)
> >         return ret;
> > ib_ctx.ib_dev_count++;
> > return 0;

You missed the above two lines from my suggestion.

> >
> > > +     }
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > +{
> > > +     struct rtrs_srv_ctx *ctx;
> > > +
> > > +     ib_ctx.ib_dev_count--;

And this line too.

> > > +
> > > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
> >
> > It is not kernel coding style.
> > if (ib_ctx.ib_dev_count)
> >         return;
> >
> > ctx = ib_ctx.srv_ctx;
> > rdma_destroy_id(ctx->cm_id_ip);
> > rdma_destroy_id(ctx->cm_id_ib);
> >
> > Thanks
> >
> > > +             /*
> > > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > > +              * only once, when the last device is removed
> > > +              */
> > > +             ctx = ib_ctx.srv_ctx;
> > > +             rdma_destroy_id(ctx->cm_id_ip);
> > > +             rdma_destroy_id(ctx->cm_id_ib);
> > > +             ib_ctx.rdma_init = false;
> > > +     }
> > > +}
> > > +
> > > +static struct ib_client rtrs_srv_client = {
> > > +     .name   = "rtrs_server",
> > > +     .add    = rtrs_srv_add_one,
> > > +     .remove = rtrs_srv_remove_one
> > > +};
> > > +
> > >  /**
> > >   * rtrs_srv_open() - open RTRS server context
> > >   * @ops:             callback functions
> > > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > >       if (!ctx)
> > >               return ERR_PTR(-ENOMEM);
> > >
> > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > +             .srv_ctx        = ctx,
> > > +             .port           = port,
> > > +     };
> > > +
> > > +     err = ib_register_client(&rtrs_srv_client);
> > >       if (err) {
> > >               free_srv_ctx(ctx);
> > >               return ERR_PTR(err);
> > >       }
> > >
> > > +     /*
> > > +      * Since ib_register_client does not propagate the device add error
> > > +      * we check if the RDMA connection init was successful or not
> > > +      */
> > > +     if (!ib_ctx.rdma_init) {
> > > +             free_srv_ctx(ctx);
> > > +             return NULL;
> > > +     }
> > > +
> > >       return ctx;
> > >  }
> > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > >   */
> > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > >  {
> > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > +     ib_unregister_client(&rtrs_srv_client);
> > >       close_ctx(ctx);
> > >       free_srv_ctx(ctx);
> > >  }
> > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > index dc95b0932f0d..6e9d9000cd8d 100644
> > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> > >       struct list_head srv_list;
> > >  };
> > >
> > > +struct rtrs_srv_ib_ctx {
> > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > +     u16                     port;
> > > +     int                     ib_dev_count;
> > > +     bool                    rdma_init;
> > > +};
> > > +
> > >  extern struct class *rtrs_dev_class;
> > >
> > >  void close_sess(struct rtrs_srv_sess *sess);
> > > --
> > > 2.25.1
> > >
>
>
>
> --
>
> Regards
> -Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Wed, Aug 5, 2020 at 2:34 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> > On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > agent module. As such, when the kernel is configured to load the
> > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > dereference occurs during the RDMA bind operation.
> > > >
> > > > Call trace below,
> > > >
> > > > [    1.904782] Call Trace:
> > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > [    1.904782]  xa_load+0x47/0x80
> > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > [    1.904782]  kernel_init+0xe/0x100
> > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > [    1.904782] Modules linked in:
> > > > [    1.904782] CR2: 0000000000000015
> > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > >
> > > > All this happens cause the cm init is in the call chain of the module init,
> > > > which is not a preferred practice.
> > > >
> > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > rdma_create_id() is called only when an ib device is added.
> > > >
> > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > ---
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > >
> > > Please don't send vX patches as reply-to in "git send-email" command.
> > >
> > > >
> > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > index 0d9241f5d9e6..916f99464d09 100644
> > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > @@ -16,6 +16,7 @@
> > > >  #include "rtrs-srv.h"
> > > >  #include "rtrs-log.h"
> > > >  #include <rdma/ib_cm.h>
> > > > +#include <rdma/ib_verbs.h>
> > > >
> > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > >  MODULE_LICENSE("GPL");
> > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > >  static mempool_t *chunk_pool;
> > > >  struct class *rtrs_dev_class;
> > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > >
> > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > >       kfree(ctx);
> > > >  }
> > > >
> > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > +{
> > > > +     struct rtrs_srv_ctx *ctx;
> > > > +     int ret;
> > > > +
> > > > +     /*
> > > > +      * Keep a track on the number of ib devices added
> > > > +      */
> > > > +     ib_ctx.ib_dev_count++;
> > > > +
> > > > +     if (!ib_ctx.rdma_init) {
> > > > +             /*
> > > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > > +              * only once
> > > > +              */
> > > > +             ctx = ib_ctx.srv_ctx;
> > > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > +             if (ret) {
> > > > +                     /*
> > > > +                      * We errored out here.
> > > > +                      * According to the ib code, if we encounter an error here then the
> > > > +                      * error code is ignored, and no more calls to our ops are made.
> > > > +                      */
> > > > +                     pr_err("Failed to initialize RDMA connection");
> > > > +                     return ret;
> > > > +             }
> > > > +             ib_ctx.rdma_init = true;
> > >
> > > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > > logic can be simplified.
> >
> > Yes, this was the first logic in my head. But I have few thoughts,
> > The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> > successful execution of rtrs_srv_rdma_init() and not really an IB
> > device count. Meaning if we have multiple calls to add, due to
> > multiple devices, our count would stay 1. And while removal we might
> > end up calling rdma_destroy_id() on our first remove call even though
> > another device is still remaining.
> >
> > If we increment "ib_ctx.ib_dev_count" every time add is called, even
> > before we call rtrs_srv_rdma_init() and irrespective of whether
> > rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> > IB devices added. However, when remove is called, we now know the
> > number of devices added, but not whether rtrs_srv_rdma_init() was
> > successful or not. We may end up calling rdma_destroy_id() on NULL
> > cm_ids
> >
> > Does this make sense or am I missing something?
> >
> >
> > >
> > > if (ib_ctx.ib_dev_count)
> > >         return 0;

My understanding is, with the above 2 lines, after one add in which
rtrs_srv_rdma_init() succeeds, we won't even go below this, and hence
subsequent increments will not happen.

> > >
> > > ctx = ib_ctx.srv_ctx;
> > > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > if (ret)
> > >         return ret;

Also here, when rtrs_srv_rdma_init() fails, we return without
incrementing. IMHO, in this logic, we are not using
"ib_ctx.ib_dev_count" to track the number of devices, but to mark
successful execution of rtrs_srv_rdma_init()

> > > ib_ctx.ib_dev_count++;
> > > return 0;
>
> You missed the above two lines from my suggestion.
>
> > >
> > > > +     }
> > > > +
> > > > +     return 0;
> > > > +}
> > > > +
> > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > +{
> > > > +     struct rtrs_srv_ctx *ctx;
> > > > +
> > > > +     ib_ctx.ib_dev_count--;
>
> And this line too.
>
> > > > +
> > > > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
> > >
> > > It is not kernel coding style.
> > > if (ib_ctx.ib_dev_count)
> > >         return;
> > >
> > > ctx = ib_ctx.srv_ctx;
> > > rdma_destroy_id(ctx->cm_id_ip);
> > > rdma_destroy_id(ctx->cm_id_ib);
> > >
> > > Thanks
> > >
> > > > +             /*
> > > > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > +              * only once, when the last device is removed
> > > > +              */
> > > > +             ctx = ib_ctx.srv_ctx;
> > > > +             rdma_destroy_id(ctx->cm_id_ip);
> > > > +             rdma_destroy_id(ctx->cm_id_ib);
> > > > +             ib_ctx.rdma_init = false;
> > > > +     }
> > > > +}
> > > > +
> > > > +static struct ib_client rtrs_srv_client = {
> > > > +     .name   = "rtrs_server",
> > > > +     .add    = rtrs_srv_add_one,
> > > > +     .remove = rtrs_srv_remove_one
> > > > +};
> > > > +
> > > >  /**
> > > >   * rtrs_srv_open() - open RTRS server context
> > > >   * @ops:             callback functions
> > > > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > >       if (!ctx)
> > > >               return ERR_PTR(-ENOMEM);
> > > >
> > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > +             .srv_ctx        = ctx,
> > > > +             .port           = port,
> > > > +     };
> > > > +
> > > > +     err = ib_register_client(&rtrs_srv_client);
> > > >       if (err) {
> > > >               free_srv_ctx(ctx);
> > > >               return ERR_PTR(err);
> > > >       }
> > > >
> > > > +     /*
> > > > +      * Since ib_register_client does not propagate the device add error
> > > > +      * we check if the RDMA connection init was successful or not
> > > > +      */
> > > > +     if (!ib_ctx.rdma_init) {
> > > > +             free_srv_ctx(ctx);
> > > > +             return NULL;
> > > > +     }
> > > > +
> > > >       return ctx;
> > > >  }
> > > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > > >   */
> > > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > > >  {
> > > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > > +     ib_unregister_client(&rtrs_srv_client);
> > > >       close_ctx(ctx);
> > > >       free_srv_ctx(ctx);
> > > >  }
> > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > index dc95b0932f0d..6e9d9000cd8d 100644
> > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> > > >       struct list_head srv_list;
> > > >  };
> > > >
> > > > +struct rtrs_srv_ib_ctx {
> > > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > > +     u16                     port;
> > > > +     int                     ib_dev_count;
> > > > +     bool                    rdma_init;
> > > > +};
> > > > +
> > > >  extern struct class *rtrs_dev_class;
> > > >
> > > >  void close_sess(struct rtrs_srv_sess *sess);
> > > > --
> > > > 2.25.1
> > > >
> >
> >
> >
> > --
> >
> > Regards
> > -Haris



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Wed, Aug 05, 2020 at 04:39:16PM +0530, Haris Iqbal wrote:
> On Wed, Aug 5, 2020 at 2:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> > > On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > agent module. As such, when the kernel is configured to load the
> > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > dereference occurs during the RDMA bind operation.
> > > > >
> > > > > Call trace below,
> > > > >
> > > > > [    1.904782] Call Trace:
> > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > [    1.904782] Modules linked in:
> > > > > [    1.904782] CR2: 0000000000000015
> > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > >
> > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > which is not a preferred practice.
> > > > >
> > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > rdma_create_id() is called only when an ib device is added.
> > > > >
> > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > ---
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > > >
> > > > Please don't send vX patches as reply-to in "git send-email" command.
> > > >
> > > > >
> > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > index 0d9241f5d9e6..916f99464d09 100644
> > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > @@ -16,6 +16,7 @@
> > > > >  #include "rtrs-srv.h"
> > > > >  #include "rtrs-log.h"
> > > > >  #include <rdma/ib_cm.h>
> > > > > +#include <rdma/ib_verbs.h>
> > > > >
> > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > >  MODULE_LICENSE("GPL");
> > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > >  static mempool_t *chunk_pool;
> > > > >  struct class *rtrs_dev_class;
> > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > >
> > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > >       kfree(ctx);
> > > > >  }
> > > > >
> > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +     int ret;
> > > > > +
> > > > > +     /*
> > > > > +      * Keep a track on the number of ib devices added
> > > > > +      */
> > > > > +     ib_ctx.ib_dev_count++;
> > > > > +
> > > > > +     if (!ib_ctx.rdma_init) {
> > > > > +             /*
> > > > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > +              * only once
> > > > > +              */
> > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > +             if (ret) {
> > > > > +                     /*
> > > > > +                      * We errored out here.
> > > > > +                      * According to the ib code, if we encounter an error here then the
> > > > > +                      * error code is ignored, and no more calls to our ops are made.
> > > > > +                      */
> > > > > +                     pr_err("Failed to initialize RDMA connection");
> > > > > +                     return ret;
> > > > > +             }
> > > > > +             ib_ctx.rdma_init = true;
> > > >
> > > > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > > > logic can be simplified.
> > >
> > > Yes, this was the first logic in my head. But I have few thoughts,
> > > The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> > > successful execution of rtrs_srv_rdma_init() and not really an IB
> > > device count. Meaning if we have multiple calls to add, due to
> > > multiple devices, our count would stay 1. And while removal we might
> > > end up calling rdma_destroy_id() on our first remove call even though
> > > another device is still remaining.
> > >
> > > If we increment "ib_ctx.ib_dev_count" every time add is called, even
> > > before we call rtrs_srv_rdma_init() and irrespective of whether
> > > rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> > > IB devices added. However, when remove is called, we now know the
> > > number of devices added, but not whether rtrs_srv_rdma_init() was
> > > successful or not. We may end up calling rdma_destroy_id() on NULL
> > > cm_ids
> > >
> > > Does this make sense or am I missing something?
> > >
> > >
> > > >
> > > > if (ib_ctx.ib_dev_count)
> > > >         return 0;
>
> My understanding is, with the above 2 lines, after one add in which
> rtrs_srv_rdma_init() succeeds, we won't even go below this, and hence
> subsequent increments will not happen.

Is it better?

if (ib_ctx.ib_dev_count)
  goto out;

....

out:
  ib_ctx.ib_dev_count++;
  return 0;

You don't need to take the code proposed in the ML as is.

>
> > > >
> > > > ctx = ib_ctx.srv_ctx;
> > > > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > if (ret)
> > > >         return ret;
>
> Also here, when rtrs_srv_rdma_init() fails, we return without
> incrementing. IMHO, in this logic, we are not using
> "ib_ctx.ib_dev_count" to track the number of devices, but to mark
> successful execution of rtrs_srv_rdma_init()

Of course, you should increment in success only.

>
> > > > ib_ctx.ib_dev_count++;
> > > > return 0;
> >
> > You missed the above two lines from my suggestion.
> >
> > > >
> > > > > +     }
> > > > > +
> > > > > +     return 0;
> > > > > +}
> > > > > +
> > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +
> > > > > +     ib_ctx.ib_dev_count--;
> >
> > And this line too.
> >
> > > > > +
> > > > > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
> > > >
> > > > It is not kernel coding style.
> > > > if (ib_ctx.ib_dev_count)
> > > >         return;
> > > >
> > > > ctx = ib_ctx.srv_ctx;
> > > > rdma_destroy_id(ctx->cm_id_ip);
> > > > rdma_destroy_id(ctx->cm_id_ib);
> > > >
> > > > Thanks
> > > >
> > > > > +             /*
> > > > > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > +              * only once, when the last device is removed
> > > > > +              */
> > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > +             rdma_destroy_id(ctx->cm_id_ip);
> > > > > +             rdma_destroy_id(ctx->cm_id_ib);
> > > > > +             ib_ctx.rdma_init = false;
> > > > > +     }
> > > > > +}
> > > > > +
> > > > > +static struct ib_client rtrs_srv_client = {
> > > > > +     .name   = "rtrs_server",
> > > > > +     .add    = rtrs_srv_add_one,
> > > > > +     .remove = rtrs_srv_remove_one
> > > > > +};
> > > > > +
> > > > >  /**
> > > > >   * rtrs_srv_open() - open RTRS server context
> > > > >   * @ops:             callback functions
> > > > > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > >       if (!ctx)
> > > > >               return ERR_PTR(-ENOMEM);
> > > > >
> > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > +             .srv_ctx        = ctx,
> > > > > +             .port           = port,
> > > > > +     };
> > > > > +
> > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > >       if (err) {
> > > > >               free_srv_ctx(ctx);
> > > > >               return ERR_PTR(err);
> > > > >       }
> > > > >
> > > > > +     /*
> > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > +      * we check if the RDMA connection init was successful or not
> > > > > +      */
> > > > > +     if (!ib_ctx.rdma_init) {
> > > > > +             free_srv_ctx(ctx);
> > > > > +             return NULL;
> > > > > +     }
> > > > > +
> > > > >       return ctx;
> > > > >  }
> > > > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > > > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > > > >   */
> > > > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > > > >  {
> > > > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > > > +     ib_unregister_client(&rtrs_srv_client);
> > > > >       close_ctx(ctx);
> > > > >       free_srv_ctx(ctx);
> > > > >  }
> > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > index dc95b0932f0d..6e9d9000cd8d 100644
> > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> > > > >       struct list_head srv_list;
> > > > >  };
> > > > >
> > > > > +struct rtrs_srv_ib_ctx {
> > > > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > > > +     u16                     port;
> > > > > +     int                     ib_dev_count;
> > > > > +     bool                    rdma_init;
> > > > > +};
> > > > > +
> > > > >  extern struct class *rtrs_dev_class;
> > > > >
> > > > >  void close_sess(struct rtrs_srv_sess *sess);
> > > > > --
> > > > > 2.25.1
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards
> > > -Haris
>
>
>
> --
>
> Regards
> -Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Wed, Aug 5, 2020 at 6:42 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Aug 05, 2020 at 04:39:16PM +0530, Haris Iqbal wrote:
> > On Wed, Aug 5, 2020 at 2:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> > > > On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > dereference occurs during the RDMA bind operation.
> > > > > >
> > > > > > Call trace below,
> > > > > >
> > > > > > [    1.904782] Call Trace:
> > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > [    1.904782] Modules linked in:
> > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > >
> > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > which is not a preferred practice.
> > > > > >
> > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > >
> > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > ---
> > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > > > >
> > > > > Please don't send vX patches as reply-to in "git send-email" command.
> > > > >
> > > > > >
> > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > index 0d9241f5d9e6..916f99464d09 100644
> > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > >  #include "rtrs-srv.h"
> > > > > >  #include "rtrs-log.h"
> > > > > >  #include <rdma/ib_cm.h>
> > > > > > +#include <rdma/ib_verbs.h>
> > > > > >
> > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > >  MODULE_LICENSE("GPL");
> > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > >  static mempool_t *chunk_pool;
> > > > > >  struct class *rtrs_dev_class;
> > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > >
> > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > >       kfree(ctx);
> > > > > >  }
> > > > > >
> > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > +{
> > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > +     int ret;
> > > > > > +
> > > > > > +     /*
> > > > > > +      * Keep a track on the number of ib devices added
> > > > > > +      */
> > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > +
> > > > > > +     if (!ib_ctx.rdma_init) {
> > > > > > +             /*
> > > > > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > +              * only once
> > > > > > +              */
> > > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > +             if (ret) {
> > > > > > +                     /*
> > > > > > +                      * We errored out here.
> > > > > > +                      * According to the ib code, if we encounter an error here then the
> > > > > > +                      * error code is ignored, and no more calls to our ops are made.
> > > > > > +                      */
> > > > > > +                     pr_err("Failed to initialize RDMA connection");
> > > > > > +                     return ret;
> > > > > > +             }
> > > > > > +             ib_ctx.rdma_init = true;
> > > > >
> > > > > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > > > > logic can be simplified.
> > > >
> > > > Yes, this was the first logic in my head. But I have few thoughts,
> > > > The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> > > > successful execution of rtrs_srv_rdma_init() and not really an IB
> > > > device count. Meaning if we have multiple calls to add, due to
> > > > multiple devices, our count would stay 1. And while removal we might
> > > > end up calling rdma_destroy_id() on our first remove call even though
> > > > another device is still remaining.
> > > >
> > > > If we increment "ib_ctx.ib_dev_count" every time add is called, even
> > > > before we call rtrs_srv_rdma_init() and irrespective of whether
> > > > rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> > > > IB devices added. However, when remove is called, we now know the
> > > > number of devices added, but not whether rtrs_srv_rdma_init() was
> > > > successful or not. We may end up calling rdma_destroy_id() on NULL
> > > > cm_ids
> > > >
> > > > Does this make sense or am I missing something?
> > > >
> > > >
> > > > >
> > > > > if (ib_ctx.ib_dev_count)
> > > > >         return 0;
> >
> > My understanding is, with the above 2 lines, after one add in which
> > rtrs_srv_rdma_init() succeeds, we won't even go below this, and hence
> > subsequent increments will not happen.
>
> Is it better?
>
> if (ib_ctx.ib_dev_count)
>   goto out;
>
> ....
>
> out:
>   ib_ctx.ib_dev_count++;
>   return 0;
>
> You don't need to take the code proposed in the ML as is.

Yes, hence I posted a theoretical scenario which discussed 2 possible scenarios.

case 1, single variable tracking only number of devices added.
when remove is called, we now know the
number of devices added, but not whether rtrs_srv_rdma_init() was
successful or not. We may end up calling rdma_destroy_id() on NULL
cm_ids

case 2, single variable tracking success of rtrs_srv_rdma_init()
If we have multiple IB devices added, while removal we won't know when
to call rdma_destroy_id(),

>
> >
> > > > >
> > > > > ctx = ib_ctx.srv_ctx;
> > > > > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > if (ret)
> > > > >         return ret;
> >
> > Also here, when rtrs_srv_rdma_init() fails, we return without
> > incrementing. IMHO, in this logic, we are not using
> > "ib_ctx.ib_dev_count" to track the number of devices, but to mark
> > successful execution of rtrs_srv_rdma_init()
>
> Of course, you should increment in success only.

I am confused again. What exactly are you suggesting we track with
"ib_ctx.ib_dev_count"?
According to my understanding, we can't possibly track both "number of
devices added" and "success of rtrs_srv_rdma_init()"

>
> >
> > > > > ib_ctx.ib_dev_count++;
> > > > > return 0;
> > >
> > > You missed the above two lines from my suggestion.
> > >
> > > > >
> > > > > > +     }
> > > > > > +
> > > > > > +     return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > +{
> > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > +
> > > > > > +     ib_ctx.ib_dev_count--;
> > >
> > > And this line too.
> > >
> > > > > > +
> > > > > > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
> > > > >
> > > > > It is not kernel coding style.
> > > > > if (ib_ctx.ib_dev_count)
> > > > >         return;
> > > > >
> > > > > ctx = ib_ctx.srv_ctx;
> > > > > rdma_destroy_id(ctx->cm_id_ip);
> > > > > rdma_destroy_id(ctx->cm_id_ib);
> > > > >
> > > > > Thanks
> > > > >
> > > > > > +             /*
> > > > > > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > +              * only once, when the last device is removed
> > > > > > +              */
> > > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > > +             rdma_destroy_id(ctx->cm_id_ip);
> > > > > > +             rdma_destroy_id(ctx->cm_id_ib);
> > > > > > +             ib_ctx.rdma_init = false;
> > > > > > +     }
> > > > > > +}
> > > > > > +
> > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > +     .name   = "rtrs_server",
> > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > +};
> > > > > > +
> > > > > >  /**
> > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > >   * @ops:             callback functions
> > > > > > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > >       if (!ctx)
> > > > > >               return ERR_PTR(-ENOMEM);
> > > > > >
> > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > +             .srv_ctx        = ctx,
> > > > > > +             .port           = port,
> > > > > > +     };
> > > > > > +
> > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > >       if (err) {
> > > > > >               free_srv_ctx(ctx);
> > > > > >               return ERR_PTR(err);
> > > > > >       }
> > > > > >
> > > > > > +     /*
> > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > +      * we check if the RDMA connection init was successful or not
> > > > > > +      */
> > > > > > +     if (!ib_ctx.rdma_init) {
> > > > > > +             free_srv_ctx(ctx);
> > > > > > +             return NULL;
> > > > > > +     }
> > > > > > +
> > > > > >       return ctx;
> > > > > >  }
> > > > > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > > > > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > > > > >   */
> > > > > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > > > > >  {
> > > > > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > +     ib_unregister_client(&rtrs_srv_client);
> > > > > >       close_ctx(ctx);
> > > > > >       free_srv_ctx(ctx);
> > > > > >  }
> > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > > index dc95b0932f0d..6e9d9000cd8d 100644
> > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> > > > > >       struct list_head srv_list;
> > > > > >  };
> > > > > >
> > > > > > +struct rtrs_srv_ib_ctx {
> > > > > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > > > > +     u16                     port;
> > > > > > +     int                     ib_dev_count;
> > > > > > +     bool                    rdma_init;
> > > > > > +};
> > > > > > +
> > > > > >  extern struct class *rtrs_dev_class;
> > > > > >
> > > > > >  void close_sess(struct rtrs_srv_sess *sess);
> > > > > > --
> > > > > > 2.25.1
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards
> > > > -Haris
> >
> >
> >
> > --
> >
> > Regards
> > -Haris



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Wed, Aug 05, 2020 at 07:23:58PM +0530, Haris Iqbal wrote:
> On Wed, Aug 5, 2020 at 6:42 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Wed, Aug 05, 2020 at 04:39:16PM +0530, Haris Iqbal wrote:
> > > On Wed, Aug 5, 2020 at 2:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> > > > > On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > >
> > > > > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > >
> > > > > > > Call trace below,
> > > > > > >
> > > > > > > [    1.904782] Call Trace:
> > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > [    1.904782] Modules linked in:
> > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > >
> > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > which is not a preferred practice.
> > > > > > >
> > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > >
> > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > ---
> > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > > > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > Please don't send vX patches as reply-to in "git send-email" command.
> > > > > >
> > > > > > >
> > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > index 0d9241f5d9e6..916f99464d09 100644
> > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > @@ -16,6 +16,7 @@
> > > > > > >  #include "rtrs-srv.h"
> > > > > > >  #include "rtrs-log.h"
> > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > >
> > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > >  static mempool_t *chunk_pool;
> > > > > > >  struct class *rtrs_dev_class;
> > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > >
> > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > >       kfree(ctx);
> > > > > > >  }
> > > > > > >
> > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > +{
> > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > +     int ret;
> > > > > > > +
> > > > > > > +     /*
> > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > +      */
> > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > +
> > > > > > > +     if (!ib_ctx.rdma_init) {
> > > > > > > +             /*
> > > > > > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > +              * only once
> > > > > > > +              */
> > > > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > +             if (ret) {
> > > > > > > +                     /*
> > > > > > > +                      * We errored out here.
> > > > > > > +                      * According to the ib code, if we encounter an error here then the
> > > > > > > +                      * error code is ignored, and no more calls to our ops are made.
> > > > > > > +                      */
> > > > > > > +                     pr_err("Failed to initialize RDMA connection");
> > > > > > > +                     return ret;
> > > > > > > +             }
> > > > > > > +             ib_ctx.rdma_init = true;
> > > > > >
> > > > > > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > > > > > logic can be simplified.
> > > > >
> > > > > Yes, this was the first logic in my head. But I have few thoughts,
> > > > > The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> > > > > successful execution of rtrs_srv_rdma_init() and not really an IB
> > > > > device count. Meaning if we have multiple calls to add, due to
> > > > > multiple devices, our count would stay 1. And while removal we might
> > > > > end up calling rdma_destroy_id() on our first remove call even though
> > > > > another device is still remaining.
> > > > >
> > > > > If we increment "ib_ctx.ib_dev_count" every time add is called, even
> > > > > before we call rtrs_srv_rdma_init() and irrespective of whether
> > > > > rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> > > > > IB devices added. However, when remove is called, we now know the
> > > > > number of devices added, but not whether rtrs_srv_rdma_init() was
> > > > > successful or not. We may end up calling rdma_destroy_id() on NULL
> > > > > cm_ids
> > > > >
> > > > > Does this make sense or am I missing something?
> > > > >
> > > > >
> > > > > >
> > > > > > if (ib_ctx.ib_dev_count)
> > > > > >         return 0;
> > >
> > > My understanding is, with the above 2 lines, after one add in which
> > > rtrs_srv_rdma_init() succeeds, we won't even go below this, and hence
> > > subsequent increments will not happen.
> >
> > Is it better?
> >
> > if (ib_ctx.ib_dev_count)
> >   goto out;
> >
> > ....
> >
> > out:
> >   ib_ctx.ib_dev_count++;
> >   return 0;
> >
> > You don't need to take the code proposed in the ML as is.
>
> Yes, hence I posted a theoretical scenario which discussed 2 possible scenarios.
>
> case 1, single variable tracking only number of devices added.
> when remove is called, we now know the
> number of devices added, but not whether rtrs_srv_rdma_init() was
> successful or not. We may end up calling rdma_destroy_id() on NULL
> cm_ids

If rtrs_srv_rdma_init() fails to initialize on first attempt, why do you
want to continue to load rtrs?

>
> case 2, single variable tracking success of rtrs_srv_rdma_init()
> If we have multiple IB devices added, while removal we won't know when
> to call rdma_destroy_id(),

Your code doesn't know it either, it calls to destroy on last ib_dev.

>
> >
> > >
> > > > > >
> > > > > > ctx = ib_ctx.srv_ctx;
> > > > > > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > if (ret)
> > > > > >         return ret;
> > >
> > > Also here, when rtrs_srv_rdma_init() fails, we return without
> > > incrementing. IMHO, in this logic, we are not using
> > > "ib_ctx.ib_dev_count" to track the number of devices, but to mark
> > > successful execution of rtrs_srv_rdma_init()
> >
> > Of course, you should increment in success only.
>
> I am confused again. What exactly are you suggesting we track with
> "ib_ctx.ib_dev_count"?
> According to my understanding, we can't possibly track both "number of
> devices added" and "success of rtrs_srv_rdma_init()".

Why don't you ask your colleagues? They will guide you.

Thanks

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Wed, Aug 5, 2020 at 8:25 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Aug 05, 2020 at 07:23:58PM +0530, Haris Iqbal wrote:
> > On Wed, Aug 5, 2020 at 6:42 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Wed, Aug 05, 2020 at 04:39:16PM +0530, Haris Iqbal wrote:
> > > > On Wed, Aug 5, 2020 at 2:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Wed, Aug 05, 2020 at 01:20:09PM +0530, Haris Iqbal wrote:
> > > > > > On Wed, Aug 5, 2020 at 11:27 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > >
> > > > > > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > >
> > > > > > > > Call trace below,
> > > > > > > >
> > > > > > > > [    1.904782] Call Trace:
> > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > >
> > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > which is not a preferred practice.
> > > > > > > >
> > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > >
> > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > ---
> > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > > > > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > > > > > >
> > > > > > > Please don't send vX patches as reply-to in "git send-email" command.
> > > > > > >
> > > > > > > >
> > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > index 0d9241f5d9e6..916f99464d09 100644
> > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > >  #include "rtrs-srv.h"
> > > > > > > >  #include "rtrs-log.h"
> > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > >
> > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > >
> > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > >       kfree(ctx);
> > > > > > > >  }
> > > > > > > >
> > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > +{
> > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > +     int ret;
> > > > > > > > +
> > > > > > > > +     /*
> > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > +      */
> > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > +
> > > > > > > > +     if (!ib_ctx.rdma_init) {
> > > > > > > > +             /*
> > > > > > > > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > +              * only once
> > > > > > > > +              */
> > > > > > > > +             ctx = ib_ctx.srv_ctx;
> > > > > > > > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > +             if (ret) {
> > > > > > > > +                     /*
> > > > > > > > +                      * We errored out here.
> > > > > > > > +                      * According to the ib code, if we encounter an error here then the
> > > > > > > > +                      * error code is ignored, and no more calls to our ops are made.
> > > > > > > > +                      */
> > > > > > > > +                     pr_err("Failed to initialize RDMA connection");
> > > > > > > > +                     return ret;
> > > > > > > > +             }
> > > > > > > > +             ib_ctx.rdma_init = true;
> > > > > > >
> > > > > > > This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> > > > > > > logic can be simplified.
> > > > > >
> > > > > > Yes, this was the first logic in my head. But I have few thoughts,
> > > > > > The below suggestions uses "ib_ctx.ib_dev_count" as a marker for
> > > > > > successful execution of rtrs_srv_rdma_init() and not really an IB
> > > > > > device count. Meaning if we have multiple calls to add, due to
> > > > > > multiple devices, our count would stay 1. And while removal we might
> > > > > > end up calling rdma_destroy_id() on our first remove call even though
> > > > > > another device is still remaining.
> > > > > >
> > > > > > If we increment "ib_ctx.ib_dev_count" every time add is called, even
> > > > > > before we call rtrs_srv_rdma_init() and irrespective of whether
> > > > > > rtrs_srv_rdma_init() succeeds or not, then we are keeping a count of
> > > > > > IB devices added. However, when remove is called, we now know the
> > > > > > number of devices added, but not whether rtrs_srv_rdma_init() was
> > > > > > successful or not. We may end up calling rdma_destroy_id() on NULL
> > > > > > cm_ids
> > > > > >
> > > > > > Does this make sense or am I missing something?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > if (ib_ctx.ib_dev_count)
> > > > > > >         return 0;
> > > >
> > > > My understanding is, with the above 2 lines, after one add in which
> > > > rtrs_srv_rdma_init() succeeds, we won't even go below this, and hence
> > > > subsequent increments will not happen.
> > >
> > > Is it better?
> > >
> > > if (ib_ctx.ib_dev_count)
> > >   goto out;
> > >
> > > ....
> > >
> > > out:
> > >   ib_ctx.ib_dev_count++;
> > >   return 0;
> > >
> > > You don't need to take the code proposed in the ML as is.
> >
> > Yes, hence I posted a theoretical scenario which discussed 2 possible scenarios.
> >
> > case 1, single variable tracking only number of devices added.
> > when remove is called, we now know the
> > number of devices added, but not whether rtrs_srv_rdma_init() was
> > successful or not. We may end up calling rdma_destroy_id() on NULL
> > cm_ids
>
> If rtrs_srv_rdma_init() fails to initialize on first attempt, why do you
> want to continue to load rtrs?

This does make sense; for the current design in which
add_client_context() would stop calling add on this client if the
first add fails.
The reliance on this is what concerned me to add another explicit
variable for rdma_init.

Also, I am unsure how simple the logic becomes in this case (see
below) which was the aim of this discussion in the first place.

>
> >
> > case 2, single variable tracking success of rtrs_srv_rdma_init()
> > If we have multiple IB devices added, while removal we won't know when
> > to call rdma_destroy_id(),
>
> Your code doesn't know it either, it calls to destroy on last ib_dev.

It calls destroy on the last ib_dev only if the rdma_init is true,
meaning rtrs_srv_rdma_init() had succeeded.

+       ib_ctx.ib_dev_count--;
+
+       if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {

>
> >
> > >
> > > >
> > > > > > >
> > > > > > > ctx = ib_ctx.srv_ctx;
> > > > > > > ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > if (ret)
> > > > > > >         return ret;
> > > >
> > > > Also here, when rtrs_srv_rdma_init() fails, we return without
> > > > incrementing. IMHO, in this logic, we are not using
> > > > "ib_ctx.ib_dev_count" to track the number of devices, but to mark
> > > > successful execution of rtrs_srv_rdma_init()
> > >
> > > Of course, you should increment in success only.
> >
> > I am confused again. What exactly are you suggesting we track with
> > "ib_ctx.ib_dev_count"?
> > According to my understanding, we can't possibly track both "number of
> > devices added" and "success of rtrs_srv_rdma_init()".
>
> Why don't you ask your colleagues? They will guide you.

(from above)
"ib_ctx.ib_dev_count" is basically tracking the number of IB devices
added, IF the first call to add succeeds.


>
> Thanks



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Danil Kipnis]

Hi Leon,

On Wed, Aug 5, 2020 at 7:57 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > The rnbd_server module's communication manager (cm) initialization depends
> > on the registration of the "network namespace subsystem" of the RDMA CM
> > agent module. As such, when the kernel is configured to load the
> > rnbd_server and the RDMA cma module during initialization; and if the
> > rnbd_server module is initialized before RDMA cma module, a null ptr
> > dereference occurs during the RDMA bind operation.
> >
> > Call trace below,
> >
> > [    1.904782] Call Trace:
> > [    1.904782]  ? xas_load+0xd/0x80
> > [    1.904782]  xa_load+0x47/0x80
> > [    1.904782]  cma_ps_find+0x44/0x70
> > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > [    1.904782]  ? get_random_bytes+0x35/0x40
> > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > [    1.904782]  rtrs_srv_open+0x102/0x180
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  do_one_initcall+0x4a/0x200
> > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > [    1.904782]  ? rest_init+0xb0/0xb0
> > [    1.904782]  kernel_init+0xe/0x100
> > [    1.904782]  ret_from_fork+0x22/0x30
> > [    1.904782] Modules linked in:
> > [    1.904782] CR2: 0000000000000015
> > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> >
> > All this happens cause the cm init is in the call chain of the module init,
> > which is not a preferred practice.
> >
> > So remove the call to rdma_create_id() from the module init call chain.
> > Instead register rtrs-srv as an ib client, which makes sure that the
> > rdma_create_id() is called only when an ib device is added.
> >
> > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > ---
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> >  2 files changed, 81 insertions(+), 3 deletions(-)
>
> Please don't send vX patches as reply-to in "git send-email" command.

I thought vX + in-reply-to makes it clear that a new version of a
patch is proposed in response to a mail reporting a problem in the
first version. Why is that a bad idea?

Thank you,
Best,
Danil.

>
> >
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > index 0d9241f5d9e6..916f99464d09 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > @@ -16,6 +16,7 @@
> >  #include "rtrs-srv.h"
> >  #include "rtrs-log.h"
> >  #include <rdma/ib_cm.h>
> > +#include <rdma/ib_verbs.h>
> >
> >  MODULE_DESCRIPTION("RDMA Transport Server");
> >  MODULE_LICENSE("GPL");
> > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> >  static struct rtrs_rdma_dev_pd dev_pd;
> >  static mempool_t *chunk_pool;
> >  struct class *rtrs_dev_class;
> > +static struct rtrs_srv_ib_ctx ib_ctx;
> >
> >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > @@ -2033,6 +2035,62 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> >       kfree(ctx);
> >  }
> >
> > +static int rtrs_srv_add_one(struct ib_device *device)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +     int ret;
> > +
> > +     /*
> > +      * Keep a track on the number of ib devices added
> > +      */
> > +     ib_ctx.ib_dev_count++;
> > +
> > +     if (!ib_ctx.rdma_init) {
> > +             /*
> > +              * Since our CM IDs are NOT bound to any ib device we will create them
> > +              * only once
> > +              */
> > +             ctx = ib_ctx.srv_ctx;
> > +             ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > +             if (ret) {
> > +                     /*
> > +                      * We errored out here.
> > +                      * According to the ib code, if we encounter an error here then the
> > +                      * error code is ignored, and no more calls to our ops are made.
> > +                      */
> > +                     pr_err("Failed to initialize RDMA connection");
> > +                     return ret;
> > +             }
> > +             ib_ctx.rdma_init = true;
>
> This rdma_init == false is equal to ib_ctx.ib_dev_count == 0 and the
> logic can be simplified.
>
> if (ib_ctx.ib_dev_count)
>         return 0;
>
> ctx = ib_ctx.srv_ctx;
> ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> if (ret)
>         return ret;
> ib_ctx.ib_dev_count++;
> return 0;
>
> > +     }
> > +
> > +     return 0;
> > +}
> > +
> > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +
> > +     ib_ctx.ib_dev_count--;
> > +
> > +     if (!ib_ctx.ib_dev_count && ib_ctx.rdma_init) {
>
> It is not kernel coding style.
> if (ib_ctx.ib_dev_count)
>         return;
>
> ctx = ib_ctx.srv_ctx;
> rdma_destroy_id(ctx->cm_id_ip);
> rdma_destroy_id(ctx->cm_id_ib);
>
> Thanks
>
> > +             /*
> > +              * Since our CM IDs are NOT bound to any ib device we will remove them
> > +              * only once, when the last device is removed
> > +              */
> > +             ctx = ib_ctx.srv_ctx;
> > +             rdma_destroy_id(ctx->cm_id_ip);
> > +             rdma_destroy_id(ctx->cm_id_ib);
> > +             ib_ctx.rdma_init = false;
> > +     }
> > +}
> > +
> > +static struct ib_client rtrs_srv_client = {
> > +     .name   = "rtrs_server",
> > +     .add    = rtrs_srv_add_one,
> > +     .remove = rtrs_srv_remove_one
> > +};
> > +
> >  /**
> >   * rtrs_srv_open() - open RTRS server context
> >   * @ops:             callback functions
> > @@ -2051,12 +2109,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> >       if (!ctx)
> >               return ERR_PTR(-ENOMEM);
> >
> > -     err = rtrs_srv_rdma_init(ctx, port);
> > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > +             .srv_ctx        = ctx,
> > +             .port           = port,
> > +     };
> > +
> > +     err = ib_register_client(&rtrs_srv_client);
> >       if (err) {
> >               free_srv_ctx(ctx);
> >               return ERR_PTR(err);
> >       }
> >
> > +     /*
> > +      * Since ib_register_client does not propagate the device add error
> > +      * we check if the RDMA connection init was successful or not
> > +      */
> > +     if (!ib_ctx.rdma_init) {
> > +             free_srv_ctx(ctx);
> > +             return NULL;
> > +     }
> > +
> >       return ctx;
> >  }
> >  EXPORT_SYMBOL(rtrs_srv_open);
> > @@ -2090,8 +2162,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> >   */
> >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> >  {
> > -     rdma_destroy_id(ctx->cm_id_ip);
> > -     rdma_destroy_id(ctx->cm_id_ib);
> > +     ib_unregister_client(&rtrs_srv_client);
> >       close_ctx(ctx);
> >       free_srv_ctx(ctx);
> >  }
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > index dc95b0932f0d..6e9d9000cd8d 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > @@ -118,6 +118,13 @@ struct rtrs_srv_ctx {
> >       struct list_head srv_list;
> >  };
> >
> > +struct rtrs_srv_ib_ctx {
> > +     struct rtrs_srv_ctx     *srv_ctx;
> > +     u16                     port;
> > +     int                     ib_dev_count;
> > +     bool                    rdma_init;
> > +};
> > +
> >  extern struct class *rtrs_dev_class;
> >
> >  void close_sess(struct rtrs_srv_sess *sess);
> > --
> > 2.25.1
> >

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2908 bytes --]

On Wed, Aug 05, 2020 at 11:09:00AM +0200, Danil Kipnis wrote:
> Hi Leon,
>
> On Wed, Aug 5, 2020 at 7:57 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > The rnbd_server module's communication manager (cm) initialization depends
> > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > agent module. As such, when the kernel is configured to load the
> > > rnbd_server and the RDMA cma module during initialization; and if the
> > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > dereference occurs during the RDMA bind operation.
> > >
> > > Call trace below,
> > >
> > > [    1.904782] Call Trace:
> > > [    1.904782]  ? xas_load+0xd/0x80
> > > [    1.904782]  xa_load+0x47/0x80
> > > [    1.904782]  cma_ps_find+0x44/0x70
> > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > [    1.904782]  kernel_init+0xe/0x100
> > > [    1.904782]  ret_from_fork+0x22/0x30
> > > [    1.904782] Modules linked in:
> > > [    1.904782] CR2: 0000000000000015
> > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > >
> > > All this happens cause the cm init is in the call chain of the module init,
> > > which is not a preferred practice.
> > >
> > > So remove the call to rdma_create_id() from the module init call chain.
> > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > rdma_create_id() is called only when an ib device is added.
> > >
> > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > ---
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > >  2 files changed, 81 insertions(+), 3 deletions(-)
> >
> > Please don't send vX patches as reply-to in "git send-email" command.
>
> I thought vX + in-reply-to makes it clear that a new version of a
> patch is proposed in response to a mail reporting a problem in the
> first version. Why is that a bad idea?

It looks very messy in e-mail client. It is hard to follow and requires
multiple iterations to understand if the reply is on previous variant or
on new one.

See attached print screen or see it in lore, where thread view is used.
https://lore.kernel.org/linux-rdma/20200623172321.GC6578@ziepe.ca/T/#t

Thanks

[-- Attachment #2: mutt-view.png --]
[-- Type: image/png, Size: 660634 bytes --]

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Danil Kipnis]

On Wed, Aug 5, 2020 at 11:16 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Aug 05, 2020 at 11:09:00AM +0200, Danil Kipnis wrote:
> > Hi Leon,
> >
> > On Wed, Aug 5, 2020 at 7:57 AM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > agent module. As such, when the kernel is configured to load the
> > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > dereference occurs during the RDMA bind operation.
> > > >
> > > > Call trace below,
> > > >
> > > > [    1.904782] Call Trace:
> > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > [    1.904782]  xa_load+0x47/0x80
> > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > [    1.904782]  kernel_init+0xe/0x100
> > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > [    1.904782] Modules linked in:
> > > > [    1.904782] CR2: 0000000000000015
> > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > >
> > > > All this happens cause the cm init is in the call chain of the module init,
> > > > which is not a preferred practice.
> > > >
> > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > rdma_create_id() is called only when an ib device is added.
> > > >
> > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > ---
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > >
> > > Please don't send vX patches as reply-to in "git send-email" command.
> >
> > I thought vX + in-reply-to makes it clear that a new version of a
> > patch is proposed in response to a mail reporting a problem in the
> > first version. Why is that a bad idea?
>
> It looks very messy in e-mail client. It is hard to follow and requires
> multiple iterations to understand if the reply is on previous variant or
> on new one.
>
> See attached print screen or see it in lore, where thread view is used.
> https://lore.kernel.org/linux-rdma/20200623172321.GC6578@ziepe.ca/T/#t

I see, thanks. Just a related question: In this particular case the
commit message changed in the second version of the patch. Should one
still use v2 in that case, or should the second version be submitted
without the vX tag at all?

>
> Thanks

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Wed, Aug 05, 2020 at 01:18:05PM +0200, Danil Kipnis wrote:
> On Wed, Aug 5, 2020 at 11:16 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Wed, Aug 05, 2020 at 11:09:00AM +0200, Danil Kipnis wrote:
> > > Hi Leon,
> > >
> > > On Wed, Aug 5, 2020 at 7:57 AM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Tue, Aug 04, 2020 at 07:07:58PM +0530, Md Haris Iqbal wrote:
> > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > agent module. As such, when the kernel is configured to load the
> > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > dereference occurs during the RDMA bind operation.
> > > > >
> > > > > Call trace below,
> > > > >
> > > > > [    1.904782] Call Trace:
> > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > [    1.904782] Modules linked in:
> > > > > [    1.904782] CR2: 0000000000000015
> > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > >
> > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > which is not a preferred practice.
> > > > >
> > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > rdma_create_id() is called only when an ib device is added.
> > > > >
> > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > ---
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 77 +++++++++++++++++++++++++-
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  7 +++
> > > > >  2 files changed, 81 insertions(+), 3 deletions(-)
> > > >
> > > > Please don't send vX patches as reply-to in "git send-email" command.
> > >
> > > I thought vX + in-reply-to makes it clear that a new version of a
> > > patch is proposed in response to a mail reporting a problem in the
> > > first version. Why is that a bad idea?
> >
> > It looks very messy in e-mail client. It is hard to follow and requires
> > multiple iterations to understand if the reply is on previous variant or
> > on new one.
> >
> > See attached print screen or see it in lore, where thread view is used.
> > https://lore.kernel.org/linux-rdma/20200623172321.GC6578@ziepe.ca/T/#t
>
> I see, thanks. Just a related question: In this particular case the
> commit message changed in the second version of the patch. Should one
> still use v2 in that case, or should the second version be submitted
> without the vX tag at all?

It depends on the change itself. If title wasn't changed, the vX would
help to distinguish between versions. Once title changes and the commit
message too, the vX is not really needed (IMHO).

Thanks

>
> >
> > Thanks

[PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Md Haris Iqbal]

The rnbd_server module's communication manager (cm) initialization depends
on the registration of the "network namespace subsystem" of the RDMA CM
agent module. As such, when the kernel is configured to load the
rnbd_server and the RDMA cma module during initialization; and if the
rnbd_server module is initialized before RDMA cma module, a null ptr
dereference occurs during the RDMA bind operation.

Call trace below,

[    1.904782] Call Trace:
[    1.904782]  ? xas_load+0xd/0x80
[    1.904782]  xa_load+0x47/0x80
[    1.904782]  cma_ps_find+0x44/0x70
[    1.904782]  rdma_bind_addr+0x782/0x8b0
[    1.904782]  ? get_random_bytes+0x35/0x40
[    1.904782]  rtrs_srv_cm_init+0x50/0x80
[    1.904782]  rtrs_srv_open+0x102/0x180
[    1.904782]  ? rnbd_client_init+0x6e/0x6e
[    1.904782]  rnbd_srv_init_module+0x34/0x84
[    1.904782]  ? rnbd_client_init+0x6e/0x6e
[    1.904782]  do_one_initcall+0x4a/0x200
[    1.904782]  kernel_init_freeable+0x1f1/0x26e
[    1.904782]  ? rest_init+0xb0/0xb0
[    1.904782]  kernel_init+0xe/0x100
[    1.904782]  ret_from_fork+0x22/0x30
[    1.904782] Modules linked in:
[    1.904782] CR2: 0000000000000015
[    1.904782] ---[ end trace c42df88d6c7b0a48 ]---

All this happens cause the cm init is in the call chain of the module init,
which is not a preferred practice.

So remove the call to rdma_create_id() from the module init call chain.
Instead register rtrs-srv as an ib client, which makes sure that the
rdma_create_id() is called only when an ib device is added.

Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
---
Change in v2:
        Use only single variable to track number of IB devices and failure
        Change according to kernel coding style

 drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
 drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
 2 files changed, 82 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
index 0d9241f5d9e6..69a37ce73b0c 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
@@ -16,6 +16,7 @@
 #include "rtrs-srv.h"
 #include "rtrs-log.h"
 #include <rdma/ib_cm.h>
+#include <rdma/ib_verbs.h>
 
 MODULE_DESCRIPTION("RDMA Transport Server");
 MODULE_LICENSE("GPL");
@@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
 static struct rtrs_rdma_dev_pd dev_pd;
 static mempool_t *chunk_pool;
 struct class *rtrs_dev_class;
+static struct rtrs_srv_ib_ctx ib_ctx;
 
 static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
 static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
@@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
 	kfree(ctx);
 }
 
+static int rtrs_srv_add_one(struct ib_device *device)
+{
+	struct rtrs_srv_ctx *ctx;
+	int ret;
+
+	if (ib_ctx.ib_dev_count)
+		goto out;
+
+	/*
+	 * Since our CM IDs are NOT bound to any ib device we will create them
+	 * only once
+	 */
+	ctx = ib_ctx.srv_ctx;
+	ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
+	if (ret) {
+		/*
+		 * We errored out here.
+		 * According to the ib code, if we encounter an error here then the
+		 * error code is ignored, and no more calls to our ops are made.
+		 */
+		pr_err("Failed to initialize RDMA connection");
+		ib_ctx.ib_dev_count = -1;
+		return ret;
+	}
+
+out:
+	/*
+	 * Keep a track on the number of ib devices added
+	 */
+	ib_ctx.ib_dev_count++;
+
+	return 0;
+}
+
+static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
+{
+	struct rtrs_srv_ctx *ctx;
+
+	ib_ctx.ib_dev_count--;
+
+	if (ib_ctx.ib_dev_count)
+		return;
+
+	/*
+	 * Since our CM IDs are NOT bound to any ib device we will remove them
+	 * only once, when the last device is removed
+	 */
+	ctx = ib_ctx.srv_ctx;
+	rdma_destroy_id(ctx->cm_id_ip);
+	rdma_destroy_id(ctx->cm_id_ib);
+}
+
+static struct ib_client rtrs_srv_client = {
+	.name	= "rtrs_server",
+	.add	= rtrs_srv_add_one,
+	.remove	= rtrs_srv_remove_one
+};
+
 /**
  * rtrs_srv_open() - open RTRS server context
  * @ops:		callback functions
@@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
 
-	err = rtrs_srv_rdma_init(ctx, port);
+	ib_ctx = (struct rtrs_srv_ib_ctx) {
+		.srv_ctx	= ctx,
+		.port		= port,
+	};
+
+	err = ib_register_client(&rtrs_srv_client);
 	if (err) {
 		free_srv_ctx(ctx);
 		return ERR_PTR(err);
 	}
 
+	/*
+	 * Since ib_register_client does not propagate the device add error
+	 * we check if .add was called and the RDMA connection init failed
+	 */
+	if (ib_ctx.ib_dev_count < 0) {
+		free_srv_ctx(ctx);
+		return ERR_PTR(-ENODEV);
+	}
+
 	return ctx;
 }
 EXPORT_SYMBOL(rtrs_srv_open);
@@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
  */
 void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
 {
-	rdma_destroy_id(ctx->cm_id_ip);
-	rdma_destroy_id(ctx->cm_id_ib);
+	ib_unregister_client(&rtrs_srv_client);
 	close_ctx(ctx);
 	free_srv_ctx(ctx);
 }
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
index dc95b0932f0d..e8f7e99a9a6e 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
@@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
 	struct list_head srv_list;
 };
 
+struct rtrs_srv_ib_ctx {
+	struct rtrs_srv_ctx	*srv_ctx;
+	u16			port;
+	int			ib_dev_count;
+};
+
 extern struct class *rtrs_dev_class;
 
 void close_sess(struct rtrs_srv_sess *sess);
-- 
2.25.1

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Danil Kipnis]

On Mon, Aug 10, 2020 at 1:51 PM Md Haris Iqbal
<haris.iqbal@cloud.ionos.com> wrote:
>
> The rnbd_server module's communication manager (cm) initialization depends
> on the registration of the "network namespace subsystem" of the RDMA CM
> agent module. As such, when the kernel is configured to load the
> rnbd_server and the RDMA cma module during initialization; and if the
> rnbd_server module is initialized before RDMA cma module, a null ptr
> dereference occurs during the RDMA bind operation.
>
> Call trace below,
>
> [    1.904782] Call Trace:
> [    1.904782]  ? xas_load+0xd/0x80
> [    1.904782]  xa_load+0x47/0x80
> [    1.904782]  cma_ps_find+0x44/0x70
> [    1.904782]  rdma_bind_addr+0x782/0x8b0
> [    1.904782]  ? get_random_bytes+0x35/0x40
> [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> [    1.904782]  rtrs_srv_open+0x102/0x180
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  rnbd_srv_init_module+0x34/0x84
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  do_one_initcall+0x4a/0x200
> [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> [    1.904782]  ? rest_init+0xb0/0xb0
> [    1.904782]  kernel_init+0xe/0x100
> [    1.904782]  ret_from_fork+0x22/0x30
> [    1.904782] Modules linked in:
> [    1.904782] CR2: 0000000000000015
> [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
>
> All this happens cause the cm init is in the call chain of the module init,
> which is not a preferred practice.
>
> So remove the call to rdma_create_id() from the module init call chain.
> Instead register rtrs-srv as an ib client, which makes sure that the
> rdma_create_id() is called only when an ib device is added.
>
> Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> ---
> Change in v2:
>         Use only single variable to track number of IB devices and failure
>         Change according to kernel coding style
>
>  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
>  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
>  2 files changed, 82 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> index 0d9241f5d9e6..69a37ce73b0c 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> @@ -16,6 +16,7 @@
>  #include "rtrs-srv.h"
>  #include "rtrs-log.h"
>  #include <rdma/ib_cm.h>
> +#include <rdma/ib_verbs.h>
>
>  MODULE_DESCRIPTION("RDMA Transport Server");
>  MODULE_LICENSE("GPL");
> @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
>  static struct rtrs_rdma_dev_pd dev_pd;
>  static mempool_t *chunk_pool;
>  struct class *rtrs_dev_class;
> +static struct rtrs_srv_ib_ctx ib_ctx;
>
>  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
>  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
>         kfree(ctx);
>  }
>
> +static int rtrs_srv_add_one(struct ib_device *device)
> +{
> +       struct rtrs_srv_ctx *ctx;
> +       int ret;
> +
> +       if (ib_ctx.ib_dev_count)
> +               goto out;
> +
> +       /*
> +        * Since our CM IDs are NOT bound to any ib device we will create them
> +        * only once
> +        */
> +       ctx = ib_ctx.srv_ctx;
> +       ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> +       if (ret) {
> +               /*
> +                * We errored out here.
> +                * According to the ib code, if we encounter an error here then the
> +                * error code is ignored, and no more calls to our ops are made.
> +                */
> +               pr_err("Failed to initialize RDMA connection");
> +               ib_ctx.ib_dev_count = -1;
> +               return ret;
> +       }
> +
> +out:
> +       /*
> +        * Keep a track on the number of ib devices added
> +        */
> +       ib_ctx.ib_dev_count++;
> +
> +       return 0;
> +}
> +
> +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> +{
> +       struct rtrs_srv_ctx *ctx;
> +
> +       ib_ctx.ib_dev_count--;
> +
> +       if (ib_ctx.ib_dev_count)
> +               return;
> +
> +       /*
> +        * Since our CM IDs are NOT bound to any ib device we will remove them
> +        * only once, when the last device is removed
> +        */
> +       ctx = ib_ctx.srv_ctx;
> +       rdma_destroy_id(ctx->cm_id_ip);
> +       rdma_destroy_id(ctx->cm_id_ib);
> +}
> +
> +static struct ib_client rtrs_srv_client = {
> +       .name   = "rtrs_server",
> +       .add    = rtrs_srv_add_one,
> +       .remove = rtrs_srv_remove_one
> +};
> +
>  /**
>   * rtrs_srv_open() - open RTRS server context
>   * @ops:               callback functions
> @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
>         if (!ctx)
>                 return ERR_PTR(-ENOMEM);
>
> -       err = rtrs_srv_rdma_init(ctx, port);
> +       ib_ctx = (struct rtrs_srv_ib_ctx) {
> +               .srv_ctx        = ctx,
> +               .port           = port,
> +       };
> +
> +       err = ib_register_client(&rtrs_srv_client);
>         if (err) {
>                 free_srv_ctx(ctx);
>                 return ERR_PTR(err);
>         }
>
> +       /*
> +        * Since ib_register_client does not propagate the device add error
> +        * we check if .add was called and the RDMA connection init failed
> +        */
> +       if (ib_ctx.ib_dev_count < 0) {
> +               free_srv_ctx(ctx);
> +               return ERR_PTR(-ENODEV);
> +       }
> +
>         return ctx;
>  }
>  EXPORT_SYMBOL(rtrs_srv_open);
> @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
>   */
>  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
>  {
> -       rdma_destroy_id(ctx->cm_id_ip);
> -       rdma_destroy_id(ctx->cm_id_ib);
> +       ib_unregister_client(&rtrs_srv_client);
>         close_ctx(ctx);
>         free_srv_ctx(ctx);
>  }
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> index dc95b0932f0d..e8f7e99a9a6e 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
>         struct list_head srv_list;
>  };
>
> +struct rtrs_srv_ib_ctx {
> +       struct rtrs_srv_ctx     *srv_ctx;
> +       u16                     port;
> +       int                     ib_dev_count;
> +};
> +
>  extern struct class *rtrs_dev_class;
>
>  void close_sess(struct rtrs_srv_sess *sess);
> --
> 2.25.1
>

This patch goes all the way registering an ib client in order to call
rdma_bind_addr only after the first device is added instead of calling
it from module_init directly.
I think the original one line fix
https://www.spinics.net/lists/linux-rdma/msg93267.html was good enough
to close a bug triggered only when all modules are build-in.
Since the convention appears to be to call ib_client_register, this
also looks good to me.
Thanks you!

Acked-by: Danil Kipnis <danil.kipnis@cloud.ionos.com>

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Jinpu Wang]

On Mon, Aug 10, 2020 at 1:51 PM Md Haris Iqbal
<haris.iqbal@cloud.ionos.com> wrote:
>
> The rnbd_server module's communication manager (cm) initialization depends
> on the registration of the "network namespace subsystem" of the RDMA CM
> agent module. As such, when the kernel is configured to load the
> rnbd_server and the RDMA cma module during initialization; and if the
> rnbd_server module is initialized before RDMA cma module, a null ptr
> dereference occurs during the RDMA bind operation.
>
> Call trace below,
>
> [    1.904782] Call Trace:
> [    1.904782]  ? xas_load+0xd/0x80
> [    1.904782]  xa_load+0x47/0x80
> [    1.904782]  cma_ps_find+0x44/0x70
> [    1.904782]  rdma_bind_addr+0x782/0x8b0
> [    1.904782]  ? get_random_bytes+0x35/0x40
> [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> [    1.904782]  rtrs_srv_open+0x102/0x180
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  rnbd_srv_init_module+0x34/0x84
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  do_one_initcall+0x4a/0x200
> [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> [    1.904782]  ? rest_init+0xb0/0xb0
> [    1.904782]  kernel_init+0xe/0x100
> [    1.904782]  ret_from_fork+0x22/0x30
> [    1.904782] Modules linked in:
> [    1.904782] CR2: 0000000000000015
> [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
>
> All this happens cause the cm init is in the call chain of the module init,
> which is not a preferred practice.
>
> So remove the call to rdma_create_id() from the module init call chain.
> Instead register rtrs-srv as an ib client, which makes sure that the
> rdma_create_id() is called only when an ib device is added.
>
> Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
Acked-by: Jack Wang <jinpu.wang@cloud.ionos.com>
Thanks!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> The rnbd_server module's communication manager (cm) initialization depends
> on the registration of the "network namespace subsystem" of the RDMA CM
> agent module. As such, when the kernel is configured to load the
> rnbd_server and the RDMA cma module during initialization; and if the
> rnbd_server module is initialized before RDMA cma module, a null ptr
> dereference occurs during the RDMA bind operation.
>
> Call trace below,
>
> [    1.904782] Call Trace:
> [    1.904782]  ? xas_load+0xd/0x80
> [    1.904782]  xa_load+0x47/0x80
> [    1.904782]  cma_ps_find+0x44/0x70
> [    1.904782]  rdma_bind_addr+0x782/0x8b0
> [    1.904782]  ? get_random_bytes+0x35/0x40
> [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> [    1.904782]  rtrs_srv_open+0x102/0x180
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  rnbd_srv_init_module+0x34/0x84
> [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> [    1.904782]  do_one_initcall+0x4a/0x200
> [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> [    1.904782]  ? rest_init+0xb0/0xb0
> [    1.904782]  kernel_init+0xe/0x100
> [    1.904782]  ret_from_fork+0x22/0x30
> [    1.904782] Modules linked in:
> [    1.904782] CR2: 0000000000000015
> [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
>
> All this happens cause the cm init is in the call chain of the module init,
> which is not a preferred practice.
>
> So remove the call to rdma_create_id() from the module init call chain.
> Instead register rtrs-srv as an ib client, which makes sure that the
> rdma_create_id() is called only when an ib device is added.
>
> Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> Reported-by: kernel test robot <rong.a.chen@intel.com>
> Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> ---
> Change in v2:
>         Use only single variable to track number of IB devices and failure
>         Change according to kernel coding style
>
>  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
>  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
>  2 files changed, 82 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> index 0d9241f5d9e6..69a37ce73b0c 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> @@ -16,6 +16,7 @@
>  #include "rtrs-srv.h"
>  #include "rtrs-log.h"
>  #include <rdma/ib_cm.h>
> +#include <rdma/ib_verbs.h>
>
>  MODULE_DESCRIPTION("RDMA Transport Server");
>  MODULE_LICENSE("GPL");
> @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
>  static struct rtrs_rdma_dev_pd dev_pd;
>  static mempool_t *chunk_pool;
>  struct class *rtrs_dev_class;
> +static struct rtrs_srv_ib_ctx ib_ctx;
>
>  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
>  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
>  	kfree(ctx);
>  }
>
> +static int rtrs_srv_add_one(struct ib_device *device)
> +{
> +	struct rtrs_srv_ctx *ctx;
> +	int ret;
> +
> +	if (ib_ctx.ib_dev_count)
> +		goto out;
> +
> +	/*
> +	 * Since our CM IDs are NOT bound to any ib device we will create them
> +	 * only once
> +	 */
> +	ctx = ib_ctx.srv_ctx;
> +	ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> +	if (ret) {
> +		/*
> +		 * We errored out here.
> +		 * According to the ib code, if we encounter an error here then the
> +		 * error code is ignored, and no more calls to our ops are made.
> +		 */
> +		pr_err("Failed to initialize RDMA connection");
> +		ib_ctx.ib_dev_count = -1;
> +		return ret;
> +	}
> +
> +out:
> +	/*
> +	 * Keep a track on the number of ib devices added
> +	 */
> +	ib_ctx.ib_dev_count++;
> +
> +	return 0;
> +}
> +
> +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> +{
> +	struct rtrs_srv_ctx *ctx;
> +
> +	ib_ctx.ib_dev_count--;
> +
> +	if (ib_ctx.ib_dev_count)
> +		return;
> +
> +	/*
> +	 * Since our CM IDs are NOT bound to any ib device we will remove them
> +	 * only once, when the last device is removed
> +	 */
> +	ctx = ib_ctx.srv_ctx;
> +	rdma_destroy_id(ctx->cm_id_ip);
> +	rdma_destroy_id(ctx->cm_id_ib);
> +}
> +
> +static struct ib_client rtrs_srv_client = {
> +	.name	= "rtrs_server",
> +	.add	= rtrs_srv_add_one,
> +	.remove	= rtrs_srv_remove_one
> +};
> +
>  /**
>   * rtrs_srv_open() - open RTRS server context
>   * @ops:		callback functions
> @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
>  	if (!ctx)
>  		return ERR_PTR(-ENOMEM);
>
> -	err = rtrs_srv_rdma_init(ctx, port);
> +	ib_ctx = (struct rtrs_srv_ib_ctx) {
> +		.srv_ctx	= ctx,
> +		.port		= port,
> +	};
> +
> +	err = ib_register_client(&rtrs_srv_client);
>  	if (err) {
>  		free_srv_ctx(ctx);
>  		return ERR_PTR(err);
>  	}
>
> +	/*
> +	 * Since ib_register_client does not propagate the device add error
> +	 * we check if .add was called and the RDMA connection init failed
> +	 */
> +	if (ib_ctx.ib_dev_count < 0) {
> +		free_srv_ctx(ctx);
> +		return ERR_PTR(-ENODEV);
> +	}

I afraid that you overcomplicated here, ib_register_client() doesn't
return error if ->add() for specific device failed, it doesn't mean
that ->add won't be called again for another device.

So you don't need to use ib_dev_count == -1, just keep it to be 0 and
leave to  rtrs_srv_close() to free srv_ctx.

Failure to call ->add shouldn't be any different from no-ib-devices situation.

Thanks

> +
>  	return ctx;
>  }
>  EXPORT_SYMBOL(rtrs_srv_open);
> @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
>   */
>  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
>  {
> -	rdma_destroy_id(ctx->cm_id_ip);
> -	rdma_destroy_id(ctx->cm_id_ib);
> +	ib_unregister_client(&rtrs_srv_client);
>  	close_ctx(ctx);
>  	free_srv_ctx(ctx);
>  }
> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> index dc95b0932f0d..e8f7e99a9a6e 100644
> --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
>  	struct list_head srv_list;
>  };
>
> +struct rtrs_srv_ib_ctx {
> +	struct rtrs_srv_ctx	*srv_ctx;
> +	u16			port;
> +	int			ib_dev_count;
> +};
> +
>  extern struct class *rtrs_dev_class;
>
>  void close_sess(struct rtrs_srv_sess *sess);
> --
> 2.25.1
>

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > The rnbd_server module's communication manager (cm) initialization depends
> > on the registration of the "network namespace subsystem" of the RDMA CM
> > agent module. As such, when the kernel is configured to load the
> > rnbd_server and the RDMA cma module during initialization; and if the
> > rnbd_server module is initialized before RDMA cma module, a null ptr
> > dereference occurs during the RDMA bind operation.
> >
> > Call trace below,
> >
> > [    1.904782] Call Trace:
> > [    1.904782]  ? xas_load+0xd/0x80
> > [    1.904782]  xa_load+0x47/0x80
> > [    1.904782]  cma_ps_find+0x44/0x70
> > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > [    1.904782]  ? get_random_bytes+0x35/0x40
> > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > [    1.904782]  rtrs_srv_open+0x102/0x180
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > [    1.904782]  do_one_initcall+0x4a/0x200
> > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > [    1.904782]  ? rest_init+0xb0/0xb0
> > [    1.904782]  kernel_init+0xe/0x100
> > [    1.904782]  ret_from_fork+0x22/0x30
> > [    1.904782] Modules linked in:
> > [    1.904782] CR2: 0000000000000015
> > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> >
> > All this happens cause the cm init is in the call chain of the module init,
> > which is not a preferred practice.
> >
> > So remove the call to rdma_create_id() from the module init call chain.
> > Instead register rtrs-srv as an ib client, which makes sure that the
> > rdma_create_id() is called only when an ib device is added.
> >
> > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > ---
> > Change in v2:
> >         Use only single variable to track number of IB devices and failure
> >         Change according to kernel coding style
> >
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> >  2 files changed, 82 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > index 0d9241f5d9e6..69a37ce73b0c 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > @@ -16,6 +16,7 @@
> >  #include "rtrs-srv.h"
> >  #include "rtrs-log.h"
> >  #include <rdma/ib_cm.h>
> > +#include <rdma/ib_verbs.h>
> >
> >  MODULE_DESCRIPTION("RDMA Transport Server");
> >  MODULE_LICENSE("GPL");
> > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> >  static struct rtrs_rdma_dev_pd dev_pd;
> >  static mempool_t *chunk_pool;
> >  struct class *rtrs_dev_class;
> > +static struct rtrs_srv_ib_ctx ib_ctx;
> >
> >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> >       kfree(ctx);
> >  }
> >
> > +static int rtrs_srv_add_one(struct ib_device *device)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +     int ret;
> > +
> > +     if (ib_ctx.ib_dev_count)
> > +             goto out;
> > +
> > +     /*
> > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > +      * only once
> > +      */
> > +     ctx = ib_ctx.srv_ctx;
> > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > +     if (ret) {
> > +             /*
> > +              * We errored out here.
> > +              * According to the ib code, if we encounter an error here then the
> > +              * error code is ignored, and no more calls to our ops are made.
> > +              */
> > +             pr_err("Failed to initialize RDMA connection");
> > +             ib_ctx.ib_dev_count = -1;
> > +             return ret;
> > +     }
> > +
> > +out:
> > +     /*
> > +      * Keep a track on the number of ib devices added
> > +      */
> > +     ib_ctx.ib_dev_count++;
> > +
> > +     return 0;
> > +}
> > +
> > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > +{
> > +     struct rtrs_srv_ctx *ctx;
> > +
> > +     ib_ctx.ib_dev_count--;
> > +
> > +     if (ib_ctx.ib_dev_count)
> > +             return;
> > +
> > +     /*
> > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > +      * only once, when the last device is removed
> > +      */
> > +     ctx = ib_ctx.srv_ctx;
> > +     rdma_destroy_id(ctx->cm_id_ip);
> > +     rdma_destroy_id(ctx->cm_id_ib);
> > +}
> > +
> > +static struct ib_client rtrs_srv_client = {
> > +     .name   = "rtrs_server",
> > +     .add    = rtrs_srv_add_one,
> > +     .remove = rtrs_srv_remove_one
> > +};
> > +
> >  /**
> >   * rtrs_srv_open() - open RTRS server context
> >   * @ops:             callback functions
> > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> >       if (!ctx)
> >               return ERR_PTR(-ENOMEM);
> >
> > -     err = rtrs_srv_rdma_init(ctx, port);
> > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > +             .srv_ctx        = ctx,
> > +             .port           = port,
> > +     };
> > +
> > +     err = ib_register_client(&rtrs_srv_client);
> >       if (err) {
> >               free_srv_ctx(ctx);
> >               return ERR_PTR(err);
> >       }
> >
> > +     /*
> > +      * Since ib_register_client does not propagate the device add error
> > +      * we check if .add was called and the RDMA connection init failed
> > +      */
> > +     if (ib_ctx.ib_dev_count < 0) {
> > +             free_srv_ctx(ctx);
> > +             return ERR_PTR(-ENODEV);
> > +     }
>
> I afraid that you overcomplicated here, ib_register_client() doesn't
> return error if ->add() for specific device failed, it doesn't mean
> that ->add won't be called again for another device.
>
> So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> leave to  rtrs_srv_close() to free srv_ctx.

Leaving it 0 when there is an error is not gonna work. Since when the
modules are all built-in, a call to ib_register_client() will not
result in a call to ->add() then and there. So ib_register_client()
will return after registering the client, but without calling ->add().
Which means, ib_dev_count would be 0.

>
> Failure to call ->add shouldn't be any different from no-ib-devices situation.
>
> Thanks
>
> > +
> >       return ctx;
> >  }
> >  EXPORT_SYMBOL(rtrs_srv_open);
> > @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> >   */
> >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> >  {
> > -     rdma_destroy_id(ctx->cm_id_ip);
> > -     rdma_destroy_id(ctx->cm_id_ib);
> > +     ib_unregister_client(&rtrs_srv_client);
> >       close_ctx(ctx);
> >       free_srv_ctx(ctx);
> >  }
> > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > index dc95b0932f0d..e8f7e99a9a6e 100644
> > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
> >       struct list_head srv_list;
> >  };
> >
> > +struct rtrs_srv_ib_ctx {
> > +     struct rtrs_srv_ctx     *srv_ctx;
> > +     u16                     port;
> > +     int                     ib_dev_count;
> > +};
> > +
> >  extern struct class *rtrs_dev_class;
> >
> >  void close_sess(struct rtrs_srv_sess *sess);
> > --
> > 2.25.1
> >



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > The rnbd_server module's communication manager (cm) initialization depends
> > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > agent module. As such, when the kernel is configured to load the
> > > rnbd_server and the RDMA cma module during initialization; and if the
> > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > dereference occurs during the RDMA bind operation.
> > >
> > > Call trace below,
> > >
> > > [    1.904782] Call Trace:
> > > [    1.904782]  ? xas_load+0xd/0x80
> > > [    1.904782]  xa_load+0x47/0x80
> > > [    1.904782]  cma_ps_find+0x44/0x70
> > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > [    1.904782]  kernel_init+0xe/0x100
> > > [    1.904782]  ret_from_fork+0x22/0x30
> > > [    1.904782] Modules linked in:
> > > [    1.904782] CR2: 0000000000000015
> > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > >
> > > All this happens cause the cm init is in the call chain of the module init,
> > > which is not a preferred practice.
> > >
> > > So remove the call to rdma_create_id() from the module init call chain.
> > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > rdma_create_id() is called only when an ib device is added.
> > >
> > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > ---
> > > Change in v2:
> > >         Use only single variable to track number of IB devices and failure
> > >         Change according to kernel coding style
> > >
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > @@ -16,6 +16,7 @@
> > >  #include "rtrs-srv.h"
> > >  #include "rtrs-log.h"
> > >  #include <rdma/ib_cm.h>
> > > +#include <rdma/ib_verbs.h>
> > >
> > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > >  MODULE_LICENSE("GPL");
> > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > >  static struct rtrs_rdma_dev_pd dev_pd;
> > >  static mempool_t *chunk_pool;
> > >  struct class *rtrs_dev_class;
> > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > >
> > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > >       kfree(ctx);
> > >  }
> > >
> > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > +{
> > > +     struct rtrs_srv_ctx *ctx;
> > > +     int ret;
> > > +
> > > +     if (ib_ctx.ib_dev_count)
> > > +             goto out;
> > > +
> > > +     /*
> > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > +      * only once
> > > +      */
> > > +     ctx = ib_ctx.srv_ctx;
> > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > +     if (ret) {
> > > +             /*
> > > +              * We errored out here.
> > > +              * According to the ib code, if we encounter an error here then the
> > > +              * error code is ignored, and no more calls to our ops are made.
> > > +              */
> > > +             pr_err("Failed to initialize RDMA connection");
> > > +             ib_ctx.ib_dev_count = -1;
> > > +             return ret;
> > > +     }
> > > +
> > > +out:
> > > +     /*
> > > +      * Keep a track on the number of ib devices added
> > > +      */
> > > +     ib_ctx.ib_dev_count++;
> > > +
> > > +     return 0;
> > > +}
> > > +
> > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > +{
> > > +     struct rtrs_srv_ctx *ctx;
> > > +
> > > +     ib_ctx.ib_dev_count--;
> > > +
> > > +     if (ib_ctx.ib_dev_count)
> > > +             return;
> > > +
> > > +     /*
> > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > +      * only once, when the last device is removed
> > > +      */
> > > +     ctx = ib_ctx.srv_ctx;
> > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > +}
> > > +
> > > +static struct ib_client rtrs_srv_client = {
> > > +     .name   = "rtrs_server",
> > > +     .add    = rtrs_srv_add_one,
> > > +     .remove = rtrs_srv_remove_one
> > > +};
> > > +
> > >  /**
> > >   * rtrs_srv_open() - open RTRS server context
> > >   * @ops:             callback functions
> > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > >       if (!ctx)
> > >               return ERR_PTR(-ENOMEM);
> > >
> > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > +             .srv_ctx        = ctx,
> > > +             .port           = port,
> > > +     };
> > > +
> > > +     err = ib_register_client(&rtrs_srv_client);
> > >       if (err) {
> > >               free_srv_ctx(ctx);
> > >               return ERR_PTR(err);
> > >       }
> > >
> > > +     /*
> > > +      * Since ib_register_client does not propagate the device add error
> > > +      * we check if .add was called and the RDMA connection init failed
> > > +      */
> > > +     if (ib_ctx.ib_dev_count < 0) {
> > > +             free_srv_ctx(ctx);
> > > +             return ERR_PTR(-ENODEV);
> > > +     }
> >
> > I afraid that you overcomplicated here, ib_register_client() doesn't
> > return error if ->add() for specific device failed, it doesn't mean
> > that ->add won't be called again for another device.
> >
> > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > leave to  rtrs_srv_close() to free srv_ctx.
>
> Leaving it 0 when there is an error is not gonna work. Since when the
> modules are all built-in, a call to ib_register_client() will not
> result in a call to ->add() then and there. So ib_register_client()
> will return after registering the client, but without calling ->add().
> Which means, ib_dev_count would be 0.

If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
to release.

Thanks

>
> >
> > Failure to call ->add shouldn't be any different from no-ib-devices situation.
> >
> > Thanks
> >
> > > +
> > >       return ctx;
> > >  }
> > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > >   */
> > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > >  {
> > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > +     ib_unregister_client(&rtrs_srv_client);
> > >       close_ctx(ctx);
> > >       free_srv_ctx(ctx);
> > >  }
> > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > index dc95b0932f0d..e8f7e99a9a6e 100644
> > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
> > >       struct list_head srv_list;
> > >  };
> > >
> > > +struct rtrs_srv_ib_ctx {
> > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > +     u16                     port;
> > > +     int                     ib_dev_count;
> > > +};
> > > +
> > >  extern struct class *rtrs_dev_class;
> > >
> > >  void close_sess(struct rtrs_srv_sess *sess);
> > > --
> > > 2.25.1
> > >
>
>
>
> --
>
> Regards
> -Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > agent module. As such, when the kernel is configured to load the
> > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > dereference occurs during the RDMA bind operation.
> > > >
> > > > Call trace below,
> > > >
> > > > [    1.904782] Call Trace:
> > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > [    1.904782]  xa_load+0x47/0x80
> > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > [    1.904782]  kernel_init+0xe/0x100
> > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > [    1.904782] Modules linked in:
> > > > [    1.904782] CR2: 0000000000000015
> > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > >
> > > > All this happens cause the cm init is in the call chain of the module init,
> > > > which is not a preferred practice.
> > > >
> > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > rdma_create_id() is called only when an ib device is added.
> > > >
> > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > ---
> > > > Change in v2:
> > > >         Use only single variable to track number of IB devices and failure
> > > >         Change according to kernel coding style
> > > >
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > @@ -16,6 +16,7 @@
> > > >  #include "rtrs-srv.h"
> > > >  #include "rtrs-log.h"
> > > >  #include <rdma/ib_cm.h>
> > > > +#include <rdma/ib_verbs.h>
> > > >
> > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > >  MODULE_LICENSE("GPL");
> > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > >  static mempool_t *chunk_pool;
> > > >  struct class *rtrs_dev_class;
> > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > >
> > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > >       kfree(ctx);
> > > >  }
> > > >
> > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > +{
> > > > +     struct rtrs_srv_ctx *ctx;
> > > > +     int ret;
> > > > +
> > > > +     if (ib_ctx.ib_dev_count)
> > > > +             goto out;
> > > > +
> > > > +     /*
> > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > +      * only once
> > > > +      */
> > > > +     ctx = ib_ctx.srv_ctx;
> > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > +     if (ret) {
> > > > +             /*
> > > > +              * We errored out here.
> > > > +              * According to the ib code, if we encounter an error here then the
> > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > +              */
> > > > +             pr_err("Failed to initialize RDMA connection");
> > > > +             ib_ctx.ib_dev_count = -1;
> > > > +             return ret;
> > > > +     }
> > > > +
> > > > +out:
> > > > +     /*
> > > > +      * Keep a track on the number of ib devices added
> > > > +      */
> > > > +     ib_ctx.ib_dev_count++;
> > > > +
> > > > +     return 0;
> > > > +}
> > > > +
> > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > +{
> > > > +     struct rtrs_srv_ctx *ctx;
> > > > +
> > > > +     ib_ctx.ib_dev_count--;
> > > > +
> > > > +     if (ib_ctx.ib_dev_count)
> > > > +             return;
> > > > +
> > > > +     /*
> > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > +      * only once, when the last device is removed
> > > > +      */
> > > > +     ctx = ib_ctx.srv_ctx;
> > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > +}
> > > > +
> > > > +static struct ib_client rtrs_srv_client = {
> > > > +     .name   = "rtrs_server",
> > > > +     .add    = rtrs_srv_add_one,
> > > > +     .remove = rtrs_srv_remove_one
> > > > +};
> > > > +
> > > >  /**
> > > >   * rtrs_srv_open() - open RTRS server context
> > > >   * @ops:             callback functions
> > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > >       if (!ctx)
> > > >               return ERR_PTR(-ENOMEM);
> > > >
> > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > +             .srv_ctx        = ctx,
> > > > +             .port           = port,
> > > > +     };
> > > > +
> > > > +     err = ib_register_client(&rtrs_srv_client);
> > > >       if (err) {
> > > >               free_srv_ctx(ctx);
> > > >               return ERR_PTR(err);
> > > >       }
> > > >
> > > > +     /*
> > > > +      * Since ib_register_client does not propagate the device add error
> > > > +      * we check if .add was called and the RDMA connection init failed
> > > > +      */
> > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > +             free_srv_ctx(ctx);
> > > > +             return ERR_PTR(-ENODEV);
> > > > +     }
> > >
> > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > return error if ->add() for specific device failed, it doesn't mean
> > > that ->add won't be called again for another device.
> > >
> > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > leave to  rtrs_srv_close() to free srv_ctx.
> >
> > Leaving it 0 when there is an error is not gonna work. Since when the
> > modules are all built-in, a call to ib_register_client() will not
> > result in a call to ->add() then and there. So ib_register_client()
> > will return after registering the client, but without calling ->add().
> > Which means, ib_dev_count would be 0.
>
> If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> to release.

True, but we have to send a failure back to the caller of
"rtrs_srv_open()" (and user of this ulp); which in our case is
rnbd-srv's function rnbd_srv_init_module().
In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
fails, meaning rtrs_srv_rdma_init() had failed.

Even if we are talking in generic terms, any module calling the
"rtrs_srv_open()" of the rtrs ulp, would want to know if the server
open failed or succeeded right?

>
> Thanks
>
> >
> > >
> > > Failure to call ->add shouldn't be any different from no-ib-devices situation.
> > >
> > > Thanks
> > >
> > > > +
> > > >       return ctx;
> > > >  }
> > > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > > @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > > >   */
> > > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > > >  {
> > > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > > +     ib_unregister_client(&rtrs_srv_client);
> > > >       close_ctx(ctx);
> > > >       free_srv_ctx(ctx);
> > > >  }
> > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > index dc95b0932f0d..e8f7e99a9a6e 100644
> > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
> > > >       struct list_head srv_list;
> > > >  };
> > > >
> > > > +struct rtrs_srv_ib_ctx {
> > > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > > +     u16                     port;
> > > > +     int                     ib_dev_count;
> > > > +};
> > > > +
> > > >  extern struct class *rtrs_dev_class;
> > > >
> > > >  void close_sess(struct rtrs_srv_sess *sess);
> > > > --
> > > > 2.25.1
> > > >
> >
> >
> >
> > --
> >
> > Regards
> > -Haris



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 11, 2020 at 04:23:46PM +0530, Haris Iqbal wrote:
> On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > agent module. As such, when the kernel is configured to load the
> > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > dereference occurs during the RDMA bind operation.
> > > > >
> > > > > Call trace below,
> > > > >
> > > > > [    1.904782] Call Trace:
> > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > [    1.904782] Modules linked in:
> > > > > [    1.904782] CR2: 0000000000000015
> > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > >
> > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > which is not a preferred practice.
> > > > >
> > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > rdma_create_id() is called only when an ib device is added.
> > > > >
> > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > ---
> > > > > Change in v2:
> > > > >         Use only single variable to track number of IB devices and failure
> > > > >         Change according to kernel coding style
> > > > >
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > @@ -16,6 +16,7 @@
> > > > >  #include "rtrs-srv.h"
> > > > >  #include "rtrs-log.h"
> > > > >  #include <rdma/ib_cm.h>
> > > > > +#include <rdma/ib_verbs.h>
> > > > >
> > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > >  MODULE_LICENSE("GPL");
> > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > >  static mempool_t *chunk_pool;
> > > > >  struct class *rtrs_dev_class;
> > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > >
> > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > >       kfree(ctx);
> > > > >  }
> > > > >
> > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +     int ret;
> > > > > +
> > > > > +     if (ib_ctx.ib_dev_count)
> > > > > +             goto out;
> > > > > +
> > > > > +     /*
> > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > +      * only once
> > > > > +      */
> > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > +     if (ret) {
> > > > > +             /*
> > > > > +              * We errored out here.
> > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > +              */
> > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > +             return ret;
> > > > > +     }
> > > > > +
> > > > > +out:
> > > > > +     /*
> > > > > +      * Keep a track on the number of ib devices added
> > > > > +      */
> > > > > +     ib_ctx.ib_dev_count++;
> > > > > +
> > > > > +     return 0;
> > > > > +}
> > > > > +
> > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +
> > > > > +     ib_ctx.ib_dev_count--;
> > > > > +
> > > > > +     if (ib_ctx.ib_dev_count)
> > > > > +             return;
> > > > > +
> > > > > +     /*
> > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > +      * only once, when the last device is removed
> > > > > +      */
> > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > +}
> > > > > +
> > > > > +static struct ib_client rtrs_srv_client = {
> > > > > +     .name   = "rtrs_server",
> > > > > +     .add    = rtrs_srv_add_one,
> > > > > +     .remove = rtrs_srv_remove_one
> > > > > +};
> > > > > +
> > > > >  /**
> > > > >   * rtrs_srv_open() - open RTRS server context
> > > > >   * @ops:             callback functions
> > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > >       if (!ctx)
> > > > >               return ERR_PTR(-ENOMEM);
> > > > >
> > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > +             .srv_ctx        = ctx,
> > > > > +             .port           = port,
> > > > > +     };
> > > > > +
> > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > >       if (err) {
> > > > >               free_srv_ctx(ctx);
> > > > >               return ERR_PTR(err);
> > > > >       }
> > > > >
> > > > > +     /*
> > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > +      */
> > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > +             free_srv_ctx(ctx);
> > > > > +             return ERR_PTR(-ENODEV);
> > > > > +     }
> > > >
> > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > return error if ->add() for specific device failed, it doesn't mean
> > > > that ->add won't be called again for another device.
> > > >
> > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > leave to  rtrs_srv_close() to free srv_ctx.
> > >
> > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > modules are all built-in, a call to ib_register_client() will not
> > > result in a call to ->add() then and there. So ib_register_client()
> > > will return after registering the client, but without calling ->add().
> > > Which means, ib_dev_count would be 0.
> >
> > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > to release.
>
> True, but we have to send a failure back to the caller of
> "rtrs_srv_open()" (and user of this ulp); which in our case is
> rnbd-srv's function rnbd_srv_init_module().
> In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> fails, meaning rtrs_srv_rdma_init() had failed.
>
> Even if we are talking in generic terms, any module calling the
> "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> open failed or succeeded right?

I see no difference between system without ib_devices and failure of rtrs_srv_rdma_init().
In both cases ib_dev_count == 0 and once new ib_device will come, the
ib_core will call to client->add() again.

ib_register_client() success => rtrs_srv_open() success.

Thanks

>
> >
> > Thanks
> >
> > >
> > > >
> > > > Failure to call ->add shouldn't be any different from no-ib-devices situation.
> > > >
> > > > Thanks
> > > >
> > > > > +
> > > > >       return ctx;
> > > > >  }
> > > > >  EXPORT_SYMBOL(rtrs_srv_open);
> > > > > @@ -2090,8 +2164,7 @@ static void close_ctx(struct rtrs_srv_ctx *ctx)
> > > > >   */
> > > > >  void rtrs_srv_close(struct rtrs_srv_ctx *ctx)
> > > > >  {
> > > > > -     rdma_destroy_id(ctx->cm_id_ip);
> > > > > -     rdma_destroy_id(ctx->cm_id_ib);
> > > > > +     ib_unregister_client(&rtrs_srv_client);
> > > > >       close_ctx(ctx);
> > > > >       free_srv_ctx(ctx);
> > > > >  }
> > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > index dc95b0932f0d..e8f7e99a9a6e 100644
> > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h
> > > > > @@ -118,6 +118,12 @@ struct rtrs_srv_ctx {
> > > > >       struct list_head srv_list;
> > > > >  };
> > > > >
> > > > > +struct rtrs_srv_ib_ctx {
> > > > > +     struct rtrs_srv_ctx     *srv_ctx;
> > > > > +     u16                     port;
> > > > > +     int                     ib_dev_count;
> > > > > +};
> > > > > +
> > > > >  extern struct class *rtrs_dev_class;
> > > > >
> > > > >  void close_sess(struct rtrs_srv_sess *sess);
> > > > > --
> > > > > 2.25.1
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards
> > > -Haris
>
>
>
> --
>
> Regards
> -Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Jinpu Wang]

On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
<haris.iqbal@cloud.ionos.com> wrote:
>
> On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > agent module. As such, when the kernel is configured to load the
> > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > dereference occurs during the RDMA bind operation.
> > > > >
> > > > > Call trace below,
> > > > >
> > > > > [    1.904782] Call Trace:
> > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > [    1.904782] Modules linked in:
> > > > > [    1.904782] CR2: 0000000000000015
> > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > >
> > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > which is not a preferred practice.
> > > > >
> > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > rdma_create_id() is called only when an ib device is added.
> > > > >
> > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > ---
> > > > > Change in v2:
> > > > >         Use only single variable to track number of IB devices and failure
> > > > >         Change according to kernel coding style
> > > > >
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > @@ -16,6 +16,7 @@
> > > > >  #include "rtrs-srv.h"
> > > > >  #include "rtrs-log.h"
> > > > >  #include <rdma/ib_cm.h>
> > > > > +#include <rdma/ib_verbs.h>
> > > > >
> > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > >  MODULE_LICENSE("GPL");
> > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > >  static mempool_t *chunk_pool;
> > > > >  struct class *rtrs_dev_class;
> > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > >
> > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > >       kfree(ctx);
> > > > >  }
> > > > >
> > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +     int ret;
> > > > > +
> > > > > +     if (ib_ctx.ib_dev_count)
> > > > > +             goto out;
> > > > > +
> > > > > +     /*
> > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > +      * only once
> > > > > +      */
> > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > +     if (ret) {
> > > > > +             /*
> > > > > +              * We errored out here.
> > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > +              */
> > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > +             return ret;
> > > > > +     }
> > > > > +
> > > > > +out:
> > > > > +     /*
> > > > > +      * Keep a track on the number of ib devices added
> > > > > +      */
> > > > > +     ib_ctx.ib_dev_count++;
> > > > > +
> > > > > +     return 0;
> > > > > +}
> > > > > +
> > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > +{
> > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > +
> > > > > +     ib_ctx.ib_dev_count--;
> > > > > +
> > > > > +     if (ib_ctx.ib_dev_count)
> > > > > +             return;
> > > > > +
> > > > > +     /*
> > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > +      * only once, when the last device is removed
> > > > > +      */
> > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > +}
> > > > > +
> > > > > +static struct ib_client rtrs_srv_client = {
> > > > > +     .name   = "rtrs_server",
> > > > > +     .add    = rtrs_srv_add_one,
> > > > > +     .remove = rtrs_srv_remove_one
> > > > > +};
> > > > > +
> > > > >  /**
> > > > >   * rtrs_srv_open() - open RTRS server context
> > > > >   * @ops:             callback functions
> > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > >       if (!ctx)
> > > > >               return ERR_PTR(-ENOMEM);
> > > > >
> > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > +             .srv_ctx        = ctx,
> > > > > +             .port           = port,
> > > > > +     };
> > > > > +
> > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > >       if (err) {
> > > > >               free_srv_ctx(ctx);
> > > > >               return ERR_PTR(err);
> > > > >       }
> > > > >
> > > > > +     /*
> > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > +      */
> > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > +             free_srv_ctx(ctx);
> > > > > +             return ERR_PTR(-ENODEV);
> > > > > +     }
> > > >
> > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > return error if ->add() for specific device failed, it doesn't mean
> > > > that ->add won't be called again for another device.
> > > >
> > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > leave to  rtrs_srv_close() to free srv_ctx.
> > >
> > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > modules are all built-in, a call to ib_register_client() will not
> > > result in a call to ->add() then and there. So ib_register_client()
> > > will return after registering the client, but without calling ->add().
> > > Which means, ib_dev_count would be 0.
> >
> > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > to release.
>
> True, but we have to send a failure back to the caller of
> "rtrs_srv_open()" (and user of this ulp); which in our case is
> rnbd-srv's function rnbd_srv_init_module().
> In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> fails, meaning rtrs_srv_rdma_init() had failed.
>
> Even if we are talking in generic terms, any module calling the
> "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> open failed or succeeded right?
I think Leon is right, any success of call to ->add, we have something
to rtrs_srv_rdma_init,
we can consider rtrs_srv_open is success instead of an error.

Thanks Leon for catching this.

Regards!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Danil Kipnis]

On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
>
> On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> <haris.iqbal@cloud.ionos.com> wrote:
> >
> > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > dereference occurs during the RDMA bind operation.
> > > > > >
> > > > > > Call trace below,
> > > > > >
> > > > > > [    1.904782] Call Trace:
> > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > [    1.904782] Modules linked in:
> > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > >
> > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > which is not a preferred practice.
> > > > > >
> > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > >
> > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > ---
> > > > > > Change in v2:
> > > > > >         Use only single variable to track number of IB devices and failure
> > > > > >         Change according to kernel coding style
> > > > > >
> > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > >
> > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > @@ -16,6 +16,7 @@
> > > > > >  #include "rtrs-srv.h"
> > > > > >  #include "rtrs-log.h"
> > > > > >  #include <rdma/ib_cm.h>
> > > > > > +#include <rdma/ib_verbs.h>
> > > > > >
> > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > >  MODULE_LICENSE("GPL");
> > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > >  static mempool_t *chunk_pool;
> > > > > >  struct class *rtrs_dev_class;
> > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > >
> > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > >       kfree(ctx);
> > > > > >  }
> > > > > >
> > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > +{
> > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > +     int ret;
> > > > > > +
> > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > +             goto out;
> > > > > > +
> > > > > > +     /*
> > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > +      * only once
> > > > > > +      */
> > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > +     if (ret) {
> > > > > > +             /*
> > > > > > +              * We errored out here.
> > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > +              */
> > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > +             return ret;
> > > > > > +     }
> > > > > > +
> > > > > > +out:
> > > > > > +     /*
> > > > > > +      * Keep a track on the number of ib devices added
> > > > > > +      */
> > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > +
> > > > > > +     return 0;
> > > > > > +}
> > > > > > +
> > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > +{
> > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > +
> > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > +
> > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > +             return;
> > > > > > +
> > > > > > +     /*
> > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > +      * only once, when the last device is removed
> > > > > > +      */
> > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > +}
> > > > > > +
> > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > +     .name   = "rtrs_server",
> > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > +};
> > > > > > +
> > > > > >  /**
> > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > >   * @ops:             callback functions
> > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > >       if (!ctx)
> > > > > >               return ERR_PTR(-ENOMEM);
> > > > > >
> > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > +             .srv_ctx        = ctx,
> > > > > > +             .port           = port,
> > > > > > +     };
> > > > > > +
> > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > >       if (err) {
> > > > > >               free_srv_ctx(ctx);
> > > > > >               return ERR_PTR(err);
> > > > > >       }
> > > > > >
> > > > > > +     /*
> > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > +      */
> > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > +             free_srv_ctx(ctx);
> > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > +     }
> > > > >
> > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > that ->add won't be called again for another device.
> > > > >
> > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > >
> > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > modules are all built-in, a call to ib_register_client() will not
> > > > result in a call to ->add() then and there. So ib_register_client()
> > > > will return after registering the client, but without calling ->add().
> > > > Which means, ib_dev_count would be 0.
> > >
> > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > to release.
> >
> > True, but we have to send a failure back to the caller of
> > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > rnbd-srv's function rnbd_srv_init_module().
> > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > fails, meaning rtrs_srv_rdma_init() had failed.
> >
> > Even if we are talking in generic terms, any module calling the
> > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > open failed or succeeded right?
> I think Leon is right, any success of call to ->add, we have something
> to rtrs_srv_rdma_init,
> we can consider rtrs_srv_open is success instead of an error.

I don't think Leon is right. In case when all modules are built in,
add is not called in place when ib_client_register is called (cause
there are no registered devices and add_client_context will be called
at some later point in time from enable_device_and_get instead). So we
have two cases: built in and not built in. Then there is a possibility
that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
fail. Particularly we need to separate the case where add hasn't been
called at all yet (i.e. modules are built in, we just need to wait
until add gets called and then can start listening) and the case where
it did get called but failed (i.e. modules are not built in it got
called but failed - in that case we need to refuse to load module and
return error since we can't start listening).
The latter case is indicated by ib_dev_count = -1. I think it would
make code easier to read if instead of setting ib_dev_count to -1 and
explicitly checking whether it's below 0, one would introduce an
additional variable for the error code returned by rtrs_srv_rdma_init
and check it instead.

>
> Thanks Leon for catching this.
>
> Regards!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> >
> > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > <haris.iqbal@cloud.ionos.com> wrote:
> > >
> > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > >
> > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > >
> > > > > > > Call trace below,
> > > > > > >
> > > > > > > [    1.904782] Call Trace:
> > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > [    1.904782] Modules linked in:
> > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > >
> > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > which is not a preferred practice.
> > > > > > >
> > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > >
> > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > ---
> > > > > > > Change in v2:
> > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > >         Change according to kernel coding style
> > > > > > >
> > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > >
> > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > @@ -16,6 +16,7 @@
> > > > > > >  #include "rtrs-srv.h"
> > > > > > >  #include "rtrs-log.h"
> > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > >
> > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > >  static mempool_t *chunk_pool;
> > > > > > >  struct class *rtrs_dev_class;
> > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > >
> > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > >       kfree(ctx);
> > > > > > >  }
> > > > > > >
> > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > +{
> > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > +     int ret;
> > > > > > > +
> > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > +             goto out;
> > > > > > > +
> > > > > > > +     /*
> > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > +      * only once
> > > > > > > +      */
> > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > +     if (ret) {
> > > > > > > +             /*
> > > > > > > +              * We errored out here.
> > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > +              */
> > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > +             return ret;
> > > > > > > +     }
> > > > > > > +
> > > > > > > +out:
> > > > > > > +     /*
> > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > +      */
> > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > +
> > > > > > > +     return 0;
> > > > > > > +}
> > > > > > > +
> > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > +{
> > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > +
> > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > +
> > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > +             return;
> > > > > > > +
> > > > > > > +     /*
> > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > +      * only once, when the last device is removed
> > > > > > > +      */
> > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > +}
> > > > > > > +
> > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > +     .name   = "rtrs_server",
> > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > +};
> > > > > > > +
> > > > > > >  /**
> > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > >   * @ops:             callback functions
> > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > >       if (!ctx)
> > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > >
> > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > +             .srv_ctx        = ctx,
> > > > > > > +             .port           = port,
> > > > > > > +     };
> > > > > > > +
> > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > >       if (err) {
> > > > > > >               free_srv_ctx(ctx);
> > > > > > >               return ERR_PTR(err);
> > > > > > >       }
> > > > > > >
> > > > > > > +     /*
> > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > +      */
> > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > +             free_srv_ctx(ctx);
> > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > +     }
> > > > > >
> > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > that ->add won't be called again for another device.
> > > > > >
> > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > >
> > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > will return after registering the client, but without calling ->add().
> > > > > Which means, ib_dev_count would be 0.
> > > >
> > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > to release.
> > >
> > > True, but we have to send a failure back to the caller of
> > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > rnbd-srv's function rnbd_srv_init_module().
> > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > fails, meaning rtrs_srv_rdma_init() had failed.
> > >
> > > Even if we are talking in generic terms, any module calling the
> > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > open failed or succeeded right?
> > I think Leon is right, any success of call to ->add, we have something
> > to rtrs_srv_rdma_init,
> > we can consider rtrs_srv_open is success instead of an error.
>
> I don't think Leon is right. In case when all modules are built in,
> add is not called in place when ib_client_register is called (cause
> there are no registered devices and add_client_context will be called
> at some later point in time from enable_device_and_get instead). So we
> have two cases: built in and not built in. Then there is a possibility
> that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> fail. Particularly we need to separate the case where add hasn't been
> called at all yet (i.e. modules are built in, we just need to wait
> until add gets called and then can start listening) and the case where
> it did get called but failed (i.e. modules are not built in it got
> called but failed - in that case we need to refuse to load module and
> return error since we can't start listening).
> The latter case is indicated by ib_dev_count = -1. I think it would
> make code easier to read if instead of setting ib_dev_count to -1 and
> explicitly checking whether it's below 0, one would introduce an
> additional variable for the error code returned by rtrs_srv_rdma_init
> and check it instead.

Sorry, but it is very hard to read this block without indentations.

Anyway, as I said there should be no difference in behaviour between no
devices and first device failed to initialize.

Thanks

>
> >
> > Thanks Leon for catching this.
> >
> > Regards!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Danil Kipnis]

On Tue, Aug 11, 2020 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> > On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> > >
> > > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > > <haris.iqbal@cloud.ionos.com> wrote:
> > > >
> > > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > >
> > > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > >
> > > > > > > > Call trace below,
> > > > > > > >
> > > > > > > > [    1.904782] Call Trace:
> > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > >
> > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > which is not a preferred practice.
> > > > > > > >
> > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > >
> > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > ---
> > > > > > > > Change in v2:
> > > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > > >         Change according to kernel coding style
> > > > > > > >
> > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > >  #include "rtrs-srv.h"
> > > > > > > >  #include "rtrs-log.h"
> > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > >
> > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > >
> > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > >       kfree(ctx);
> > > > > > > >  }
> > > > > > > >
> > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > +{
> > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > +     int ret;
> > > > > > > > +
> > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > +             goto out;
> > > > > > > > +
> > > > > > > > +     /*
> > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > +      * only once
> > > > > > > > +      */
> > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > +     if (ret) {
> > > > > > > > +             /*
> > > > > > > > +              * We errored out here.
> > > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > > +              */
> > > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > > +             return ret;
> > > > > > > > +     }
> > > > > > > > +
> > > > > > > > +out:
> > > > > > > > +     /*
> > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > +      */
> > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > +
> > > > > > > > +     return 0;
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > > +{
> > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > +
> > > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > > +
> > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > +             return;
> > > > > > > > +
> > > > > > > > +     /*
> > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > > +      * only once, when the last device is removed
> > > > > > > > +      */
> > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > > +     .name   = "rtrs_server",
> > > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > > +};
> > > > > > > > +
> > > > > > > >  /**
> > > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > > >   * @ops:             callback functions
> > > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > > >       if (!ctx)
> > > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > > >
> > > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > > +             .srv_ctx        = ctx,
> > > > > > > > +             .port           = port,
> > > > > > > > +     };
> > > > > > > > +
> > > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > > >       if (err) {
> > > > > > > >               free_srv_ctx(ctx);
> > > > > > > >               return ERR_PTR(err);
> > > > > > > >       }
> > > > > > > >
> > > > > > > > +     /*
> > > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > > +      */
> > > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > > +             free_srv_ctx(ctx);
> > > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > > +     }
> > > > > > >
> > > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > > that ->add won't be called again for another device.
> > > > > > >
> > > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > > >
> > > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > > will return after registering the client, but without calling ->add().
> > > > > > Which means, ib_dev_count would be 0.
> > > > >
> > > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > > to release.
> > > >
> > > > True, but we have to send a failure back to the caller of
> > > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > > rnbd-srv's function rnbd_srv_init_module().
> > > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > > fails, meaning rtrs_srv_rdma_init() had failed.
> > > >
> > > > Even if we are talking in generic terms, any module calling the
> > > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > > open failed or succeeded right?
> > > I think Leon is right, any success of call to ->add, we have something
> > > to rtrs_srv_rdma_init,
> > > we can consider rtrs_srv_open is success instead of an error.
> >
> > I don't think Leon is right. In case when all modules are built in,
> > add is not called in place when ib_client_register is called (cause
> > there are no registered devices and add_client_context will be called
> > at some later point in time from enable_device_and_get instead). So we
> > have two cases: built in and not built in. Then there is a possibility
> > that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> > fail. Particularly we need to separate the case where add hasn't been
> > called at all yet (i.e. modules are built in, we just need to wait
> > until add gets called and then can start listening) and the case where
> > it did get called but failed (i.e. modules are not built in it got
> > called but failed - in that case we need to refuse to load module and
> > return error since we can't start listening).
> > The latter case is indicated by ib_dev_count = -1. I think it would
> > make code easier to read if instead of setting ib_dev_count to -1 and
> > explicitly checking whether it's below 0, one would introduce an
> > additional variable for the error code returned by rtrs_srv_rdma_init
> > and check it instead.
>
> Sorry, but it is very hard to read this block without indentations.

Do you mean like empty lines between text paragraphs?

>
> Anyway, as I said there should be no difference in behaviour between no
> devices and first device failed to initialize.

If we load the server module and fail to initialize the device, we
return an error to the user (insmod fails). We could just log
something into dmesg and stay there doing nothing. Is that what you
suggest? Then one would also probably want to have a way to say to the
server it should try to init again (currently one just tries insmod
again)...

Thank you,
Danil

>
> Thanks
>
> >
> > >
> > > Thanks Leon for catching this.
> > >
> > > Regards!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 11, 2020 at 02:32:38PM +0200, Danil Kipnis wrote:
> On Tue, Aug 11, 2020 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> > > On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> > > >
> > > > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > > > <haris.iqbal@cloud.ionos.com> wrote:
> > > > >
> > > > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > >
> > > > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > > >
> > > > > > > > > Call trace below,
> > > > > > > > >
> > > > > > > > > [    1.904782] Call Trace:
> > > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > > >
> > > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > > which is not a preferred practice.
> > > > > > > > >
> > > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > > >
> > > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > > ---
> > > > > > > > > Change in v2:
> > > > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > > > >         Change according to kernel coding style
> > > > > > > > >
> > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > >  #include "rtrs-srv.h"
> > > > > > > > >  #include "rtrs-log.h"
> > > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > > >
> > > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > > >
> > > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > > >       kfree(ctx);
> > > > > > > > >  }
> > > > > > > > >
> > > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > > +{
> > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > +     int ret;
> > > > > > > > > +
> > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > +             goto out;
> > > > > > > > > +
> > > > > > > > > +     /*
> > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > > +      * only once
> > > > > > > > > +      */
> > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > > +     if (ret) {
> > > > > > > > > +             /*
> > > > > > > > > +              * We errored out here.
> > > > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > > > +              */
> > > > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > > > +             return ret;
> > > > > > > > > +     }
> > > > > > > > > +
> > > > > > > > > +out:
> > > > > > > > > +     /*
> > > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > > +      */
> > > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > > +
> > > > > > > > > +     return 0;
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > > > +{
> > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > +
> > > > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > > > +
> > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > +             return;
> > > > > > > > > +
> > > > > > > > > +     /*
> > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > > > +      * only once, when the last device is removed
> > > > > > > > > +      */
> > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > > > +     .name   = "rtrs_server",
> > > > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > > > +};
> > > > > > > > > +
> > > > > > > > >  /**
> > > > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > > > >   * @ops:             callback functions
> > > > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > > > >       if (!ctx)
> > > > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > > > >
> > > > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > > > +             .srv_ctx        = ctx,
> > > > > > > > > +             .port           = port,
> > > > > > > > > +     };
> > > > > > > > > +
> > > > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > > > >       if (err) {
> > > > > > > > >               free_srv_ctx(ctx);
> > > > > > > > >               return ERR_PTR(err);
> > > > > > > > >       }
> > > > > > > > >
> > > > > > > > > +     /*
> > > > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > > > +      */
> > > > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > > > +             free_srv_ctx(ctx);
> > > > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > > > +     }
> > > > > > > >
> > > > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > > > that ->add won't be called again for another device.
> > > > > > > >
> > > > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > > > >
> > > > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > > > will return after registering the client, but without calling ->add().
> > > > > > > Which means, ib_dev_count would be 0.
> > > > > >
> > > > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > > > to release.
> > > > >
> > > > > True, but we have to send a failure back to the caller of
> > > > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > > > rnbd-srv's function rnbd_srv_init_module().
> > > > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > > > fails, meaning rtrs_srv_rdma_init() had failed.
> > > > >
> > > > > Even if we are talking in generic terms, any module calling the
> > > > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > > > open failed or succeeded right?
> > > > I think Leon is right, any success of call to ->add, we have something
> > > > to rtrs_srv_rdma_init,
> > > > we can consider rtrs_srv_open is success instead of an error.
> > >
> > > I don't think Leon is right. In case when all modules are built in,
> > > add is not called in place when ib_client_register is called (cause
> > > there are no registered devices and add_client_context will be called
> > > at some later point in time from enable_device_and_get instead). So we
> > > have two cases: built in and not built in. Then there is a possibility
> > > that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> > > fail. Particularly we need to separate the case where add hasn't been
> > > called at all yet (i.e. modules are built in, we just need to wait
> > > until add gets called and then can start listening) and the case where
> > > it did get called but failed (i.e. modules are not built in it got
> > > called but failed - in that case we need to refuse to load module and
> > > return error since we can't start listening).
> > > The latter case is indicated by ib_dev_count = -1. I think it would
> > > make code easier to read if instead of setting ib_dev_count to -1 and
> > > explicitly checking whether it's below 0, one would introduce an
> > > additional variable for the error code returned by rtrs_srv_rdma_init
> > > and check it instead.
> >
> > Sorry, but it is very hard to read this block without indentations.
>
> Do you mean like empty lines between text paragraphs?

Yes

>
> >
> > Anyway, as I said there should be no difference in behaviour between no
> > devices and first device failed to initialize.
>
> If we load the server module and fail to initialize the device, we
> return an error to the user (insmod fails). We could just log
> something into dmesg and stay there doing nothing. Is that what you
> suggest? Then one would also probably want to have a way to say to the
> server it should try to init again (currently one just tries insmod
> again)...

First, we need to agree that "failure to initialize" is equal to the
situation where are no ib devices in the system.

Second, once device is registered, it needs to appear in relevant
sys/class with symlink to it. In similar way to uverbs:

[leonro@vm ~]$ ls -l /sys/class/infiniband_verbs/uverbs0/
....
lrwxrwxrwx 1 root root    0 Aug 12 05:40 device -> ../../../0000:00:09.0

So users will see if server succeeded to initialize or not.

Third, retrigger ib_device and rtrs will try to reconnect, no need to
call insmod again.

Thanks

>
> Thank you,
> Danil
>
> >
> > Thanks
> >
> > >
> > > >
> > > > Thanks Leon for catching this.
> > > >
> > > > Regards!

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Wed, Aug 12, 2020 at 11:18 AM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 11, 2020 at 02:32:38PM +0200, Danil Kipnis wrote:
> > On Tue, Aug 11, 2020 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> > > > On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> > > > >
> > > > > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > > > > <haris.iqbal@cloud.ionos.com> wrote:
> > > > > >
> > > > > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > >
> > > > > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > > >
> > > > > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > > > >
> > > > > > > > > > Call trace below,
> > > > > > > > > >
> > > > > > > > > > [    1.904782] Call Trace:
> > > > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > > > >
> > > > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > > > which is not a preferred practice.
> > > > > > > > > >
> > > > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > > > >
> > > > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > > > ---
> > > > > > > > > > Change in v2:
> > > > > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > > > > >         Change according to kernel coding style
> > > > > > > > > >
> > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > >  #include "rtrs-srv.h"
> > > > > > > > > >  #include "rtrs-log.h"
> > > > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > > > >
> > > > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > > > >
> > > > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > > > >       kfree(ctx);
> > > > > > > > > >  }
> > > > > > > > > >
> > > > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > > > +{
> > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > +     int ret;
> > > > > > > > > > +
> > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > +             goto out;
> > > > > > > > > > +
> > > > > > > > > > +     /*
> > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > > > +      * only once
> > > > > > > > > > +      */
> > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > > > +     if (ret) {
> > > > > > > > > > +             /*
> > > > > > > > > > +              * We errored out here.
> > > > > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > > > > +              */
> > > > > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > > > > +             return ret;
> > > > > > > > > > +     }
> > > > > > > > > > +
> > > > > > > > > > +out:
> > > > > > > > > > +     /*
> > > > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > > > +      */
> > > > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > > > +
> > > > > > > > > > +     return 0;
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > > > > +{
> > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > +
> > > > > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > > > > +
> > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > +             return;
> > > > > > > > > > +
> > > > > > > > > > +     /*
> > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > > > > +      * only once, when the last device is removed
> > > > > > > > > > +      */
> > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > > > > +     .name   = "rtrs_server",
> > > > > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > > > > +};
> > > > > > > > > > +
> > > > > > > > > >  /**
> > > > > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > > > > >   * @ops:             callback functions
> > > > > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > > > > >       if (!ctx)
> > > > > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > > > > >
> > > > > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > > > > +             .srv_ctx        = ctx,
> > > > > > > > > > +             .port           = port,
> > > > > > > > > > +     };
> > > > > > > > > > +
> > > > > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > > > > >       if (err) {
> > > > > > > > > >               free_srv_ctx(ctx);
> > > > > > > > > >               return ERR_PTR(err);
> > > > > > > > > >       }
> > > > > > > > > >
> > > > > > > > > > +     /*
> > > > > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > > > > +      */
> > > > > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > > > > +             free_srv_ctx(ctx);
> > > > > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > > > > +     }
> > > > > > > > >
> > > > > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > > > > that ->add won't be called again for another device.
> > > > > > > > >
> > > > > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > > > > >
> > > > > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > > > > will return after registering the client, but without calling ->add().
> > > > > > > > Which means, ib_dev_count would be 0.
> > > > > > >
> > > > > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > > > > to release.
> > > > > >
> > > > > > True, but we have to send a failure back to the caller of
> > > > > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > > > > rnbd-srv's function rnbd_srv_init_module().
> > > > > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > > > > fails, meaning rtrs_srv_rdma_init() had failed.
> > > > > >
> > > > > > Even if we are talking in generic terms, any module calling the
> > > > > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > > > > open failed or succeeded right?
> > > > > I think Leon is right, any success of call to ->add, we have something
> > > > > to rtrs_srv_rdma_init,
> > > > > we can consider rtrs_srv_open is success instead of an error.
> > > >
> > > > I don't think Leon is right. In case when all modules are built in,
> > > > add is not called in place when ib_client_register is called (cause
> > > > there are no registered devices and add_client_context will be called
> > > > at some later point in time from enable_device_and_get instead). So we
> > > > have two cases: built in and not built in. Then there is a possibility
> > > > that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> > > > fail. Particularly we need to separate the case where add hasn't been
> > > > called at all yet (i.e. modules are built in, we just need to wait
> > > > until add gets called and then can start listening) and the case where
> > > > it did get called but failed (i.e. modules are not built in it got
> > > > called but failed - in that case we need to refuse to load module and
> > > > return error since we can't start listening).
> > > > The latter case is indicated by ib_dev_count = -1. I think it would
> > > > make code easier to read if instead of setting ib_dev_count to -1 and
> > > > explicitly checking whether it's below 0, one would introduce an
> > > > additional variable for the error code returned by rtrs_srv_rdma_init
> > > > and check it instead.
> > >
> > > Sorry, but it is very hard to read this block without indentations.
> >
> > Do you mean like empty lines between text paragraphs?
>
> Yes
>
> >
> > >
> > > Anyway, as I said there should be no difference in behaviour between no
> > > devices and first device failed to initialize.
> >
> > If we load the server module and fail to initialize the device, we
> > return an error to the user (insmod fails). We could just log
> > something into dmesg and stay there doing nothing. Is that what you
> > suggest? Then one would also probably want to have a way to say to the
> > server it should try to init again (currently one just tries insmod
> > again)...
>
> First, we need to agree that "failure to initialize" is equal to the
> situation where are no ib devices in the system.
>
> Second, once device is registered, it needs to appear in relevant
> sys/class with symlink to it. In similar way to uverbs:
>
> [leonro@vm ~]$ ls -l /sys/class/infiniband_verbs/uverbs0/
> ....
> lrwxrwxrwx 1 root root    0 Aug 12 05:40 device -> ../../../0000:00:09.0
>
> So users will see if server succeeded to initialize or not.
>
> Third, retrigger ib_device and rtrs will try to reconnect, no need to
> call insmod again.

Could you elaborate a little on what you mean when you say "retrigger
ib_device", and how to do it.

I looked around and the only thing I found are the below options, and
they do not trigger the add functions. I am surely missing something
here.

$ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/rescan
$ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/reset

>
> Thanks
>
> >
> > Thank you,
> > Danil
> >
> > >
> > > Thanks
> > >
> > > >
> > > > >
> > > > > Thanks Leon for catching this.
> > > > >
> > > > > Regards!



-- 

Regards
-Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Leon Romanovsky]

On Tue, Aug 18, 2020 at 04:34:29AM +0530, Haris Iqbal wrote:
> On Wed, Aug 12, 2020 at 11:18 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, Aug 11, 2020 at 02:32:38PM +0200, Danil Kipnis wrote:
> > > On Tue, Aug 11, 2020 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> > > > > On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> > > > > >
> > > > > > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > > > > > <haris.iqbal@cloud.ionos.com> wrote:
> > > > > > >
> > > > > > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > > > >
> > > > > > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > > > > >
> > > > > > > > > > > Call trace below,
> > > > > > > > > > >
> > > > > > > > > > > [    1.904782] Call Trace:
> > > > > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > > > > >
> > > > > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > > > > which is not a preferred practice.
> > > > > > > > > > >
> > > > > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > > > > >
> > > > > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > > > > ---
> > > > > > > > > > > Change in v2:
> > > > > > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > > > > > >         Change according to kernel coding style
> > > > > > > > > > >
> > > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > > >  #include "rtrs-srv.h"
> > > > > > > > > > >  #include "rtrs-log.h"
> > > > > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > > > > >
> > > > > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > > > > >
> > > > > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > > > > >       kfree(ctx);
> > > > > > > > > > >  }
> > > > > > > > > > >
> > > > > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > > > > +{
> > > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > > +     int ret;
> > > > > > > > > > > +
> > > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > > +             goto out;
> > > > > > > > > > > +
> > > > > > > > > > > +     /*
> > > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > > > > +      * only once
> > > > > > > > > > > +      */
> > > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > > > > +     if (ret) {
> > > > > > > > > > > +             /*
> > > > > > > > > > > +              * We errored out here.
> > > > > > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > > > > > +              */
> > > > > > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > > > > > +             return ret;
> > > > > > > > > > > +     }
> > > > > > > > > > > +
> > > > > > > > > > > +out:
> > > > > > > > > > > +     /*
> > > > > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > > > > +      */
> > > > > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > > > > +
> > > > > > > > > > > +     return 0;
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > > > > > +{
> > > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > > +
> > > > > > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > > > > > +
> > > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > > +             return;
> > > > > > > > > > > +
> > > > > > > > > > > +     /*
> > > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > > > > > +      * only once, when the last device is removed
> > > > > > > > > > > +      */
> > > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > > > > > +     .name   = "rtrs_server",
> > > > > > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > > > > > +};
> > > > > > > > > > > +
> > > > > > > > > > >  /**
> > > > > > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > > > > > >   * @ops:             callback functions
> > > > > > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > > > > > >       if (!ctx)
> > > > > > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > > > > > >
> > > > > > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > > > > > +             .srv_ctx        = ctx,
> > > > > > > > > > > +             .port           = port,
> > > > > > > > > > > +     };
> > > > > > > > > > > +
> > > > > > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > > > > > >       if (err) {
> > > > > > > > > > >               free_srv_ctx(ctx);
> > > > > > > > > > >               return ERR_PTR(err);
> > > > > > > > > > >       }
> > > > > > > > > > >
> > > > > > > > > > > +     /*
> > > > > > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > > > > > +      */
> > > > > > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > > > > > +             free_srv_ctx(ctx);
> > > > > > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > > > > > +     }
> > > > > > > > > >
> > > > > > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > > > > > that ->add won't be called again for another device.
> > > > > > > > > >
> > > > > > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > > > > > >
> > > > > > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > > > > > will return after registering the client, but without calling ->add().
> > > > > > > > > Which means, ib_dev_count would be 0.
> > > > > > > >
> > > > > > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > > > > > to release.
> > > > > > >
> > > > > > > True, but we have to send a failure back to the caller of
> > > > > > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > > > > > rnbd-srv's function rnbd_srv_init_module().
> > > > > > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > > > > > fails, meaning rtrs_srv_rdma_init() had failed.
> > > > > > >
> > > > > > > Even if we are talking in generic terms, any module calling the
> > > > > > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > > > > > open failed or succeeded right?
> > > > > > I think Leon is right, any success of call to ->add, we have something
> > > > > > to rtrs_srv_rdma_init,
> > > > > > we can consider rtrs_srv_open is success instead of an error.
> > > > >
> > > > > I don't think Leon is right. In case when all modules are built in,
> > > > > add is not called in place when ib_client_register is called (cause
> > > > > there are no registered devices and add_client_context will be called
> > > > > at some later point in time from enable_device_and_get instead). So we
> > > > > have two cases: built in and not built in. Then there is a possibility
> > > > > that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> > > > > fail. Particularly we need to separate the case where add hasn't been
> > > > > called at all yet (i.e. modules are built in, we just need to wait
> > > > > until add gets called and then can start listening) and the case where
> > > > > it did get called but failed (i.e. modules are not built in it got
> > > > > called but failed - in that case we need to refuse to load module and
> > > > > return error since we can't start listening).
> > > > > The latter case is indicated by ib_dev_count = -1. I think it would
> > > > > make code easier to read if instead of setting ib_dev_count to -1 and
> > > > > explicitly checking whether it's below 0, one would introduce an
> > > > > additional variable for the error code returned by rtrs_srv_rdma_init
> > > > > and check it instead.
> > > >
> > > > Sorry, but it is very hard to read this block without indentations.
> > >
> > > Do you mean like empty lines between text paragraphs?
> >
> > Yes
> >
> > >
> > > >
> > > > Anyway, as I said there should be no difference in behaviour between no
> > > > devices and first device failed to initialize.
> > >
> > > If we load the server module and fail to initialize the device, we
> > > return an error to the user (insmod fails). We could just log
> > > something into dmesg and stay there doing nothing. Is that what you
> > > suggest? Then one would also probably want to have a way to say to the
> > > server it should try to init again (currently one just tries insmod
> > > again)...
> >
> > First, we need to agree that "failure to initialize" is equal to the
> > situation where are no ib devices in the system.
> >
> > Second, once device is registered, it needs to appear in relevant
> > sys/class with symlink to it. In similar way to uverbs:
> >
> > [leonro@vm ~]$ ls -l /sys/class/infiniband_verbs/uverbs0/
> > ....
> > lrwxrwxrwx 1 root root    0 Aug 12 05:40 device -> ../../../0000:00:09.0
> >
> > So users will see if server succeeded to initialize or not.
> >
> > Third, retrigger ib_device and rtrs will try to reconnect, no need to
> > call insmod again.
>
> Could you elaborate a little on what you mean when you say "retrigger
> ib_device", and how to do it.
>
> I looked around and the only thing I found are the below options, and
> they do not trigger the add functions. I am surely missing something
> here.
>
> $ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/rescan
> $ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/reset

This will do the trick:
modprobe -r mlx5_ib
modprobe mlx5_ib

Thanks

>
> >
> > Thanks
> >
> > >
> > > Thank you,
> > > Danil
> > >
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > >
> > > > > > Thanks Leon for catching this.
> > > > > >
> > > > > > Regards!
>
>
>
> --
>
> Regards
> -Haris

Re: [PATCH v2] RDMA/rtrs-srv: Incorporate ib_register_client into rtrs server init

[Haris Iqbal]

On Tue, Aug 18, 2020 at 1:21 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, Aug 18, 2020 at 04:34:29AM +0530, Haris Iqbal wrote:
> > On Wed, Aug 12, 2020 at 11:18 AM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, Aug 11, 2020 at 02:32:38PM +0200, Danil Kipnis wrote:
> > > > On Tue, Aug 11, 2020 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > >
> > > > > On Tue, Aug 11, 2020 at 01:44:58PM +0200, Danil Kipnis wrote:
> > > > > > On Tue, Aug 11, 2020 at 1:13 PM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> > > > > > >
> > > > > > > On Tue, Aug 11, 2020 at 12:53 PM Haris Iqbal
> > > > > > > <haris.iqbal@cloud.ionos.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Aug 11, 2020 at 4:17 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Aug 11, 2020 at 02:27:12PM +0530, Haris Iqbal wrote:
> > > > > > > > > > On Tue, Aug 11, 2020 at 2:15 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Aug 10, 2020 at 05:20:49PM +0530, Md Haris Iqbal wrote:
> > > > > > > > > > > > The rnbd_server module's communication manager (cm) initialization depends
> > > > > > > > > > > > on the registration of the "network namespace subsystem" of the RDMA CM
> > > > > > > > > > > > agent module. As such, when the kernel is configured to load the
> > > > > > > > > > > > rnbd_server and the RDMA cma module during initialization; and if the
> > > > > > > > > > > > rnbd_server module is initialized before RDMA cma module, a null ptr
> > > > > > > > > > > > dereference occurs during the RDMA bind operation.
> > > > > > > > > > > >
> > > > > > > > > > > > Call trace below,
> > > > > > > > > > > >
> > > > > > > > > > > > [    1.904782] Call Trace:
> > > > > > > > > > > > [    1.904782]  ? xas_load+0xd/0x80
> > > > > > > > > > > > [    1.904782]  xa_load+0x47/0x80
> > > > > > > > > > > > [    1.904782]  cma_ps_find+0x44/0x70
> > > > > > > > > > > > [    1.904782]  rdma_bind_addr+0x782/0x8b0
> > > > > > > > > > > > [    1.904782]  ? get_random_bytes+0x35/0x40
> > > > > > > > > > > > [    1.904782]  rtrs_srv_cm_init+0x50/0x80
> > > > > > > > > > > > [    1.904782]  rtrs_srv_open+0x102/0x180
> > > > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > > > [    1.904782]  rnbd_srv_init_module+0x34/0x84
> > > > > > > > > > > > [    1.904782]  ? rnbd_client_init+0x6e/0x6e
> > > > > > > > > > > > [    1.904782]  do_one_initcall+0x4a/0x200
> > > > > > > > > > > > [    1.904782]  kernel_init_freeable+0x1f1/0x26e
> > > > > > > > > > > > [    1.904782]  ? rest_init+0xb0/0xb0
> > > > > > > > > > > > [    1.904782]  kernel_init+0xe/0x100
> > > > > > > > > > > > [    1.904782]  ret_from_fork+0x22/0x30
> > > > > > > > > > > > [    1.904782] Modules linked in:
> > > > > > > > > > > > [    1.904782] CR2: 0000000000000015
> > > > > > > > > > > > [    1.904782] ---[ end trace c42df88d6c7b0a48 ]---
> > > > > > > > > > > >
> > > > > > > > > > > > All this happens cause the cm init is in the call chain of the module init,
> > > > > > > > > > > > which is not a preferred practice.
> > > > > > > > > > > >
> > > > > > > > > > > > So remove the call to rdma_create_id() from the module init call chain.
> > > > > > > > > > > > Instead register rtrs-srv as an ib client, which makes sure that the
> > > > > > > > > > > > rdma_create_id() is called only when an ib device is added.
> > > > > > > > > > > >
> > > > > > > > > > > > Fixes: 9cb837480424 ("RDMA/rtrs: server: main functionality")
> > > > > > > > > > > > Reported-by: kernel test robot <rong.a.chen@intel.com>
> > > > > > > > > > > > Signed-off-by: Md Haris Iqbal <haris.iqbal@cloud.ionos.com>
> > > > > > > > > > > > ---
> > > > > > > > > > > > Change in v2:
> > > > > > > > > > > >         Use only single variable to track number of IB devices and failure
> > > > > > > > > > > >         Change according to kernel coding style
> > > > > > > > > > > >
> > > > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.c | 79 +++++++++++++++++++++++++-
> > > > > > > > > > > >  drivers/infiniband/ulp/rtrs/rtrs-srv.h |  6 ++
> > > > > > > > > > > >  2 files changed, 82 insertions(+), 3 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > > index 0d9241f5d9e6..69a37ce73b0c 100644
> > > > > > > > > > > > --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > > +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c
> > > > > > > > > > > > @@ -16,6 +16,7 @@
> > > > > > > > > > > >  #include "rtrs-srv.h"
> > > > > > > > > > > >  #include "rtrs-log.h"
> > > > > > > > > > > >  #include <rdma/ib_cm.h>
> > > > > > > > > > > > +#include <rdma/ib_verbs.h>
> > > > > > > > > > > >
> > > > > > > > > > > >  MODULE_DESCRIPTION("RDMA Transport Server");
> > > > > > > > > > > >  MODULE_LICENSE("GPL");
> > > > > > > > > > > > @@ -31,6 +32,7 @@ MODULE_LICENSE("GPL");
> > > > > > > > > > > >  static struct rtrs_rdma_dev_pd dev_pd;
> > > > > > > > > > > >  static mempool_t *chunk_pool;
> > > > > > > > > > > >  struct class *rtrs_dev_class;
> > > > > > > > > > > > +static struct rtrs_srv_ib_ctx ib_ctx;
> > > > > > > > > > > >
> > > > > > > > > > > >  static int __read_mostly max_chunk_size = DEFAULT_MAX_CHUNK_SIZE;
> > > > > > > > > > > >  static int __read_mostly sess_queue_depth = DEFAULT_SESS_QUEUE_DEPTH;
> > > > > > > > > > > > @@ -2033,6 +2035,64 @@ static void free_srv_ctx(struct rtrs_srv_ctx *ctx)
> > > > > > > > > > > >       kfree(ctx);
> > > > > > > > > > > >  }
> > > > > > > > > > > >
> > > > > > > > > > > > +static int rtrs_srv_add_one(struct ib_device *device)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > > > +     int ret;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > > > +             goto out;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     /*
> > > > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will create them
> > > > > > > > > > > > +      * only once
> > > > > > > > > > > > +      */
> > > > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > > > +     ret = rtrs_srv_rdma_init(ctx, ib_ctx.port);
> > > > > > > > > > > > +     if (ret) {
> > > > > > > > > > > > +             /*
> > > > > > > > > > > > +              * We errored out here.
> > > > > > > > > > > > +              * According to the ib code, if we encounter an error here then the
> > > > > > > > > > > > +              * error code is ignored, and no more calls to our ops are made.
> > > > > > > > > > > > +              */
> > > > > > > > > > > > +             pr_err("Failed to initialize RDMA connection");
> > > > > > > > > > > > +             ib_ctx.ib_dev_count = -1;
> > > > > > > > > > > > +             return ret;
> > > > > > > > > > > > +     }
> > > > > > > > > > > > +
> > > > > > > > > > > > +out:
> > > > > > > > > > > > +     /*
> > > > > > > > > > > > +      * Keep a track on the number of ib devices added
> > > > > > > > > > > > +      */
> > > > > > > > > > > > +     ib_ctx.ib_dev_count++;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     return 0;
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +static void rtrs_srv_remove_one(struct ib_device *device, void *client_data)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +     struct rtrs_srv_ctx *ctx;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     ib_ctx.ib_dev_count--;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     if (ib_ctx.ib_dev_count)
> > > > > > > > > > > > +             return;
> > > > > > > > > > > > +
> > > > > > > > > > > > +     /*
> > > > > > > > > > > > +      * Since our CM IDs are NOT bound to any ib device we will remove them
> > > > > > > > > > > > +      * only once, when the last device is removed
> > > > > > > > > > > > +      */
> > > > > > > > > > > > +     ctx = ib_ctx.srv_ctx;
> > > > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ip);
> > > > > > > > > > > > +     rdma_destroy_id(ctx->cm_id_ib);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > > +static struct ib_client rtrs_srv_client = {
> > > > > > > > > > > > +     .name   = "rtrs_server",
> > > > > > > > > > > > +     .add    = rtrs_srv_add_one,
> > > > > > > > > > > > +     .remove = rtrs_srv_remove_one
> > > > > > > > > > > > +};
> > > > > > > > > > > > +
> > > > > > > > > > > >  /**
> > > > > > > > > > > >   * rtrs_srv_open() - open RTRS server context
> > > > > > > > > > > >   * @ops:             callback functions
> > > > > > > > > > > > @@ -2051,12 +2111,26 @@ struct rtrs_srv_ctx *rtrs_srv_open(struct rtrs_srv_ops *ops, u16 port)
> > > > > > > > > > > >       if (!ctx)
> > > > > > > > > > > >               return ERR_PTR(-ENOMEM);
> > > > > > > > > > > >
> > > > > > > > > > > > -     err = rtrs_srv_rdma_init(ctx, port);
> > > > > > > > > > > > +     ib_ctx = (struct rtrs_srv_ib_ctx) {
> > > > > > > > > > > > +             .srv_ctx        = ctx,
> > > > > > > > > > > > +             .port           = port,
> > > > > > > > > > > > +     };
> > > > > > > > > > > > +
> > > > > > > > > > > > +     err = ib_register_client(&rtrs_srv_client);
> > > > > > > > > > > >       if (err) {
> > > > > > > > > > > >               free_srv_ctx(ctx);
> > > > > > > > > > > >               return ERR_PTR(err);
> > > > > > > > > > > >       }
> > > > > > > > > > > >
> > > > > > > > > > > > +     /*
> > > > > > > > > > > > +      * Since ib_register_client does not propagate the device add error
> > > > > > > > > > > > +      * we check if .add was called and the RDMA connection init failed
> > > > > > > > > > > > +      */
> > > > > > > > > > > > +     if (ib_ctx.ib_dev_count < 0) {
> > > > > > > > > > > > +             free_srv_ctx(ctx);
> > > > > > > > > > > > +             return ERR_PTR(-ENODEV);
> > > > > > > > > > > > +     }
> > > > > > > > > > >
> > > > > > > > > > > I afraid that you overcomplicated here, ib_register_client() doesn't
> > > > > > > > > > > return error if ->add() for specific device failed, it doesn't mean
> > > > > > > > > > > that ->add won't be called again for another device.
> > > > > > > > > > >
> > > > > > > > > > > So you don't need to use ib_dev_count == -1, just keep it to be 0 and
> > > > > > > > > > > leave to  rtrs_srv_close() to free srv_ctx.
> > > > > > > > > >
> > > > > > > > > > Leaving it 0 when there is an error is not gonna work. Since when the
> > > > > > > > > > modules are all built-in, a call to ib_register_client() will not
> > > > > > > > > > result in a call to ->add() then and there. So ib_register_client()
> > > > > > > > > > will return after registering the client, but without calling ->add().
> > > > > > > > > > Which means, ib_dev_count would be 0.
> > > > > > > > >
> > > > > > > > > If ib_dev_count == 0 => rtrs_srv_rdma_init() didn't success => nothing
> > > > > > > > > to release.
> > > > > > > >
> > > > > > > > True, but we have to send a failure back to the caller of
> > > > > > > > "rtrs_srv_open()" (and user of this ulp); which in our case is
> > > > > > > > rnbd-srv's function rnbd_srv_init_module().
> > > > > > > > In our case, the rnbd-drv module init would fail if "rtrs_srv_open()"
> > > > > > > > fails, meaning rtrs_srv_rdma_init() had failed.
> > > > > > > >
> > > > > > > > Even if we are talking in generic terms, any module calling the
> > > > > > > > "rtrs_srv_open()" of the rtrs ulp, would want to know if the server
> > > > > > > > open failed or succeeded right?
> > > > > > > I think Leon is right, any success of call to ->add, we have something
> > > > > > > to rtrs_srv_rdma_init,
> > > > > > > we can consider rtrs_srv_open is success instead of an error.
> > > > > >
> > > > > > I don't think Leon is right. In case when all modules are built in,
> > > > > > add is not called in place when ib_client_register is called (cause
> > > > > > there are no registered devices and add_client_context will be called
> > > > > > at some later point in time from enable_device_and_get instead). So we
> > > > > > have two cases: built in and not built in. Then there is a possibility
> > > > > > that rtrs_srv_rdma_init fails (create_id, bind_addr, etc.) or doesn't
> > > > > > fail. Particularly we need to separate the case where add hasn't been
> > > > > > called at all yet (i.e. modules are built in, we just need to wait
> > > > > > until add gets called and then can start listening) and the case where
> > > > > > it did get called but failed (i.e. modules are not built in it got
> > > > > > called but failed - in that case we need to refuse to load module and
> > > > > > return error since we can't start listening).
> > > > > > The latter case is indicated by ib_dev_count = -1. I think it would
> > > > > > make code easier to read if instead of setting ib_dev_count to -1 and
> > > > > > explicitly checking whether it's below 0, one would introduce an
> > > > > > additional variable for the error code returned by rtrs_srv_rdma_init
> > > > > > and check it instead.
> > > > >
> > > > > Sorry, but it is very hard to read this block without indentations.
> > > >
> > > > Do you mean like empty lines between text paragraphs?
> > >
> > > Yes
> > >
> > > >
> > > > >
> > > > > Anyway, as I said there should be no difference in behaviour between no
> > > > > devices and first device failed to initialize.
> > > >
> > > > If we load the server module and fail to initialize the device, we
> > > > return an error to the user (insmod fails). We could just log
> > > > something into dmesg and stay there doing nothing. Is that what you
> > > > suggest? Then one would also probably want to have a way to say to the
> > > > server it should try to init again (currently one just tries insmod
> > > > again)...
> > >
> > > First, we need to agree that "failure to initialize" is equal to the
> > > situation where are no ib devices in the system.
> > >
> > > Second, once device is registered, it needs to appear in relevant
> > > sys/class with symlink to it. In similar way to uverbs:
> > >
> > > [leonro@vm ~]$ ls -l /sys/class/infiniband_verbs/uverbs0/
> > > ....
> > > lrwxrwxrwx 1 root root    0 Aug 12 05:40 device -> ../../../0000:00:09.0
> > >
> > > So users will see if server succeeded to initialize or not.
> > >
> > > Third, retrigger ib_device and rtrs will try to reconnect, no need to
> > > call insmod again.
> >
> > Could you elaborate a little on what you mean when you say "retrigger
> > ib_device", and how to do it.
> >
> > I looked around and the only thing I found are the below options, and
> > they do not trigger the add functions. I am surely missing something
> > here.
> >
> > $ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/rescan
> > $ echo 1 > /sys/class/infiniband_verbs/uverbs0/device/reset
>
> This will do the trick:
> modprobe -r mlx5_ib
> modprobe mlx5_ib

Thanks Leon, this is working.
Will send the updated patch.

>
> Thanks
>
> >
> > >
> > > Thanks
> > >
> > > >
> > > > Thank you,
> > > > Danil
> > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > >
> > > > > > >
> > > > > > > Thanks Leon for catching this.
> > > > > > >
> > > > > > > Regards!
> >
> >
> >
> > --
> >
> > Regards
> > -Haris



-- 

Regards
-Haris