* [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create()
@ 2020-08-12 11:14 Kamal Heib
2020-08-15 6:58 ` Zhu Yanjun
0 siblings, 1 reply; 11+ messages in thread
From: Kamal Heib @ 2020-08-12 11:14 UTC (permalink / raw)
To: linux-rdma; +Cc: Doug Ledford, Jason Gunthorpe, Zhu Yanjun, Kamal Heib
To avoid the following kernel panic when calling kmem_cache_create()
with a NULL pointer from pool_cache(), move the rxe_cache_init() to the
context of device creation.
BUG: unable to handle kernel NULL pointer dereference at 000000000000000b
PGD 0 P4D 0
Oops: 0000 [#1] SMP NOPTI
CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1
Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018
RIP: 0010:kmem_cache_alloc+0xd1/0x1b0
Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6
RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005
RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000
RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0
R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0
R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000
FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0
Call Trace:
rxe_alloc+0xc8/0x160 [rdma_rxe]
rxe_get_dma_mr+0x25/0xb0 [rdma_rxe]
__ib_alloc_pd+0xcb/0x160 [ib_core]
ib_mad_init_device+0x296/0x8b0 [ib_core]
add_client_context+0x11a/0x160 [ib_core]
enable_device_and_get+0xdc/0x1d0 [ib_core]
ib_register_device+0x572/0x6b0 [ib_core]
? crypto_create_tfm+0x32/0xe0
? crypto_create_tfm+0x7a/0xe0
? crypto_alloc_tfm+0x58/0xf0
rxe_register_device+0x19d/0x1c0 [rdma_rxe]
rxe_net_add+0x3d/0x70 [rdma_rxe]
? dev_get_by_name_rcu+0x73/0x90
rxe_param_set_add+0xaf/0xc0 [rdma_rxe]
parse_args+0x179/0x370
? ref_module+0x1b0/0x1b0
load_module+0x135e/0x17e0
? ref_module+0x1b0/0x1b0
? __do_sys_init_module+0x13b/0x180
__do_sys_init_module+0x13b/0x180
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x65/0xca
RIP: 0033:0x7f9137ed296e
Fixes: 8700e3e7c485 ("Soft RoCE driver")
Signed-off-by: Kamal Heib <kamalheib1@gmail.com>
---
drivers/infiniband/sw/rxe/rxe.c | 14 +++++++-------
drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++
drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++
3 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c
index 5642eefb4ba1..60d5086dd34d 100644
--- a/drivers/infiniband/sw/rxe/rxe.c
+++ b/drivers/infiniband/sw/rxe/rxe.c
@@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)
goto err;
}
+ /* initialize slab caches for managed objects */
+ err = rxe_cache_init();
+ if (err) {
+ pr_err("unable to init object pools\n");
+ goto err;
+ }
+
err = rxe_net_add(ibdev_name, ndev);
if (err) {
pr_err("failed to add %s\n", ndev->name);
@@ -336,13 +343,6 @@ static int __init rxe_module_init(void)
{
int err;
- /* initialize slab caches for managed objects */
- err = rxe_cache_init();
- if (err) {
- pr_err("unable to init object pools\n");
- return err;
- }
-
err = rxe_net_init();
if (err)
return err;
diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c
index fbcbac52290b..06c6d1f835b7 100644
--- a/drivers/infiniband/sw/rxe/rxe_pool.c
+++ b/drivers/infiniband/sw/rxe/rxe_pool.c
@@ -139,6 +139,9 @@ int rxe_cache_init(void)
for (i = 0; i < RXE_NUM_TYPES; i++) {
type = &rxe_type_info[i];
size = ALIGN(type->size, RXE_POOL_ALIGN);
+ if (type->cache)
+ continue;
+
if (!(type->flags & RXE_POOL_NO_ALLOC)) {
type->cache =
kmem_cache_create(type->name, size,
diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c
index ccda5f5a3bc0..d0af48ba0110 100644
--- a/drivers/infiniband/sw/rxe/rxe_sysfs.c
+++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c
@@ -81,6 +81,13 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp)
goto err;
}
+ /* initialize slab caches for managed objects */
+ err = rxe_cache_init();
+ if (err) {
+ pr_err("unable to init object pools\n");
+ goto err;
+ }
+
err = rxe_net_add("rxe%d", ndev);
if (err) {
pr_err("failed to add %s\n", intf);
--
2.25.4
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-12 11:14 [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() Kamal Heib @ 2020-08-15 6:58 ` Zhu Yanjun 2020-08-16 22:12 ` Kamal Heib 0 siblings, 1 reply; 11+ messages in thread From: Zhu Yanjun @ 2020-08-15 6:58 UTC (permalink / raw) To: Kamal Heib, linux-rdma; +Cc: Doug Ledford, Jason Gunthorpe On 8/12/2020 7:14 PM, Kamal Heib wrote: > To avoid the following kernel panic when calling kmem_cache_create() > with a NULL pointer from pool_cache(), What is the root cause of this kernel panic? Zhu Yanjun > move the rxe_cache_init() to the > context of device creation. > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > PGD 0 P4D 0 > Oops: 0000 [#1] SMP NOPTI > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > Call Trace: > rxe_alloc+0xc8/0x160 [rdma_rxe] > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > __ib_alloc_pd+0xcb/0x160 [ib_core] > ib_mad_init_device+0x296/0x8b0 [ib_core] > add_client_context+0x11a/0x160 [ib_core] > enable_device_and_get+0xdc/0x1d0 [ib_core] > ib_register_device+0x572/0x6b0 [ib_core] > ? crypto_create_tfm+0x32/0xe0 > ? crypto_create_tfm+0x7a/0xe0 > ? crypto_alloc_tfm+0x58/0xf0 > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > rxe_net_add+0x3d/0x70 [rdma_rxe] > ? dev_get_by_name_rcu+0x73/0x90 > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > parse_args+0x179/0x370 > ? ref_module+0x1b0/0x1b0 > load_module+0x135e/0x17e0 > ? ref_module+0x1b0/0x1b0 > ? __do_sys_init_module+0x13b/0x180 > __do_sys_init_module+0x13b/0x180 > do_syscall_64+0x5b/0x1a0 > entry_SYSCALL_64_after_hwframe+0x65/0xca > RIP: 0033:0x7f9137ed296e > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > --- > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > 3 files changed, 17 insertions(+), 7 deletions(-) > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > index 5642eefb4ba1..60d5086dd34d 100644 > --- a/drivers/infiniband/sw/rxe/rxe.c > +++ b/drivers/infiniband/sw/rxe/rxe.c > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > goto err; > } > > + /* initialize slab caches for managed objects */ > + err = rxe_cache_init(); > + if (err) { > + pr_err("unable to init object pools\n"); > + goto err; > + } > + > err = rxe_net_add(ibdev_name, ndev); > if (err) { > pr_err("failed to add %s\n", ndev->name); > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > { > int err; > > - /* initialize slab caches for managed objects */ > - err = rxe_cache_init(); > - if (err) { > - pr_err("unable to init object pools\n"); > - return err; > - } > - > err = rxe_net_init(); > if (err) > return err; > diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c > index fbcbac52290b..06c6d1f835b7 100644 > --- a/drivers/infiniband/sw/rxe/rxe_pool.c > +++ b/drivers/infiniband/sw/rxe/rxe_pool.c > @@ -139,6 +139,9 @@ int rxe_cache_init(void) > for (i = 0; i < RXE_NUM_TYPES; i++) { > type = &rxe_type_info[i]; > size = ALIGN(type->size, RXE_POOL_ALIGN); > + if (type->cache) > + continue; > + > if (!(type->flags & RXE_POOL_NO_ALLOC)) { > type->cache = > kmem_cache_create(type->name, size, > diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c > index ccda5f5a3bc0..d0af48ba0110 100644 > --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c > +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c > @@ -81,6 +81,13 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp) > goto err; > } > > + /* initialize slab caches for managed objects */ > + err = rxe_cache_init(); > + if (err) { > + pr_err("unable to init object pools\n"); > + goto err; > + } > + > err = rxe_net_add("rxe%d", ndev); > if (err) { > pr_err("failed to add %s\n", intf); ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-15 6:58 ` Zhu Yanjun @ 2020-08-16 22:12 ` Kamal Heib 2020-08-18 1:48 ` Zhu Yanjun 0 siblings, 1 reply; 11+ messages in thread From: Kamal Heib @ 2020-08-16 22:12 UTC (permalink / raw) To: Zhu Yanjun; +Cc: linux-rdma, Doug Ledford, Jason Gunthorpe On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > To avoid the following kernel panic when calling kmem_cache_create() > > with a NULL pointer from pool_cache(), > > What is the root cause of this kernel panic? > The kernel panic is triggered using the following command and it happen because the cache is not getting initialized. modprobe rdma_rxe add=eno1 Thanks, Kamal > Zhu Yanjun > > > move the rxe_cache_init() to the > > context of device creation. > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > PGD 0 P4D 0 > > Oops: 0000 [#1] SMP NOPTI > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > Call Trace: > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > add_client_context+0x11a/0x160 [ib_core] > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > ib_register_device+0x572/0x6b0 [ib_core] > > ? crypto_create_tfm+0x32/0xe0 > > ? crypto_create_tfm+0x7a/0xe0 > > ? crypto_alloc_tfm+0x58/0xf0 > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > ? dev_get_by_name_rcu+0x73/0x90 > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > parse_args+0x179/0x370 > > ? ref_module+0x1b0/0x1b0 > > load_module+0x135e/0x17e0 > > ? ref_module+0x1b0/0x1b0 > > ? __do_sys_init_module+0x13b/0x180 > > __do_sys_init_module+0x13b/0x180 > > do_syscall_64+0x5b/0x1a0 > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > RIP: 0033:0x7f9137ed296e > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > --- > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > index 5642eefb4ba1..60d5086dd34d 100644 > > --- a/drivers/infiniband/sw/rxe/rxe.c > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > goto err; > > } > > + /* initialize slab caches for managed objects */ > > + err = rxe_cache_init(); > > + if (err) { > > + pr_err("unable to init object pools\n"); > > + goto err; > > + } > > + > > err = rxe_net_add(ibdev_name, ndev); > > if (err) { > > pr_err("failed to add %s\n", ndev->name); > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > { > > int err; > > - /* initialize slab caches for managed objects */ > > - err = rxe_cache_init(); > > - if (err) { > > - pr_err("unable to init object pools\n"); > > - return err; > > - } > > - > > err = rxe_net_init(); > > if (err) > > return err; > > diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c > > index fbcbac52290b..06c6d1f835b7 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_pool.c > > +++ b/drivers/infiniband/sw/rxe/rxe_pool.c > > @@ -139,6 +139,9 @@ int rxe_cache_init(void) > > for (i = 0; i < RXE_NUM_TYPES; i++) { > > type = &rxe_type_info[i]; > > size = ALIGN(type->size, RXE_POOL_ALIGN); > > + if (type->cache) > > + continue; > > + > > if (!(type->flags & RXE_POOL_NO_ALLOC)) { > > type->cache = > > kmem_cache_create(type->name, size, > > diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c > > index ccda5f5a3bc0..d0af48ba0110 100644 > > --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c > > +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c > > @@ -81,6 +81,13 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp) > > goto err; > > } > > + /* initialize slab caches for managed objects */ > > + err = rxe_cache_init(); > > + if (err) { > > + pr_err("unable to init object pools\n"); > > + goto err; > > + } > > + > > err = rxe_net_add("rxe%d", ndev); > > if (err) { > > pr_err("failed to add %s\n", intf); > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-16 22:12 ` Kamal Heib @ 2020-08-18 1:48 ` Zhu Yanjun 2020-08-18 5:50 ` Kamal Heib 0 siblings, 1 reply; 11+ messages in thread From: Zhu Yanjun @ 2020-08-18 1:48 UTC (permalink / raw) To: Kamal Heib; +Cc: linux-rdma, Doug Ledford, Jason Gunthorpe On 8/17/2020 6:12 AM, Kamal Heib wrote: > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: >> On 8/12/2020 7:14 PM, Kamal Heib wrote: >>> To avoid the following kernel panic when calling kmem_cache_create() >>> with a NULL pointer from pool_cache(), >> What is the root cause of this kernel panic? >> > The kernel panic is triggered using the following command and it happen > because the cache is not getting initialized. > > modprobe rdma_rxe add=eno1 > > Thanks, > Kamal > >> Zhu Yanjun >> >>> move the rxe_cache_init() to the >>> context of device creation. >>> >>> BUG: unable to handle kernel NULL pointer dereference at 000000000000000b >>> PGD 0 P4D 0 >>> Oops: 0000 [#1] SMP NOPTI >>> CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 >>> Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 >>> RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 >>> Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 >>> RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 >>> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 >>> RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 >>> RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 >>> R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 >>> R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 >>> FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 >>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 >>> Call Trace: >>> rxe_alloc+0xc8/0x160 [rdma_rxe] >>> rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] >>> __ib_alloc_pd+0xcb/0x160 [ib_core] >>> ib_mad_init_device+0x296/0x8b0 [ib_core] >>> add_client_context+0x11a/0x160 [ib_core] >>> enable_device_and_get+0xdc/0x1d0 [ib_core] >>> ib_register_device+0x572/0x6b0 [ib_core] >>> ? crypto_create_tfm+0x32/0xe0 >>> ? crypto_create_tfm+0x7a/0xe0 >>> ? crypto_alloc_tfm+0x58/0xf0 >>> rxe_register_device+0x19d/0x1c0 [rdma_rxe] >>> rxe_net_add+0x3d/0x70 [rdma_rxe] >>> ? dev_get_by_name_rcu+0x73/0x90 >>> rxe_param_set_add+0xaf/0xc0 [rdma_rxe] >>> parse_args+0x179/0x370 >>> ? ref_module+0x1b0/0x1b0 >>> load_module+0x135e/0x17e0 >>> ? ref_module+0x1b0/0x1b0 >>> ? __do_sys_init_module+0x13b/0x180 >>> __do_sys_init_module+0x13b/0x180 >>> do_syscall_64+0x5b/0x1a0 >>> entry_SYSCALL_64_after_hwframe+0x65/0xca >>> RIP: 0033:0x7f9137ed296e >>> >>> Fixes: 8700e3e7c485 ("Soft RoCE driver") >>> Signed-off-by: Kamal Heib <kamalheib1@gmail.com> >>> --- >>> drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- >>> drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ >>> drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ >>> 3 files changed, 17 insertions(+), 7 deletions(-) >>> >>> diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c >>> index 5642eefb4ba1..60d5086dd34d 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe.c >>> +++ b/drivers/infiniband/sw/rxe/rxe.c >>> @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) >>> goto err; >>> } >>> + /* initialize slab caches for managed objects */ >>> + err = rxe_cache_init(); >>> + if (err) { >>> + pr_err("unable to init object pools\n"); >>> + goto err; >>> + } >>> + >>> err = rxe_net_add(ibdev_name, ndev); >>> if (err) { >>> pr_err("failed to add %s\n", ndev->name); >>> @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) >>> { >>> int err; >>> - /* initialize slab caches for managed objects */ >>> - err = rxe_cache_init(); When modprobe rdma_rxe, rxe_module_init should be called. Then rxe_cache_init should be also called. Why does the above call trace occur? Zhu Yanjun >>> - if (err) { >>> - pr_err("unable to init object pools\n"); >>> - return err; >>> - } >>> - >>> err = rxe_net_init(); >>> if (err) >>> return err; >>> diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c >>> index fbcbac52290b..06c6d1f835b7 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_pool.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_pool.c >>> @@ -139,6 +139,9 @@ int rxe_cache_init(void) >>> for (i = 0; i < RXE_NUM_TYPES; i++) { >>> type = &rxe_type_info[i]; >>> size = ALIGN(type->size, RXE_POOL_ALIGN); >>> + if (type->cache) >>> + continue; >>> + >>> if (!(type->flags & RXE_POOL_NO_ALLOC)) { >>> type->cache = >>> kmem_cache_create(type->name, size, >>> diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c >>> index ccda5f5a3bc0..d0af48ba0110 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c >>> @@ -81,6 +81,13 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp) >>> goto err; >>> } >>> + /* initialize slab caches for managed objects */ >>> + err = rxe_cache_init(); >>> + if (err) { >>> + pr_err("unable to init object pools\n"); >>> + goto err; >>> + } >>> + >>> err = rxe_net_add("rxe%d", ndev); >>> if (err) { >>> pr_err("failed to add %s\n", intf); >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-18 1:48 ` Zhu Yanjun @ 2020-08-18 5:50 ` Kamal Heib 2020-08-18 7:49 ` Leon Romanovsky 0 siblings, 1 reply; 11+ messages in thread From: Kamal Heib @ 2020-08-18 5:50 UTC (permalink / raw) To: Zhu Yanjun; +Cc: linux-rdma, Doug Ledford, Jason Gunthorpe On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > with a NULL pointer from pool_cache(), > > > What is the root cause of this kernel panic? > > > > > The kernel panic is triggered using the following command and it happen > > because the cache is not getting initialized. > > > > modprobe rdma_rxe add=eno1 > > > > Thanks, > > Kamal > > > > > Zhu Yanjun > > > > > > > move the rxe_cache_init() to the > > > > context of device creation. > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > PGD 0 P4D 0 > > > > Oops: 0000 [#1] SMP NOPTI > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > Call Trace: > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > add_client_context+0x11a/0x160 [ib_core] > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > ? crypto_create_tfm+0x32/0xe0 > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > parse_args+0x179/0x370 > > > > ? ref_module+0x1b0/0x1b0 > > > > load_module+0x135e/0x17e0 > > > > ? ref_module+0x1b0/0x1b0 > > > > ? __do_sys_init_module+0x13b/0x180 > > > > __do_sys_init_module+0x13b/0x180 > > > > do_syscall_64+0x5b/0x1a0 > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > --- > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > goto err; > > > > } > > > > + /* initialize slab caches for managed objects */ > > > > + err = rxe_cache_init(); > > > > + if (err) { > > > > + pr_err("unable to init object pools\n"); > > > > + goto err; > > > > + } > > > > + > > > > err = rxe_net_add(ibdev_name, ndev); > > > > if (err) { > > > > pr_err("failed to add %s\n", ndev->name); > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > { > > > > int err; > > > > - /* initialize slab caches for managed objects */ > > > > - err = rxe_cache_init(); > > When modprobe rdma_rxe, rxe_module_init should be called. Then > rxe_cache_init should be also called. > > Why does the above call trace occur? > > Zhu Yanjun > As you can see in the call trace attached to the commit message, When running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() is called before rxe_module_init() (without init the caches), so the call trace occurs when trying to register the allocated rxe device from the context of rxe_param_set_add() without initialize the caches. Thanks, Kamal > > > > - if (err) { > > > > - pr_err("unable to init object pools\n"); > > > > - return err; > > > > - } > > > > - > > > > err = rxe_net_init(); > > > > if (err) > > > > return err; > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c > > > > index fbcbac52290b..06c6d1f835b7 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_pool.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_pool.c > > > > @@ -139,6 +139,9 @@ int rxe_cache_init(void) > > > > for (i = 0; i < RXE_NUM_TYPES; i++) { > > > > type = &rxe_type_info[i]; > > > > size = ALIGN(type->size, RXE_POOL_ALIGN); > > > > + if (type->cache) > > > > + continue; > > > > + > > > > if (!(type->flags & RXE_POOL_NO_ALLOC)) { > > > > type->cache = > > > > kmem_cache_create(type->name, size, > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c > > > > index ccda5f5a3bc0..d0af48ba0110 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c > > > > @@ -81,6 +81,13 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp) > > > > goto err; > > > > } > > > > + /* initialize slab caches for managed objects */ > > > > + err = rxe_cache_init(); > > > > + if (err) { > > > > + pr_err("unable to init object pools\n"); > > > > + goto err; > > > > + } > > > > + > > > > err = rxe_net_add("rxe%d", ndev); > > > > if (err) { > > > > pr_err("failed to add %s\n", intf); > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-18 5:50 ` Kamal Heib @ 2020-08-18 7:49 ` Leon Romanovsky 2020-08-18 14:18 ` Kamal Heib 2020-08-19 3:07 ` Zhu Yanjun 0 siblings, 2 replies; 11+ messages in thread From: Leon Romanovsky @ 2020-08-18 7:49 UTC (permalink / raw) To: Kamal Heib; +Cc: Zhu Yanjun, linux-rdma, Doug Ledford, Jason Gunthorpe On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: > On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > > with a NULL pointer from pool_cache(), > > > > What is the root cause of this kernel panic? > > > > > > > The kernel panic is triggered using the following command and it happen > > > because the cache is not getting initialized. > > > > > > modprobe rdma_rxe add=eno1 > > > > > > Thanks, > > > Kamal > > > > > > > Zhu Yanjun > > > > > > > > > move the rxe_cache_init() to the > > > > > context of device creation. > > > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > > PGD 0 P4D 0 > > > > > Oops: 0000 [#1] SMP NOPTI > > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > > Call Trace: > > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > > add_client_context+0x11a/0x160 [ib_core] > > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > > ? crypto_create_tfm+0x32/0xe0 > > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > > parse_args+0x179/0x370 > > > > > ? ref_module+0x1b0/0x1b0 > > > > > load_module+0x135e/0x17e0 > > > > > ? ref_module+0x1b0/0x1b0 > > > > > ? __do_sys_init_module+0x13b/0x180 > > > > > __do_sys_init_module+0x13b/0x180 > > > > > do_syscall_64+0x5b/0x1a0 > > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > > --- > > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > > goto err; > > > > > } > > > > > + /* initialize slab caches for managed objects */ > > > > > + err = rxe_cache_init(); > > > > > + if (err) { > > > > > + pr_err("unable to init object pools\n"); > > > > > + goto err; > > > > > + } > > > > > + > > > > > err = rxe_net_add(ibdev_name, ndev); > > > > > if (err) { > > > > > pr_err("failed to add %s\n", ndev->name); > > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > > { > > > > > int err; > > > > > - /* initialize slab caches for managed objects */ > > > > > - err = rxe_cache_init(); > > > > When modprobe rdma_rxe, rxe_module_init should be called. Then > > rxe_cache_init should be also called. > > > > Why does the above call trace occur? > > > > Zhu Yanjun > > > > As you can see in the call trace attached to the commit message, When > running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() > is called before rxe_module_init() (without init the caches), so the > call trace occurs when trying to register the allocated rxe device from > the context of rxe_param_set_add() without initialize the caches. I would expect the fix being in rxe_init() instead of putting calls to rxe_cache_init() in all places. Thanks ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-18 7:49 ` Leon Romanovsky @ 2020-08-18 14:18 ` Kamal Heib 2020-08-19 3:07 ` Zhu Yanjun 1 sibling, 0 replies; 11+ messages in thread From: Kamal Heib @ 2020-08-18 14:18 UTC (permalink / raw) To: Leon Romanovsky; +Cc: Zhu Yanjun, linux-rdma, Doug Ledford, Jason Gunthorpe On Tue, Aug 18, 2020 at 10:49:56AM +0300, Leon Romanovsky wrote: > On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: > > On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > > > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > > > with a NULL pointer from pool_cache(), > > > > > What is the root cause of this kernel panic? > > > > > > > > > The kernel panic is triggered using the following command and it happen > > > > because the cache is not getting initialized. > > > > > > > > modprobe rdma_rxe add=eno1 > > > > > > > > Thanks, > > > > Kamal > > > > > > > > > Zhu Yanjun > > > > > > > > > > > move the rxe_cache_init() to the > > > > > > context of device creation. > > > > > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > > > PGD 0 P4D 0 > > > > > > Oops: 0000 [#1] SMP NOPTI > > > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > > > Call Trace: > > > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > > > add_client_context+0x11a/0x160 [ib_core] > > > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > > > ? crypto_create_tfm+0x32/0xe0 > > > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > > > parse_args+0x179/0x370 > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > load_module+0x135e/0x17e0 > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > ? __do_sys_init_module+0x13b/0x180 > > > > > > __do_sys_init_module+0x13b/0x180 > > > > > > do_syscall_64+0x5b/0x1a0 > > > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > > > --- > > > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > > > goto err; > > > > > > } > > > > > > + /* initialize slab caches for managed objects */ > > > > > > + err = rxe_cache_init(); > > > > > > + if (err) { > > > > > > + pr_err("unable to init object pools\n"); > > > > > > + goto err; > > > > > > + } > > > > > > + > > > > > > err = rxe_net_add(ibdev_name, ndev); > > > > > > if (err) { > > > > > > pr_err("failed to add %s\n", ndev->name); > > > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > > > { > > > > > > int err; > > > > > > - /* initialize slab caches for managed objects */ > > > > > > - err = rxe_cache_init(); > > > > > > When modprobe rdma_rxe, rxe_module_init should be called. Then > > > rxe_cache_init should be also called. > > > > > > Why does the above call trace occur? > > > > > > Zhu Yanjun > > > > > > > As you can see in the call trace attached to the commit message, When > > running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() > > is called before rxe_module_init() (without init the caches), so the > > call trace occurs when trying to register the allocated rxe device from > > the context of rxe_param_set_add() without initialize the caches. > > I would expect the fix being in rxe_init() instead of putting calls to > rxe_cache_init() in all places. > > Thanks OK, I agree. I'll post v2. Thanks, Kamal ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-18 7:49 ` Leon Romanovsky 2020-08-18 14:18 ` Kamal Heib @ 2020-08-19 3:07 ` Zhu Yanjun 2020-08-19 4:58 ` Leon Romanovsky 1 sibling, 1 reply; 11+ messages in thread From: Zhu Yanjun @ 2020-08-19 3:07 UTC (permalink / raw) To: Leon Romanovsky, Kamal Heib; +Cc: linux-rdma, Doug Ledford, Jason Gunthorpe On 8/18/2020 3:49 PM, Leon Romanovsky wrote: > On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: >> On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: >>> On 8/17/2020 6:12 AM, Kamal Heib wrote: >>>> On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: >>>>> On 8/12/2020 7:14 PM, Kamal Heib wrote: >>>>>> To avoid the following kernel panic when calling kmem_cache_create() >>>>>> with a NULL pointer from pool_cache(), >>>>> What is the root cause of this kernel panic? >>>>> >>>> The kernel panic is triggered using the following command and it happen >>>> because the cache is not getting initialized. >>>> >>>> modprobe rdma_rxe add=eno1 >>>> >>>> Thanks, >>>> Kamal >>>> >>>>> Zhu Yanjun >>>>> >>>>>> move the rxe_cache_init() to the >>>>>> context of device creation. >>>>>> >>>>>> BUG: unable to handle kernel NULL pointer dereference at 000000000000000b >>>>>> PGD 0 P4D 0 >>>>>> Oops: 0000 [#1] SMP NOPTI >>>>>> CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 >>>>>> Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 >>>>>> RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 >>>>>> Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 >>>>>> RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 >>>>>> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 >>>>>> RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 >>>>>> RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 >>>>>> R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 >>>>>> R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 >>>>>> FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 >>>>>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>> CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 >>>>>> Call Trace: >>>>>> rxe_alloc+0xc8/0x160 [rdma_rxe] >>>>>> rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] >>>>>> __ib_alloc_pd+0xcb/0x160 [ib_core] >>>>>> ib_mad_init_device+0x296/0x8b0 [ib_core] >>>>>> add_client_context+0x11a/0x160 [ib_core] >>>>>> enable_device_and_get+0xdc/0x1d0 [ib_core] >>>>>> ib_register_device+0x572/0x6b0 [ib_core] >>>>>> ? crypto_create_tfm+0x32/0xe0 >>>>>> ? crypto_create_tfm+0x7a/0xe0 >>>>>> ? crypto_alloc_tfm+0x58/0xf0 >>>>>> rxe_register_device+0x19d/0x1c0 [rdma_rxe] >>>>>> rxe_net_add+0x3d/0x70 [rdma_rxe] >>>>>> ? dev_get_by_name_rcu+0x73/0x90 >>>>>> rxe_param_set_add+0xaf/0xc0 [rdma_rxe] >>>>>> parse_args+0x179/0x370 >>>>>> ? ref_module+0x1b0/0x1b0 >>>>>> load_module+0x135e/0x17e0 >>>>>> ? ref_module+0x1b0/0x1b0 >>>>>> ? __do_sys_init_module+0x13b/0x180 >>>>>> __do_sys_init_module+0x13b/0x180 >>>>>> do_syscall_64+0x5b/0x1a0 >>>>>> entry_SYSCALL_64_after_hwframe+0x65/0xca >>>>>> RIP: 0033:0x7f9137ed296e >>>>>> >>>>>> Fixes: 8700e3e7c485 ("Soft RoCE driver") >>>>>> Signed-off-by: Kamal Heib <kamalheib1@gmail.com> >>>>>> --- >>>>>> drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- >>>>>> drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ >>>>>> drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ >>>>>> 3 files changed, 17 insertions(+), 7 deletions(-) >>>>>> >>>>>> diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c >>>>>> index 5642eefb4ba1..60d5086dd34d 100644 >>>>>> --- a/drivers/infiniband/sw/rxe/rxe.c >>>>>> +++ b/drivers/infiniband/sw/rxe/rxe.c >>>>>> @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) >>>>>> goto err; >>>>>> } >>>>>> + /* initialize slab caches for managed objects */ >>>>>> + err = rxe_cache_init(); >>>>>> + if (err) { >>>>>> + pr_err("unable to init object pools\n"); >>>>>> + goto err; >>>>>> + } >>>>>> + >>>>>> err = rxe_net_add(ibdev_name, ndev); >>>>>> if (err) { >>>>>> pr_err("failed to add %s\n", ndev->name); >>>>>> @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) >>>>>> { >>>>>> int err; >>>>>> - /* initialize slab caches for managed objects */ >>>>>> - err = rxe_cache_init(); >>> When modprobe rdma_rxe, rxe_module_init should be called. Then >>> rxe_cache_init should be also called. >>> >>> Why does the above call trace occur? >>> >>> Zhu Yanjun >>> >> As you can see in the call trace attached to the commit message, When >> running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() >> is called before rxe_module_init() (without init the caches), so the >> call trace occurs when trying to register the allocated rxe device from >> the context of rxe_param_set_add() without initialize the caches. > I would expect the fix being in rxe_init() instead of putting calls to > rxe_cache_init() in all places. I agree with you. Is it possible to make rxe_module_init be called before rxe_param_set_add? Thanks > > Thanks ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-19 3:07 ` Zhu Yanjun @ 2020-08-19 4:58 ` Leon Romanovsky 2020-08-19 6:19 ` Zhu Yanjun 0 siblings, 1 reply; 11+ messages in thread From: Leon Romanovsky @ 2020-08-19 4:58 UTC (permalink / raw) To: Zhu Yanjun; +Cc: Kamal Heib, linux-rdma, Doug Ledford, Jason Gunthorpe On Wed, Aug 19, 2020 at 11:07:56AM +0800, Zhu Yanjun wrote: > On 8/18/2020 3:49 PM, Leon Romanovsky wrote: > > On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: > > > On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > > > > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > > > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > > > > with a NULL pointer from pool_cache(), > > > > > > What is the root cause of this kernel panic? > > > > > > > > > > > The kernel panic is triggered using the following command and it happen > > > > > because the cache is not getting initialized. > > > > > > > > > > modprobe rdma_rxe add=eno1 > > > > > > > > > > Thanks, > > > > > Kamal > > > > > > > > > > > Zhu Yanjun > > > > > > > > > > > > > move the rxe_cache_init() to the > > > > > > > context of device creation. > > > > > > > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > > > > PGD 0 P4D 0 > > > > > > > Oops: 0000 [#1] SMP NOPTI > > > > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > > > > Call Trace: > > > > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > > > > add_client_context+0x11a/0x160 [ib_core] > > > > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > > > > ? crypto_create_tfm+0x32/0xe0 > > > > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > > > > parse_args+0x179/0x370 > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > load_module+0x135e/0x17e0 > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > ? __do_sys_init_module+0x13b/0x180 > > > > > > > __do_sys_init_module+0x13b/0x180 > > > > > > > do_syscall_64+0x5b/0x1a0 > > > > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > > > > --- > > > > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > > > > goto err; > > > > > > > } > > > > > > > + /* initialize slab caches for managed objects */ > > > > > > > + err = rxe_cache_init(); > > > > > > > + if (err) { > > > > > > > + pr_err("unable to init object pools\n"); > > > > > > > + goto err; > > > > > > > + } > > > > > > > + > > > > > > > err = rxe_net_add(ibdev_name, ndev); > > > > > > > if (err) { > > > > > > > pr_err("failed to add %s\n", ndev->name); > > > > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > > > > { > > > > > > > int err; > > > > > > > - /* initialize slab caches for managed objects */ > > > > > > > - err = rxe_cache_init(); > > > > When modprobe rdma_rxe, rxe_module_init should be called. Then > > > > rxe_cache_init should be also called. > > > > > > > > Why does the above call trace occur? > > > > > > > > Zhu Yanjun > > > > > > > As you can see in the call trace attached to the commit message, When > > > running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() > > > is called before rxe_module_init() (without init the caches), so the > > > call trace occurs when trying to register the allocated rxe device from > > > the context of rxe_param_set_add() without initialize the caches. > > I would expect the fix being in rxe_init() instead of putting calls to > > rxe_cache_init() in all places. > > I agree with you. > > Is it possible to make rxe_module_init be called before rxe_param_set_add? The best solution will be to delete module_parameters() from RXE. Thanks > > Thanks > > > > > Thanks > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-19 4:58 ` Leon Romanovsky @ 2020-08-19 6:19 ` Zhu Yanjun 2020-08-19 7:20 ` Leon Romanovsky 0 siblings, 1 reply; 11+ messages in thread From: Zhu Yanjun @ 2020-08-19 6:19 UTC (permalink / raw) To: Leon Romanovsky; +Cc: Kamal Heib, linux-rdma, Doug Ledford, Jason Gunthorpe On Wed, Aug 19, 2020 at 12:58 PM Leon Romanovsky <leon@kernel.org> wrote: > > On Wed, Aug 19, 2020 at 11:07:56AM +0800, Zhu Yanjun wrote: > > On 8/18/2020 3:49 PM, Leon Romanovsky wrote: > > > On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: > > > > On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > > > > > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > > > > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > > > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > > > > > with a NULL pointer from pool_cache(), > > > > > > > What is the root cause of this kernel panic? > > > > > > > > > > > > > The kernel panic is triggered using the following command and it happen > > > > > > because the cache is not getting initialized. > > > > > > > > > > > > modprobe rdma_rxe add=eno1 > > > > > > > > > > > > Thanks, > > > > > > Kamal > > > > > > > > > > > > > Zhu Yanjun > > > > > > > > > > > > > > > move the rxe_cache_init() to the > > > > > > > > context of device creation. > > > > > > > > > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > > > > > PGD 0 P4D 0 > > > > > > > > Oops: 0000 [#1] SMP NOPTI > > > > > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > > > > > Call Trace: > > > > > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > > > > > add_client_context+0x11a/0x160 [ib_core] > > > > > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > > > > > ? crypto_create_tfm+0x32/0xe0 > > > > > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > > > > > parse_args+0x179/0x370 > > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > > load_module+0x135e/0x17e0 > > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > > ? __do_sys_init_module+0x13b/0x180 > > > > > > > > __do_sys_init_module+0x13b/0x180 > > > > > > > > do_syscall_64+0x5b/0x1a0 > > > > > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > > > > > --- > > > > > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > > > > > goto err; > > > > > > > > } > > > > > > > > + /* initialize slab caches for managed objects */ > > > > > > > > + err = rxe_cache_init(); > > > > > > > > + if (err) { > > > > > > > > + pr_err("unable to init object pools\n"); > > > > > > > > + goto err; > > > > > > > > + } > > > > > > > > + > > > > > > > > err = rxe_net_add(ibdev_name, ndev); > > > > > > > > if (err) { > > > > > > > > pr_err("failed to add %s\n", ndev->name); > > > > > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > > > > > { > > > > > > > > int err; > > > > > > > > - /* initialize slab caches for managed objects */ > > > > > > > > - err = rxe_cache_init(); > > > > > When modprobe rdma_rxe, rxe_module_init should be called. Then > > > > > rxe_cache_init should be also called. > > > > > > > > > > Why does the above call trace occur? > > > > > > > > > > Zhu Yanjun > > > > > > > > > As you can see in the call trace attached to the commit message, When > > > > running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() > > > > is called before rxe_module_init() (without init the caches), so the > > > > call trace occurs when trying to register the allocated rxe device from > > > > the context of rxe_param_set_add() without initialize the caches. > > > I would expect the fix being in rxe_init() instead of putting calls to > > > rxe_cache_init() in all places. > > > > I agree with you. > > > > Is it possible to make rxe_module_init be called before rxe_param_set_add? > > The best solution will be to delete module_parameters() from RXE. Sure. I am curious why the parameters are set before rxe_module_init. Is this a bug? Zhu Yanjun > > Thanks > > > > > Thanks > > > > > > > > Thanks > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() 2020-08-19 6:19 ` Zhu Yanjun @ 2020-08-19 7:20 ` Leon Romanovsky 0 siblings, 0 replies; 11+ messages in thread From: Leon Romanovsky @ 2020-08-19 7:20 UTC (permalink / raw) To: Zhu Yanjun; +Cc: Kamal Heib, linux-rdma, Doug Ledford, Jason Gunthorpe On Wed, Aug 19, 2020 at 02:19:20PM +0800, Zhu Yanjun wrote: > On Wed, Aug 19, 2020 at 12:58 PM Leon Romanovsky <leon@kernel.org> wrote: > > > > On Wed, Aug 19, 2020 at 11:07:56AM +0800, Zhu Yanjun wrote: > > > On 8/18/2020 3:49 PM, Leon Romanovsky wrote: > > > > On Tue, Aug 18, 2020 at 08:50:57AM +0300, Kamal Heib wrote: > > > > > On Tue, Aug 18, 2020 at 09:48:43AM +0800, Zhu Yanjun wrote: > > > > > > On 8/17/2020 6:12 AM, Kamal Heib wrote: > > > > > > > On Sat, Aug 15, 2020 at 02:58:45PM +0800, Zhu Yanjun wrote: > > > > > > > > On 8/12/2020 7:14 PM, Kamal Heib wrote: > > > > > > > > > To avoid the following kernel panic when calling kmem_cache_create() > > > > > > > > > with a NULL pointer from pool_cache(), > > > > > > > > What is the root cause of this kernel panic? > > > > > > > > > > > > > > > The kernel panic is triggered using the following command and it happen > > > > > > > because the cache is not getting initialized. > > > > > > > > > > > > > > modprobe rdma_rxe add=eno1 > > > > > > > > > > > > > > Thanks, > > > > > > > Kamal > > > > > > > > > > > > > > > Zhu Yanjun > > > > > > > > > > > > > > > > > move the rxe_cache_init() to the > > > > > > > > > context of device creation. > > > > > > > > > > > > > > > > > > BUG: unable to handle kernel NULL pointer dereference at 000000000000000b > > > > > > > > > PGD 0 P4D 0 > > > > > > > > > Oops: 0000 [#1] SMP NOPTI > > > > > > > > > CPU: 4 PID: 8512 Comm: modprobe Kdump: loaded Not tainted 4.18.0-231.el8.x86_64 #1 > > > > > > > > > Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 > > > > > > > > > RIP: 0010:kmem_cache_alloc+0xd1/0x1b0 > > > > > > > > > Code: 8b 57 18 45 8b 77 1c 48 8b 5c 24 30 0f 1f 44 00 00 5b 48 89 e8 5d 41 5c 41 5d 41 5e 41 5f c3 81 e3 00 00 10 00 75 0e 4d 89 fe <41> f6 47 0b 04 0f 84 6c ff ff ff 4c 89 ff e8 cc da 01 00 49 89 c6 > > > > > > > > > RSP: 0018:ffffa2b8c773f9d0 EFLAGS: 00010246 > > > > > > > > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005 > > > > > > > > > RDX: 0000000000000004 RSI: 00000000006080c0 RDI: 0000000000000000 > > > > > > > > > RBP: ffff8ea0a8634fd0 R08: ffffa2b8c773f988 R09: 00000000006000c0 > > > > > > > > > R10: 0000000000000000 R11: 0000000000000230 R12: 00000000006080c0 > > > > > > > > > R13: ffffffffc0a97fc8 R14: 0000000000000000 R15: 0000000000000000 > > > > > > > > > FS: 00007f9138ed9740(0000) GS:ffff8ea4ae800000(0000) knlGS:0000000000000000 > > > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > > > > > > CR2: 000000000000000b CR3: 000000046d59a000 CR4: 00000000003406e0 > > > > > > > > > Call Trace: > > > > > > > > > rxe_alloc+0xc8/0x160 [rdma_rxe] > > > > > > > > > rxe_get_dma_mr+0x25/0xb0 [rdma_rxe] > > > > > > > > > __ib_alloc_pd+0xcb/0x160 [ib_core] > > > > > > > > > ib_mad_init_device+0x296/0x8b0 [ib_core] > > > > > > > > > add_client_context+0x11a/0x160 [ib_core] > > > > > > > > > enable_device_and_get+0xdc/0x1d0 [ib_core] > > > > > > > > > ib_register_device+0x572/0x6b0 [ib_core] > > > > > > > > > ? crypto_create_tfm+0x32/0xe0 > > > > > > > > > ? crypto_create_tfm+0x7a/0xe0 > > > > > > > > > ? crypto_alloc_tfm+0x58/0xf0 > > > > > > > > > rxe_register_device+0x19d/0x1c0 [rdma_rxe] > > > > > > > > > rxe_net_add+0x3d/0x70 [rdma_rxe] > > > > > > > > > ? dev_get_by_name_rcu+0x73/0x90 > > > > > > > > > rxe_param_set_add+0xaf/0xc0 [rdma_rxe] > > > > > > > > > parse_args+0x179/0x370 > > > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > > > load_module+0x135e/0x17e0 > > > > > > > > > ? ref_module+0x1b0/0x1b0 > > > > > > > > > ? __do_sys_init_module+0x13b/0x180 > > > > > > > > > __do_sys_init_module+0x13b/0x180 > > > > > > > > > do_syscall_64+0x5b/0x1a0 > > > > > > > > > entry_SYSCALL_64_after_hwframe+0x65/0xca > > > > > > > > > RIP: 0033:0x7f9137ed296e > > > > > > > > > > > > > > > > > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > > > > > > > > > Signed-off-by: Kamal Heib <kamalheib1@gmail.com> > > > > > > > > > --- > > > > > > > > > drivers/infiniband/sw/rxe/rxe.c | 14 +++++++------- > > > > > > > > > drivers/infiniband/sw/rxe/rxe_pool.c | 3 +++ > > > > > > > > > drivers/infiniband/sw/rxe/rxe_sysfs.c | 7 +++++++ > > > > > > > > > 3 files changed, 17 insertions(+), 7 deletions(-) > > > > > > > > > > > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > > index 5642eefb4ba1..60d5086dd34d 100644 > > > > > > > > > --- a/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > > +++ b/drivers/infiniband/sw/rxe/rxe.c > > > > > > > > > @@ -318,6 +318,13 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) > > > > > > > > > goto err; > > > > > > > > > } > > > > > > > > > + /* initialize slab caches for managed objects */ > > > > > > > > > + err = rxe_cache_init(); > > > > > > > > > + if (err) { > > > > > > > > > + pr_err("unable to init object pools\n"); > > > > > > > > > + goto err; > > > > > > > > > + } > > > > > > > > > + > > > > > > > > > err = rxe_net_add(ibdev_name, ndev); > > > > > > > > > if (err) { > > > > > > > > > pr_err("failed to add %s\n", ndev->name); > > > > > > > > > @@ -336,13 +343,6 @@ static int __init rxe_module_init(void) > > > > > > > > > { > > > > > > > > > int err; > > > > > > > > > - /* initialize slab caches for managed objects */ > > > > > > > > > - err = rxe_cache_init(); > > > > > > When modprobe rdma_rxe, rxe_module_init should be called. Then > > > > > > rxe_cache_init should be also called. > > > > > > > > > > > > Why does the above call trace occur? > > > > > > > > > > > > Zhu Yanjun > > > > > > > > > > > As you can see in the call trace attached to the commit message, When > > > > > running the "modprobe rdma_rxe add=eno1" command the rxe_param_set_add() > > > > > is called before rxe_module_init() (without init the caches), so the > > > > > call trace occurs when trying to register the allocated rxe device from > > > > > the context of rxe_param_set_add() without initialize the caches. > > > > I would expect the fix being in rxe_init() instead of putting calls to > > > > rxe_cache_init() in all places. > > > > > > I agree with you. > > > > > > Is it possible to make rxe_module_init be called before rxe_param_set_add? > > > > The best solution will be to delete module_parameters() from RXE. > > Sure. I am curious why the parameters are set before rxe_module_init. > Is this a bug? Yes and no. The part of receiving user input is correct and it should be done before rxe_module_init(), so RXE can initialize properly based on the input. The call to rxe_net_add() later inside of rxe_param_set_add() is wrong. It should be done after rxe_module_init() finishes. Thanks > > Zhu Yanjun > > > > Thanks > > > > > > > > Thanks > > > > > > > > > > > Thanks > > > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-08-19 7:20 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-08-12 11:14 [PATCH for-rc] RDMA/rxe: Fix panic when calling kmem_cache_create() Kamal Heib 2020-08-15 6:58 ` Zhu Yanjun 2020-08-16 22:12 ` Kamal Heib 2020-08-18 1:48 ` Zhu Yanjun 2020-08-18 5:50 ` Kamal Heib 2020-08-18 7:49 ` Leon Romanovsky 2020-08-18 14:18 ` Kamal Heib 2020-08-19 3:07 ` Zhu Yanjun 2020-08-19 4:58 ` Leon Romanovsky 2020-08-19 6:19 ` Zhu Yanjun 2020-08-19 7:20 ` Leon Romanovsky
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.