* RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr
@ 2021-12-21 8:04 Tony Lu
2021-12-21 8:20 ` Karsten Graul
2021-12-22 7:37 ` Thorsten Leemhuis
0 siblings, 2 replies; 5+ messages in thread
From: Tony Lu @ 2021-12-21 8:04 UTC (permalink / raw)
To: Alaa Hleihel; +Cc: Leon Romanovsky, Jason Gunthorpe, Karsten Graul, linux-rdma
Hello,
During developing and testing of SMC (net/smc), We found a problem,
when SMC released linkgroup or link, it called ib_dereg_mr to release
resources, then it panicked in mlx5_ib_dereg_mr. After investigation,
we found this panic was introduce by this commit:
f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")
After reverting this patch, SMC works fine. It looks like that
mlx5_ib_dereg_mr should check udata to determine to release umem,
because umem is union in struct, it is available when both kernel mr
and user mr. It is determined by the value of udata to distinguish
from ibv_reg_mr and ib_dereg_mr_user.
int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
// udata is NULL when called from ib_dereg_mr
if (mr->umem) { // check udata too
bool is_odp = is_odp_mr(mr);
if (!is_odp)
atomic_sub(ib_umem_num_pages(mr->umem),
&dev->mdev->priv.reg_pages);
ib_umem_release(mr->umem);
if (is_odp)
mlx5_ib_free_odp_mr(mr);
}
To be caution, this issue would cause local kernel panic, also,
it would cause remote kernel panic. SMC would setup passive close
progress when server's gone, the clients connected to this server would
go to release link, call ib_dreg_mr, and then panic.
[ 30.083527] smc: adding ib device mlx5_0 with port count 1
[ 30.084281] smc: ib device mlx5_0 port 1 has pnetid
[ 30.085006] smc: adding ib device mlx5_1 with port count 1
[ 30.085765] smc: ib device mlx5_1 port 1 has pnetid
[ 33.883596] smc: SMC-R lg 00010000 link added: id 00000101, peerid 00000101, ibdev mlx5_1, ibport 1
[ 33.884894] smc: SMC-R lg 00010000 state changed: SINGLE, pnetid
[ 33.894387] smc: SMC-R lg 00010000 link added: id 00000102, peerid 00000102, ibdev mlx5_0, ibport 1
[ 33.895612] smc: SMC-R lg 00010000 state changed: SYMMETRIC, pnetid
[ 696.351054] general protection fault, probably for non-canonical address 0x300610d01000000: 0000 [#1] PREEMPTI
[ 696.352522] CPU: 0 PID: 976 Comm: kworker/0:0 Not tainted 5.16.0-rc5+ #41
[ 696.353490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.q4
[ 696.355112] Workqueue: events smc_lgr_terminate_work [smc]
[ 696.355914] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs]
[ 696.356751] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76
[ 696.359372] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246
[ 696.360096] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0
[ 696.361110] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000
[ 696.362113] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000
[ 696.363136] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000
[ 696.364145] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168
[ 696.365153] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000
[ 696.366279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 696.367101] CR2: 00007ffeb4ede000 CR3: 0000000147b06006 CR4: 0000000000770ef0
[ 696.368121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.369118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 696.370112] PKRU: 55555554
[ 696.370528] Call Trace:
[ 696.370877] <TASK>
[ 696.371187] ib_umem_release+0x2a/0x90 [ib_uverbs]
[ 696.371889] mlx5_ib_dereg_mr+0x19b/0x400 [mlx5_ib]
[ 696.372612] ib_dereg_mr_user+0x40/0xc0 [ib_core]
[ 696.373293] smcr_buf_unmap_link+0x3b/0xa0 [smc]
[ 696.373962] smcr_link_clear.part.33+0x6d/0x1e0 [smc]
[ 696.374685] smc_lgr_free+0x101/0x150 [smc]
[ 696.375271] process_one_work+0x1af/0x3c0
[ 696.375865] worker_thread+0x4c/0x390
[ 696.376383] ? preempt_count_add+0x56/0xa0
[ 696.376961] ? rescuer_thread+0x300/0x300
[ 696.377543] kthread+0x149/0x190
[ 696.378003] ? set_kthread_struct+0x40/0x40
[ 696.378584] ret_from_fork+0x1f/0x30
[ 696.379763] </TASK>
[ 696.380723] Modules linked in: smc rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_isers
[ 696.406206] ---[ end trace 235afb848459d626 ]---
[ 696.407707] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs]
[ 696.409254] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76
[ 696.413326] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246
[ 696.414811] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0
[ 696.416544] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000
[ 696.418257] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000
[ 696.420076] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000
[ 696.421776] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168
[ 696.423456] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000
[ 696.425284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 696.426733] CR2: 00007fc639600000 CR3: 0000000147b06006 CR4: 0000000000770ef0
[ 696.428347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 696.429953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 696.431575] PKRU: 55555554
[ 696.432641] Kernel panic - not syncing: Fatal exception
[ 696.434024] Kernel Offset: disabled
[ 696.435126] Rebooting in 1 seconds..
Thanks,
Tony Lu
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr 2021-12-21 8:04 RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr Tony Lu @ 2021-12-21 8:20 ` Karsten Graul 2021-12-21 10:26 ` Leon Romanovsky 2021-12-22 7:37 ` Thorsten Leemhuis 1 sibling, 1 reply; 5+ messages in thread From: Karsten Graul @ 2021-12-21 8:20 UTC (permalink / raw) To: Tony Lu, Alaa Hleihel; +Cc: Leon Romanovsky, Jason Gunthorpe, linux-rdma On 21/12/2021 09:04, Tony Lu wrote: > Hello, > > During developing and testing of SMC (net/smc), We found a problem, > when SMC released linkgroup or link, it called ib_dereg_mr to release > resources, then it panicked in mlx5_ib_dereg_mr. After investigation, > we found this panic was introduce by this commit: > > f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") +1, this panic in our environment: [ 380.055202] smc: SMC-R lg 00000200 link removed: id 00000201, peerid 00000101, ibdev mlx5_0, ibport 1 [ 380.055230] smc: SMC-R lg 00000100 state changed: SINGLE, pnetid NET10 [ 380.055605] Unable to handle kernel pointer dereference in virtual kernel address space [ 380.055607] Failing address: 7563745f64657000 TEID: 7563745f64657803 [ 380.055609] Fault in home space mode while using kernel ASCE. [ 380.055613] AS:0000000124abc007 R3:0000000000000024 [ 380.055650] Oops: 0038 ilc:3 [#1] SMP [ 380.055655] Modules linked in: dummy smc_diag smc tcp_diag ... [ 380.055698] CPU: 2 PID: 21939 Comm: kworker/2:22 Not tainted 5.16.0-20211220.rc5.git0.c4a510cd6ab8.300.fc35.s390x #1 [ 380.055700] Hardware name: IBM 8561 T01 701 (z/VM 7.2.0) [ 380.055702] Workqueue: events smc_link_down_work [smc] [ 380.055717] Krnl PSW : 0704e00180000000 000000012311abbc (dma_unmap_sg_attrs+0x1c/0x68) [ 380.055729] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 [ 380.055732] Krnl GPRS: 0000000000000018 000000012311aba0 7563745f64657461 000000010232f003 [ 380.055735] 0000000002330003 0000000000000000 0000000000000000 0000000000000000 [ 380.055738] 0000000000000000 000000008fe64000 0000000084cd6000 000000008fe64000 [ 380.055740] 0000000035244200 00000000b669c248 000003800a077a68 000003800a077a10 [ 380.055748] Krnl Code: 000000012311abac: b90400ef lgr %r14,%r15 000000012311abb0: e3f0ffa8ff71 lay %r15,-88(%r15) #000000012311abb6: e3e0f0980024 stg %r14,152(%r15) >000000012311abbc: e3b021300002 ltg %r11,304(%r2) 000000012311abc2: a7840013 brc 8,000000012311abe8 000000012311abc6: ec52001d027f clij %r5,2,2,000000012311ac00 000000012311abcc: e310b0580002 ltg %r1,88(%r11) 000000012311abd2: a7840005 brc 8,000000012311abdc [ 380.055775] Call Trace: [ 380.055777] [<000000012311abbc>] dma_unmap_sg_attrs+0x1c/0x68 [ 380.055780] [<000003ff80560bd2>] __ib_umem_release+0xc2/0xd8 [ib_uverbs] [ 380.055797] [<000003ff805610a6>] ib_umem_release+0x4e/0xe0 [ib_uverbs] [ 380.055806] [<000003ff804fe7ca>] mlx5_ib_dereg_mr.localalias+0x212/0x480 [mlx5_ib] [ 380.055830] [<000003ff803a0ddc>] ib_dereg_mr_user+0x5c/0xe0 [ib_core] [ 380.055878] [<000003ff806c249c>] smcr_buf_unmap_link+0x64/0xe0 [smc] [ 380.055887] [<000003ff806c2cb2>] smcr_link_clear.part.0+0x72/0x230 [smc] [ 380.055896] [<000003ff806c6364>] smcr_link_down+0xc4/0x1b8 [smc] [ 380.055902] [<000003ff806c64be>] smc_link_down_work+0x66/0x88 [smc] [ 380.055909] [<00000001230a2b02>] process_one_work+0x1fa/0x470 [ 380.055913] [<00000001230a32a4>] worker_thread+0x64/0x498 [ 380.055915] [<00000001230aaf5c>] kthread+0x17c/0x188 [ 380.055919] [<00000001230333c4>] __ret_from_fork+0x3c/0x58 [ 380.055922] [<0000000123bc46ba>] ret_from_fork+0xa/0x40 [ 380.055927] Last Breaking-Event-Address: [ 380.055929] [<000003ff8054e2a8>] 0x3ff8054e2a8 [ 380.055940] Kernel panic - not syncing: Fatal exception: panic_on_oops ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr 2021-12-21 8:20 ` Karsten Graul @ 2021-12-21 10:26 ` Leon Romanovsky 0 siblings, 0 replies; 5+ messages in thread From: Leon Romanovsky @ 2021-12-21 10:26 UTC (permalink / raw) To: Karsten Graul; +Cc: Tony Lu, Alaa Hleihel, Jason Gunthorpe, linux-rdma On Tue, Dec 21, 2021 at 09:20:28AM +0100, Karsten Graul wrote: > On 21/12/2021 09:04, Tony Lu wrote: > > Hello, > > > > During developing and testing of SMC (net/smc), We found a problem, > > when SMC released linkgroup or link, it called ib_dereg_mr to release > > resources, then it panicked in mlx5_ib_dereg_mr. After investigation, > > we found this panic was introduce by this commit: > > > > f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > > +1, this panic in our environment: I assume that this patch will fix. https://lore.kernel.org/all/f298db4ec5fdf7a2d1d166ca2f66020fd9397e5c.1640079962.git.leonro@nvidia.com Thanks ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr 2021-12-21 8:04 RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr Tony Lu 2021-12-21 8:20 ` Karsten Graul @ 2021-12-22 7:37 ` Thorsten Leemhuis 2022-01-08 11:41 ` RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr #forregzbot Thorsten Leemhuis 1 sibling, 1 reply; 5+ messages in thread From: Thorsten Leemhuis @ 2021-12-22 7:37 UTC (permalink / raw) To: Tony Lu, Alaa Hleihel Cc: Leon Romanovsky, Jason Gunthorpe, Karsten Graul, linux-rdma, regressions [TLDR: I'm adding this regression to regzbot, the Linux kernel regression tracking bot; most text you find below is compiled from a few templates paragraphs some of you might have seen already.] Hi, this is your Linux kernel regression tracker speaking. Top-posting for once, to make this easy accessible to everyone. Adding the regression mailing list to the list of recipients, as it should be in the loop for all regressions, as explained here: https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html To be sure this issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot: #regzbot ^introduced f0ae4afe3d35 #regzbot monitor https://lore.kernel.org/all/f298db4ec5fdf7a2d1d166ca2f66020fd9397e5c.1640079962.git.leonro@nvidia.com/ #regzbot ignore-activity Reminder: when fixing the issue, please add a 'Link:' tag with the URL to the report (the parent of this mail) using the kernel.org redirector, as explained in 'Documentation/process/submitting-patches.rst'. Regzbot then will automatically mark the regression as resolved once the fix lands in the appropriate tree. For more details about regzbot see footer. Sending this to everyone that got the initial report, to make all aware of the tracking. I also hope that messages like this motivate people to directly get at least the regression mailing list and ideally even regzbot involved when dealing with regressions, as messages like this wouldn't be needed then. Don't worry, I'll send further messages wrt to this regression just to the lists (with a tag in the subject so people can filter them away), as long as they are intended just for regzbot. With a bit of luck no such messages will be needed anyway. Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat). P.S.: As a Linux kernel regression tracker I'm getting a lot of reports on my table. I can only look briefly into most of them. Unfortunately therefore I sometimes will get things wrong or miss something important. I hope that's not the case here; if you think it is, don't hesitate to tell me about it in a public reply. That's in everyone's interest, as what I wrote above might be misleading to everyone reading this; any suggestion I gave thus might sent someone reading this down the wrong rabbit hole, which none of us wants. BTW, I have no personal interest in this issue, which is tracked using regzbot, my Linux kernel regression tracking bot (https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting this mail to get things rolling again and hence don't need to be CC on all further activities wrt to this regression. On 21.12.21 09:04, Tony Lu wrote: > Hello, > > During developing and testing of SMC (net/smc), We found a problem, > when SMC released linkgroup or link, it called ib_dereg_mr to release > resources, then it panicked in mlx5_ib_dereg_mr. After investigation, > we found this panic was introduce by this commit: > > f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") > > After reverting this patch, SMC works fine. It looks like that > mlx5_ib_dereg_mr should check udata to determine to release umem, > because umem is union in struct, it is available when both kernel mr > and user mr. It is determined by the value of udata to distinguish > from ibv_reg_mr and ib_dereg_mr_user. > > int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) > { > // udata is NULL when called from ib_dereg_mr > if (mr->umem) { // check udata too > bool is_odp = is_odp_mr(mr); > > if (!is_odp) > atomic_sub(ib_umem_num_pages(mr->umem), > &dev->mdev->priv.reg_pages); > ib_umem_release(mr->umem); > if (is_odp) > mlx5_ib_free_odp_mr(mr); > } > > To be caution, this issue would cause local kernel panic, also, > it would cause remote kernel panic. SMC would setup passive close > progress when server's gone, the clients connected to this server would > go to release link, call ib_dreg_mr, and then panic. > > [ 30.083527] smc: adding ib device mlx5_0 with port count 1 > [ 30.084281] smc: ib device mlx5_0 port 1 has pnetid > [ 30.085006] smc: adding ib device mlx5_1 with port count 1 > [ 30.085765] smc: ib device mlx5_1 port 1 has pnetid > [ 33.883596] smc: SMC-R lg 00010000 link added: id 00000101, peerid 00000101, ibdev mlx5_1, ibport 1 > [ 33.884894] smc: SMC-R lg 00010000 state changed: SINGLE, pnetid > [ 33.894387] smc: SMC-R lg 00010000 link added: id 00000102, peerid 00000102, ibdev mlx5_0, ibport 1 > [ 33.895612] smc: SMC-R lg 00010000 state changed: SYMMETRIC, pnetid > [ 696.351054] general protection fault, probably for non-canonical address 0x300610d01000000: 0000 [#1] PREEMPTI > [ 696.352522] CPU: 0 PID: 976 Comm: kworker/0:0 Not tainted 5.16.0-rc5+ #41 > [ 696.353490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.q4 > [ 696.355112] Workqueue: events smc_lgr_terminate_work [smc] > [ 696.355914] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs] > [ 696.356751] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76 > [ 696.359372] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246 > [ 696.360096] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0 > [ 696.361110] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000 > [ 696.362113] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000 > [ 696.363136] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000 > [ 696.364145] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168 > [ 696.365153] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000 > [ 696.366279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 696.367101] CR2: 00007ffeb4ede000 CR3: 0000000147b06006 CR4: 0000000000770ef0 > [ 696.368121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 696.369118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 696.370112] PKRU: 55555554 > [ 696.370528] Call Trace: > [ 696.370877] <TASK> > [ 696.371187] ib_umem_release+0x2a/0x90 [ib_uverbs] > [ 696.371889] mlx5_ib_dereg_mr+0x19b/0x400 [mlx5_ib] > [ 696.372612] ib_dereg_mr_user+0x40/0xc0 [ib_core] > [ 696.373293] smcr_buf_unmap_link+0x3b/0xa0 [smc] > [ 696.373962] smcr_link_clear.part.33+0x6d/0x1e0 [smc] > [ 696.374685] smc_lgr_free+0x101/0x150 [smc] > [ 696.375271] process_one_work+0x1af/0x3c0 > [ 696.375865] worker_thread+0x4c/0x390 > [ 696.376383] ? preempt_count_add+0x56/0xa0 > [ 696.376961] ? rescuer_thread+0x300/0x300 > [ 696.377543] kthread+0x149/0x190 > [ 696.378003] ? set_kthread_struct+0x40/0x40 > [ 696.378584] ret_from_fork+0x1f/0x30 > [ 696.379763] </TASK> > [ 696.380723] Modules linked in: smc rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_isers > [ 696.406206] ---[ end trace 235afb848459d626 ]--- > [ 696.407707] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs] > [ 696.409254] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76 > [ 696.413326] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246 > [ 696.414811] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0 > [ 696.416544] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000 > [ 696.418257] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000 > [ 696.420076] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000 > [ 696.421776] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168 > [ 696.423456] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000 > [ 696.425284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 696.426733] CR2: 00007fc639600000 CR3: 0000000147b06006 CR4: 0000000000770ef0 > [ 696.428347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 696.429953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 696.431575] PKRU: 55555554 > [ 696.432641] Kernel panic - not syncing: Fatal exception > [ 696.434024] Kernel Offset: disabled > [ 696.435126] Rebooting in 1 seconds.. > > Thanks, > Tony Lu --- Additional information about regzbot: If you want to know more about regzbot, check out its web-interface, the getting start guide, and/or the references documentation: https://linux-regtracking.leemhuis.info/regzbot/ https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md The last two documents will explain how you can interact with regzbot yourself if your want to. Hint for reporters: when reporting a regression it's in your interest to tell #regzbot about it in the report, as that will ensure the regression gets on the radar of regzbot and the regression tracker. That's in your interest, as they will make sure the report won't fall through the cracks unnoticed. Hint for developers: you normally don't need to care about regzbot once it's involved. Fix the issue as you normally would, just remember to include a 'Link:' tag to the report in the commit message, as explained in Documentation/process/submitting-patches.rst That aspect was recently was made more explicit in commit 1f57bd42b77c: https://git.kernel.org/linus/1f57bd42b77c ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr #forregzbot 2021-12-22 7:37 ` Thorsten Leemhuis @ 2022-01-08 11:41 ` Thorsten Leemhuis 0 siblings, 0 replies; 5+ messages in thread From: Thorsten Leemhuis @ 2022-01-08 11:41 UTC (permalink / raw) To: Tony Lu, Alaa Hleihel, Maor Gottlieb Cc: Leon Romanovsky, Jason Gunthorpe, Karsten Graul, linux-rdma, regressions Hi, this is your Linux kernel regression tracker speaking. Top-posting for once, to make this easy accessible to everyone. This is now fixed, thx. A quick note to the patch author and the reviewers: Next time when you fix a issue, please include a "Link:" tag to all issue reports on the list and in bug trackers, as explained in Documentation/process/submitting-patches.rst. To quote: ``` If related discussions or any other background information behind the change can be found on the web, add 'Link:' tags pointing to it. In case your patch fixes a bug, for example, add a tag with a URL referencing the report in the mailing list archives or a bug tracker; ``` This concept is old, but the text was reworked recently to make this use case for the Link: tag clearer. For details see: https://git.kernel.org/linus/1f57bd42b77c These link help others that want to look into the issue now or in a year from now. There are also tools out there that rely on these links to connect reports and fixes. Regzbot, the regression tracking bot is such one such tool which I'm running (there might be others). And because the link was missing, I now have to tell the bot manually about the fix. :-/ #regzbot fixed-by: 4163cb3d1980383220ad7043002b930995dcba33 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=master&id=4163cb3d1980383220ad7043002b930995dcba33 Ciao, Thorsten On 22.12.21 08:37, Thorsten Leemhuis wrote: > > [TLDR: I'm adding this regression to regzbot, the Linux kernel > regression tracking bot; most text you find below is compiled from a few > templates paragraphs some of you might have seen already.] > > Hi, this is your Linux kernel regression tracker speaking. > > Top-posting for once, to make this easy accessible to everyone. > > Adding the regression mailing list to the list of recipients, as it > should be in the loop for all regressions, as explained here: > https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html > > To be sure this issue doesn't fall through the cracks unnoticed, I'm > adding it to regzbot, my Linux kernel regression tracking bot: > > #regzbot ^introduced f0ae4afe3d35 > #regzbot monitor > https://lore.kernel.org/all/f298db4ec5fdf7a2d1d166ca2f66020fd9397e5c.1640079962.git.leonro@nvidia.com/ > #regzbot ignore-activity > > Reminder: when fixing the issue, please add a 'Link:' tag with the URL > to the report (the parent of this mail) using the kernel.org redirector, > as explained in 'Documentation/process/submitting-patches.rst'. Regzbot > then will automatically mark the regression as resolved once the fix > lands in the appropriate tree. For more details about regzbot see footer. > > Sending this to everyone that got the initial report, to make all aware > of the tracking. I also hope that messages like this motivate people to > directly get at least the regression mailing list and ideally even > regzbot involved when dealing with regressions, as messages like this > wouldn't be needed then. > > Don't worry, I'll send further messages wrt to this regression just to > the lists (with a tag in the subject so people can filter them away), as > long as they are intended just for regzbot. With a bit of luck no such > messages will be needed anyway. > > Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat). > > P.S.: As a Linux kernel regression tracker I'm getting a lot of reports > on my table. I can only look briefly into most of them. Unfortunately > therefore I sometimes will get things wrong or miss something important. > I hope that's not the case here; if you think it is, don't hesitate to > tell me about it in a public reply. That's in everyone's interest, as > what I wrote above might be misleading to everyone reading this; any > suggestion I gave thus might sent someone reading this down the wrong > rabbit hole, which none of us wants. > > BTW, I have no personal interest in this issue, which is tracked using > regzbot, my Linux kernel regression tracking bot > (https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting > this mail to get things rolling again and hence don't need to be CC on > all further activities wrt to this regression. > > > On 21.12.21 09:04, Tony Lu wrote: >> Hello, >> >> During developing and testing of SMC (net/smc), We found a problem, >> when SMC released linkgroup or link, it called ib_dereg_mr to release >> resources, then it panicked in mlx5_ib_dereg_mr. After investigation, >> we found this panic was introduce by this commit: >> >> f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow") >> >> After reverting this patch, SMC works fine. It looks like that >> mlx5_ib_dereg_mr should check udata to determine to release umem, >> because umem is union in struct, it is available when both kernel mr >> and user mr. It is determined by the value of udata to distinguish >> from ibv_reg_mr and ib_dereg_mr_user. >> >> int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) >> { >> // udata is NULL when called from ib_dereg_mr >> if (mr->umem) { // check udata too >> bool is_odp = is_odp_mr(mr); >> >> if (!is_odp) >> atomic_sub(ib_umem_num_pages(mr->umem), >> &dev->mdev->priv.reg_pages); >> ib_umem_release(mr->umem); >> if (is_odp) >> mlx5_ib_free_odp_mr(mr); >> } >> >> To be caution, this issue would cause local kernel panic, also, >> it would cause remote kernel panic. SMC would setup passive close >> progress when server's gone, the clients connected to this server would >> go to release link, call ib_dreg_mr, and then panic. >> >> [ 30.083527] smc: adding ib device mlx5_0 with port count 1 >> [ 30.084281] smc: ib device mlx5_0 port 1 has pnetid >> [ 30.085006] smc: adding ib device mlx5_1 with port count 1 >> [ 30.085765] smc: ib device mlx5_1 port 1 has pnetid >> [ 33.883596] smc: SMC-R lg 00010000 link added: id 00000101, peerid 00000101, ibdev mlx5_1, ibport 1 >> [ 33.884894] smc: SMC-R lg 00010000 state changed: SINGLE, pnetid >> [ 33.894387] smc: SMC-R lg 00010000 link added: id 00000102, peerid 00000102, ibdev mlx5_0, ibport 1 >> [ 33.895612] smc: SMC-R lg 00010000 state changed: SYMMETRIC, pnetid >> [ 696.351054] general protection fault, probably for non-canonical address 0x300610d01000000: 0000 [#1] PREEMPTI >> [ 696.352522] CPU: 0 PID: 976 Comm: kworker/0:0 Not tainted 5.16.0-rc5+ #41 >> [ 696.353490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.q4 >> [ 696.355112] Workqueue: events smc_lgr_terminate_work [smc] >> [ 696.355914] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs] >> [ 696.356751] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76 >> [ 696.359372] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246 >> [ 696.360096] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0 >> [ 696.361110] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000 >> [ 696.362113] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000 >> [ 696.363136] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000 >> [ 696.364145] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168 >> [ 696.365153] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000 >> [ 696.366279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 696.367101] CR2: 00007ffeb4ede000 CR3: 0000000147b06006 CR4: 0000000000770ef0 >> [ 696.368121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 696.369118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 696.370112] PKRU: 55555554 >> [ 696.370528] Call Trace: >> [ 696.370877] <TASK> >> [ 696.371187] ib_umem_release+0x2a/0x90 [ib_uverbs] >> [ 696.371889] mlx5_ib_dereg_mr+0x19b/0x400 [mlx5_ib] >> [ 696.372612] ib_dereg_mr_user+0x40/0xc0 [ib_core] >> [ 696.373293] smcr_buf_unmap_link+0x3b/0xa0 [smc] >> [ 696.373962] smcr_link_clear.part.33+0x6d/0x1e0 [smc] >> [ 696.374685] smc_lgr_free+0x101/0x150 [smc] >> [ 696.375271] process_one_work+0x1af/0x3c0 >> [ 696.375865] worker_thread+0x4c/0x390 >> [ 696.376383] ? preempt_count_add+0x56/0xa0 >> [ 696.376961] ? rescuer_thread+0x300/0x300 >> [ 696.377543] kthread+0x149/0x190 >> [ 696.378003] ? set_kthread_struct+0x40/0x40 >> [ 696.378584] ret_from_fork+0x1f/0x30 >> [ 696.379763] </TASK> >> [ 696.380723] Modules linked in: smc rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_isers >> [ 696.406206] ---[ end trace 235afb848459d626 ]--- >> [ 696.407707] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs] >> [ 696.409254] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76 >> [ 696.413326] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246 >> [ 696.414811] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0 >> [ 696.416544] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000 >> [ 696.418257] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000 >> [ 696.420076] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000 >> [ 696.421776] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168 >> [ 696.423456] FS: 0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000 >> [ 696.425284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 696.426733] CR2: 00007fc639600000 CR3: 0000000147b06006 CR4: 0000000000770ef0 >> [ 696.428347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 696.429953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 696.431575] PKRU: 55555554 >> [ 696.432641] Kernel panic - not syncing: Fatal exception >> [ 696.434024] Kernel Offset: disabled >> [ 696.435126] Rebooting in 1 seconds.. >> >> Thanks, >> Tony Lu > > --- > Additional information about regzbot: > > If you want to know more about regzbot, check out its web-interface, the > getting start guide, and/or the references documentation: > > https://linux-regtracking.leemhuis.info/regzbot/ > https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md > https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md > > The last two documents will explain how you can interact with regzbot > yourself if your want to. > > Hint for reporters: when reporting a regression it's in your interest to > tell #regzbot about it in the report, as that will ensure the regression > gets on the radar of regzbot and the regression tracker. That's in your > interest, as they will make sure the report won't fall through the > cracks unnoticed. > > Hint for developers: you normally don't need to care about regzbot once > it's involved. Fix the issue as you normally would, just remember to > include a 'Link:' tag to the report in the commit message, as explained > in Documentation/process/submitting-patches.rst > That aspect was recently was made more explicit in commit 1f57bd42b77c: > https://git.kernel.org/linus/1f57bd42b77c ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-01-08 11:41 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-12-21 8:04 RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr Tony Lu 2021-12-21 8:20 ` Karsten Graul 2021-12-21 10:26 ` Leon Romanovsky 2021-12-22 7:37 ` Thorsten Leemhuis 2022-01-08 11:41 ` RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr #forregzbot Thorsten Leemhuis
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.