linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tony Lu <tonylu@linux.alibaba.com>
To: Alaa Hleihel <alaa@nvidia.com>
Cc: Leon Romanovsky <leon@kernel.org>, Jason Gunthorpe <jgg@ziepe.ca>,
	Karsten Graul <kgraul@linux.ibm.com>,
	linux-rdma@vger.kernel.org
Subject: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr
Date: Tue, 21 Dec 2021 16:04:58 +0800	[thread overview]
Message-ID: <9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com> (raw)

Hello,

During developing and testing of SMC (net/smc), We found a problem,
when SMC released linkgroup or link, it called ib_dereg_mr to release
resources, then it panicked in mlx5_ib_dereg_mr. After investigation,
we found this panic was introduce by this commit:

    f0ae4afe3d35 ("RDMA/mlx5: Fix releasing unallocated memory in dereg MR flow")

After reverting this patch, SMC works fine. It looks like that
mlx5_ib_dereg_mr should check udata to determine to release umem,
because umem is union in struct, it is available when both kernel mr
and user mr. It is determined by the value of udata to distinguish
from ibv_reg_mr and ib_dereg_mr_user.

int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
        // udata is NULL when called from ib_dereg_mr
        if (mr->umem) { // check udata too
                bool is_odp = is_odp_mr(mr);

                if (!is_odp)
                        atomic_sub(ib_umem_num_pages(mr->umem),
                                   &dev->mdev->priv.reg_pages);
                ib_umem_release(mr->umem);
                if (is_odp)
                        mlx5_ib_free_odp_mr(mr);
        }

To be caution, this issue would cause local kernel panic, also,
it would cause remote kernel panic. SMC would setup passive close
progress when server's gone, the clients connected to this server would
go to release link, call ib_dreg_mr, and then panic.

[   30.083527] smc: adding ib device mlx5_0 with port count 1
[   30.084281] smc:    ib device mlx5_0 port 1 has pnetid
[   30.085006] smc: adding ib device mlx5_1 with port count 1
[   30.085765] smc:    ib device mlx5_1 port 1 has pnetid
[   33.883596] smc: SMC-R lg 00010000 link added: id 00000101, peerid 00000101, ibdev mlx5_1, ibport 1
[   33.884894] smc: SMC-R lg 00010000 state changed: SINGLE, pnetid
[   33.894387] smc: SMC-R lg 00010000 link added: id 00000102, peerid 00000102, ibdev mlx5_0, ibport 1
[   33.895612] smc: SMC-R lg 00010000 state changed: SYMMETRIC, pnetid
[  696.351054] general protection fault, probably for non-canonical address 0x300610d01000000: 0000 [#1] PREEMPTI
[  696.352522] CPU: 0 PID: 976 Comm: kworker/0:0 Not tainted 5.16.0-rc5+ #41
[  696.353490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.q4
[  696.355112] Workqueue: events smc_lgr_terminate_work [smc]
[  696.355914] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs]
[  696.356751] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76
[  696.359372] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246
[  696.360096] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0
[  696.361110] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000
[  696.362113] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000
[  696.363136] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000
[  696.364145] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168
[  696.365153] FS:  0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000
[  696.366279] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  696.367101] CR2: 00007ffeb4ede000 CR3: 0000000147b06006 CR4: 0000000000770ef0
[  696.368121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  696.369118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  696.370112] PKRU: 55555554
[  696.370528] Call Trace:
[  696.370877]  <TASK>
[  696.371187]  ib_umem_release+0x2a/0x90 [ib_uverbs]
[  696.371889]  mlx5_ib_dereg_mr+0x19b/0x400 [mlx5_ib]
[  696.372612]  ib_dereg_mr_user+0x40/0xc0 [ib_core]
[  696.373293]  smcr_buf_unmap_link+0x3b/0xa0 [smc]
[  696.373962]  smcr_link_clear.part.33+0x6d/0x1e0 [smc]
[  696.374685]  smc_lgr_free+0x101/0x150 [smc]
[  696.375271]  process_one_work+0x1af/0x3c0
[  696.375865]  worker_thread+0x4c/0x390
[  696.376383]  ? preempt_count_add+0x56/0xa0
[  696.376961]  ? rescuer_thread+0x300/0x300
[  696.377543]  kthread+0x149/0x190
[  696.378003]  ? set_kthread_struct+0x40/0x40
[  696.378584]  ret_from_fork+0x1f/0x30
[  696.379763]  </TASK>
[  696.380723] Modules linked in: smc rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_isers
[  696.406206] ---[ end trace 235afb848459d626 ]---
[  696.407707] RIP: 0010:__ib_umem_release+0x21/0xa0 [ib_uverbs]
[  696.409254] Code: ff ff 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 55 41 54 41 89 d4 55 53 48 89 f5 f6 46 28 01 76
[  696.413326] RSP: 0018:ffffc9000045bd30 EFLAGS: 00010246
[  696.414811] RAX: 0000000000000000 RBX: ffff8881108bd000 RCX: ffff888141a3a1a0
[  696.416544] RDX: 0000000000000001 RSI: ffff8881108bd000 RDI: 0300610d01000000
[  696.418257] RBP: ffff8881108bd000 R08: ffffc9000045bd60 R09: 0000000000000000
[  696.420076] R10: ffff888140052864 R11: 0000000000000008 R12: 0000000000000000
[  696.421776] R13: ffff888114310000 R14: 0000000000000000 R15: ffff8881426ac168
[  696.423456] FS:  0000000000000000(0000) GS:ffff88881fc00000(0000) knlGS:0000000000000000
[  696.425284] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  696.426733] CR2: 00007fc639600000 CR3: 0000000147b06006 CR4: 0000000000770ef0
[  696.428347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  696.429953] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  696.431575] PKRU: 55555554
[  696.432641] Kernel panic - not syncing: Fatal exception
[  696.434024] Kernel Offset: disabled
[  696.435126] Rebooting in 1 seconds..

Thanks,
Tony Lu

             reply	other threads:[~2021-12-21  8:05 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-21  8:04 Tony Lu [this message]
2021-12-21  8:20 ` Karsten Graul
2021-12-21 10:26   ` Leon Romanovsky
2021-12-22  7:37 ` Thorsten Leemhuis
2022-01-08 11:41   ` RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr #forregzbot Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9974ea8c-f1cb-aeb4-cf1b-19d37536894a@linux.alibaba.com \
    --to=tonylu@linux.alibaba.com \
    --cc=alaa@nvidia.com \
    --cc=jgg@ziepe.ca \
    --cc=kgraul@linux.ibm.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --subject='Re: RDMA/mlx5: Regression since v5.15-rc5: Kernel panic when called ib_dereg_mr' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).