From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
To: linux-rdma@vger.kernel.org, Jason Gunthorpe <jgg@ziepe.ca>,
Leon Romanovsky <leon@kernel.org>
Cc: linux-nvme@lists.infradead.org,
Damien Le Moal <dlemoal@kernel.org>,
Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Subject: [PATCH v2] RDMA/cma: prevent rdma id destroy during cma_iw_handler
Date: Mon, 12 Jun 2023 14:42:37 +0900 [thread overview]
Message-ID: <20230612054237.1855292-1-shinichiro.kawasaki@wdc.com> (raw)
When rdma_destroy_id() and cma_iw_handler() race, struct rdma_id_private
*id_priv can be destroyed during cma_iw_handler call. This causes "BUG:
KASAN: slab-use-after-free" at mutex_lock() in cma_iw_handler() [1].
To prevent the destroy of id_priv, keep its reference count by calling
cma_id_get() and cma_id_put() at start and end of cma_iw_handler().
[1]
==================================================================
BUG: KASAN: slab-use-after-free in __mutex_lock+0x1324/0x18f0
Read of size 8 at addr ffff888197b37418 by task kworker/u8:0/9
CPU: 0 PID: 9 Comm: kworker/u8:0 Not tainted 6.3.0 #62
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
Workqueue: iw_cm_wq cm_work_handler [iw_cm]
Call Trace:
<TASK>
dump_stack_lvl+0x57/0x90
print_report+0xcf/0x660
? __mutex_lock+0x1324/0x18f0
kasan_report+0xa4/0xe0
? __mutex_lock+0x1324/0x18f0
__mutex_lock+0x1324/0x18f0
? cma_iw_handler+0xac/0x4f0 [rdma_cm]
? _raw_spin_unlock_irqrestore+0x30/0x60
? rcu_is_watching+0x11/0xb0
? _raw_spin_unlock_irqrestore+0x30/0x60
? trace_hardirqs_on+0x12/0x100
? __pfx___mutex_lock+0x10/0x10
? __percpu_counter_sum+0x147/0x1e0
? domain_dirty_limits+0x246/0x390
? wb_over_bg_thresh+0x4d5/0x610
? rcu_is_watching+0x11/0xb0
? cma_iw_handler+0xac/0x4f0 [rdma_cm]
cma_iw_handler+0xac/0x4f0 [rdma_cm]
? rcu_is_watching+0x11/0xb0
? __pfx_cma_iw_handler+0x10/0x10 [rdma_cm]
? attach_entity_load_avg+0x4e2/0x920
? _raw_spin_unlock_irqrestore+0x30/0x60
? rcu_is_watching+0x11/0xb0
cm_work_handler+0x139e/0x1c50 [iw_cm]
? __pfx_cm_work_handler+0x10/0x10 [iw_cm]
? rcu_is_watching+0x11/0xb0
? __pfx_try_to_wake_up+0x10/0x10
? __pfx_do_raw_spin_lock+0x10/0x10
? __pfx___might_resched+0x10/0x10
? _raw_spin_unlock_irq+0x24/0x50
process_one_work+0x843/0x1350
? __pfx_lock_acquire+0x10/0x10
? __pfx_process_one_work+0x10/0x10
? __pfx_do_raw_spin_lock+0x10/0x10
worker_thread+0xfc/0x1260
? __pfx_worker_thread+0x10/0x10
kthread+0x29e/0x340
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Allocated by task 4225:
kasan_save_stack+0x2f/0x50
kasan_set_track+0x21/0x30
__kasan_kmalloc+0xa6/0xb0
__rdma_create_id+0x5b/0x5d0 [rdma_cm]
__rdma_create_kernel_id+0x12/0x40 [rdma_cm]
nvme_rdma_alloc_queue+0x26a/0x5f0 [nvme_rdma]
nvme_rdma_setup_ctrl+0xb84/0x1d90 [nvme_rdma]
nvme_rdma_create_ctrl+0x7b5/0xd20 [nvme_rdma]
nvmf_dev_write+0xddd/0x22b0 [nvme_fabrics]
vfs_write+0x211/0xd50
ksys_write+0x100/0x1e0
do_syscall_64+0x5b/0x80
entry_SYSCALL_64_after_hwframe+0x72/0xdc
Freed by task 4227:
kasan_save_stack+0x2f/0x50
kasan_set_track+0x21/0x30
kasan_save_free_info+0x2a/0x50
____kasan_slab_free+0x169/0x1c0
slab_free_freelist_hook+0xdb/0x1b0
__kmem_cache_free+0xb8/0x2e0
nvme_rdma_free_queue+0x4a/0x70 [nvme_rdma]
nvme_rdma_teardown_io_queues.part.0+0x14a/0x1e0 [nvme_rdma]
nvme_rdma_delete_ctrl+0x4f/0x100 [nvme_rdma]
nvme_do_delete_ctrl+0x14e/0x240 [nvme_core]
nvme_sysfs_delete+0xcb/0x100 [nvme_core]
kernfs_fop_write_iter+0x359/0x530
vfs_write+0x58f/0xd50
ksys_write+0x100/0x1e0
do_syscall_64+0x5b/0x80
entry_SYSCALL_64_after_hwframe+0x72/0xdc
The buggy address belongs to the object at ffff888197b37000
which belongs to the cache kmalloc-2k of size 2048
The buggy address is located 1048 bytes inside of
freed 2048-byte region [ffff888197b37000, ffff888197b37800)
The buggy address belongs to the physical page:
page:00000000fbe33a6e refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x197b30
head:00000000fbe33a6e order:3 entire_mapcount:0 nr_pages_mapped:0 pincount:0
anon flags: 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
raw: 0017ffffc0010200 ffff888100042f00 0000000000000000 dead000000000001
raw: 0000000000000000 0000000000080008 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff888197b37300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888197b37380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888197b37400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff888197b37480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888197b37500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
Fixes: de910bd92137 ("RDMA/cma: Simplify locking needed for serialization of callbacks")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Cc: stable@vger.kernel.org
---
The BUG KASAN was observed with blktests at test cases nvme/030 or nvme/031,
using SIW transport [*]. To reproduce it, it is required to repeat the test
cases from 30 to 50 times on my test system.
[*] https://lore.kernel.org/linux-block/rsmmxrchy6voi5qhl4irss5sprna3f5owkqtvybxglcv2pnylm@xmrnpfu3tfpe/
Changes from v1:
* Improved the commit message per comments on the list
drivers/infiniband/core/cma.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 93a1c48d0c32..c5267d9bb184 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -2477,6 +2477,7 @@ static int cma_iw_handler(struct iw_cm_id *iw_id, struct iw_cm_event *iw_event)
struct sockaddr *laddr = (struct sockaddr *)&iw_event->local_addr;
struct sockaddr *raddr = (struct sockaddr *)&iw_event->remote_addr;
+ cma_id_get(id_priv);
mutex_lock(&id_priv->handler_mutex);
if (READ_ONCE(id_priv->state) != RDMA_CM_CONNECT)
goto out;
@@ -2524,12 +2525,14 @@ static int cma_iw_handler(struct iw_cm_id *iw_id, struct iw_cm_event *iw_event)
if (ret) {
/* Destroy the CM ID by returning a non-zero value. */
id_priv->cm_id.iw = NULL;
+ cma_id_put(id_priv);
destroy_id_handler_unlock(id_priv);
return ret;
}
out:
mutex_unlock(&id_priv->handler_mutex);
+ cma_id_put(id_priv);
return ret;
}
--
2.40.1
next reply other threads:[~2023-06-12 5:47 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-12 5:42 Shin'ichiro Kawasaki [this message]
2023-06-12 7:13 ` [PATCH v2] RDMA/cma: prevent rdma id destroy during cma_iw_handler Leon Romanovsky
2023-06-12 14:18 ` Jason Gunthorpe
2023-06-13 1:43 ` Shinichiro Kawasaki
2023-06-13 6:47 ` Shinichiro Kawasaki
2023-06-13 13:30 ` Jason Gunthorpe
2023-06-13 18:07 ` Leon Romanovsky
2023-06-14 7:53 ` Shinichiro Kawasaki
2023-06-14 17:36 ` Jason Gunthorpe
2023-06-15 0:45 ` Shinichiro Kawasaki
2023-09-05 0:39 ` Shinichiro Kawasaki
2023-09-06 8:54 ` Daniel Wagner
2023-09-06 11:31 ` Shinichiro Kawasaki
2023-09-07 7:47 ` Yi Zhang
2023-09-07 16:17 ` Jason Gunthorpe
2023-09-08 7:18 ` Hannes Reinecke
2023-09-08 8:33 ` Yi Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230612054237.1855292-1-shinichiro.kawasaki@wdc.com \
--to=shinichiro.kawasaki@wdc.com \
--cc=dlemoal@kernel.org \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.