* WARNING in cma_exit_net
@ 2019-04-01 17:36 syzbot
2019-04-01 17:45 ` Jason Gunthorpe
0 siblings, 1 reply; 5+ messages in thread
From: syzbot @ 2019-04-01 17:36 UTC (permalink / raw)
To: danielj, danitg, dledford, jgg, leon, linux-kernel, linux-rdma,
parav, swise, syzkaller-bugs
Hello,
syzbot found the following crash on:
HEAD commit: e3ecb83e Add linux-next specific files for 20190401
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=13bc36cd200000
kernel config: https://syzkaller.appspot.com/x/.config?x=db6c9f2bfeb91a99
dashboard link: https://syzkaller.appspot.com/bug?extid=2e3e485d5697ea610460
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+2e3e485d5697ea610460@syzkaller.appspotmail.com
WARNING: CPU: 1 PID: 7 at drivers/infiniband/core/cma.c:4674
cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
Kernel panic - not syncing: panic_on_warn set ...
CPU: 1 PID: 7 Comm: kworker/u4:0 Not tainted 5.1.0-rc2-next-20190401 #15
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
panic+0x2cb/0x65c kernel/panic.c:214
__warn.cold+0x20/0x45 kernel/panic.c:571
report_bug+0x263/0x2b0 lib/bug.c:186
fixup_bug arch/x86/kernel/traps.c:179 [inline]
fixup_bug arch/x86/kernel/traps.c:174 [inline]
do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:272
do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:291
invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:973
RIP: 0010:cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
Code: 00 48 c7 c7 c0 4e ea 87 c6 05 89 86 4a 04 01 e8 ba cc 46 fc e9 c8 fd
ff ff e8 25 6c 5f fc 0f 0b e9 d7 fe ff ff e8 19 6c 5f fc <0f> 0b e8 12 6c
5f fc 48 8d 65 e8 5b 41 5c 41 5d 5d c3 e8 02 6c 5f
RSP: 0000:ffff8880aa267bf0 EFLAGS: 00010293
RAX: ffff8880aa2581c0 RBX: ffff8880a538d500 RCX: 1ffff1101544b146
RDX: 0000000000000000 RSI: ffffffff85111537 RDI: ffff8880a538d618
RBP: ffff8880aa267c08 R08: ffff8880aa2581c0 R09: 0000000000000004
R10: ffffed1015d25bcf R11: ffff8880ae92de7b R12: 0000000000000001
R13: ffff8880a90e84c0 R14: ffffffff891e29f8 R15: dffffc0000000000
ops_exit_list.isra.0+0xb0/0x160 net/core/net_namespace.c:153
cleanup_net+0x3fb/0x960 net/core/net_namespace.c:552
process_one_work+0x98e/0x1790 kernel/workqueue.c:2270
worker_thread+0x98/0xe40 kernel/workqueue.c:2416
kthread+0x357/0x430 kernel/kthread.c:254
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352
Kernel Offset: disabled
Rebooting in 86400 seconds..
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: WARNING in cma_exit_net
2019-04-01 17:36 WARNING in cma_exit_net syzbot
@ 2019-04-01 17:45 ` Jason Gunthorpe
2019-04-01 18:29 ` Leon Romanovsky
0 siblings, 1 reply; 5+ messages in thread
From: Jason Gunthorpe @ 2019-04-01 17:45 UTC (permalink / raw)
To: syzbot, Matthew Wilcox
Cc: danielj, danitg, dledford, leon, linux-kernel, linux-rdma, parav,
swise, syzkaller-bugs
On Mon, Apr 01, 2019 at 10:36:05AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: e3ecb83e Add linux-next specific files for 20190401
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=13bc36cd200000
> kernel config: https://syzkaller.appspot.com/x/.config?x=db6c9f2bfeb91a99
> dashboard link: https://syzkaller.appspot.com/bug?extid=2e3e485d5697ea610460
> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+2e3e485d5697ea610460@syzkaller.appspotmail.com
>
> WARNING: CPU: 1 PID: 7 at drivers/infiniband/core/cma.c:4674
> cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
> Kernel panic - not syncing: panic_on_warn set ...
Matt: This is why the WARN_ON(!xa_empty()) is so valuable. Magically
syzkaller can find something in this code is buggy.
Mellanox is also showing a different testing failure over the weekend
(use after free or something) from your 'cma: Convert portspace IDRs
to XArray'
If you seem something obvious wrong in that patch please speak
up.. Below is what I applied.
Thanks,
Jason
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 68c997be242930..a8b9c66c85252b 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -39,7 +39,7 @@
#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/igmp.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/inetdevice.h>
#include <linux/slab.h>
#include <linux/module.h>
@@ -191,10 +191,10 @@ static struct workqueue_struct *cma_wq;
static unsigned int cma_pernet_id;
struct cma_pernet {
- struct idr tcp_ps;
- struct idr udp_ps;
- struct idr ipoib_ps;
- struct idr ib_ps;
+ struct xarray tcp_ps;
+ struct xarray udp_ps;
+ struct xarray ipoib_ps;
+ struct xarray ib_ps;
};
static struct cma_pernet *cma_pernet(struct net *net)
@@ -202,7 +202,8 @@ static struct cma_pernet *cma_pernet(struct net *net)
return net_generic(net, cma_pernet_id);
}
-static struct idr *cma_pernet_idr(struct net *net, enum rdma_ucm_port_space ps)
+static
+struct xarray *cma_pernet_xa(struct net *net, enum rdma_ucm_port_space ps)
{
struct cma_pernet *pernet = cma_pernet(net);
@@ -247,25 +248,25 @@ struct class_port_info_context {
static int cma_ps_alloc(struct net *net, enum rdma_ucm_port_space ps,
struct rdma_bind_list *bind_list, int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- return idr_alloc(idr, bind_list, snum, snum + 1, GFP_KERNEL);
+ return xa_insert(xa, snum, bind_list, GFP_KERNEL);
}
static struct rdma_bind_list *cma_ps_find(struct net *net,
enum rdma_ucm_port_space ps, int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- return idr_find(idr, snum);
+ return xa_load(xa, snum);
}
static void cma_ps_remove(struct net *net, enum rdma_ucm_port_space ps,
int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- idr_remove(idr, snum);
+ xa_erase(xa, snum);
}
enum {
@@ -4655,10 +4656,10 @@ static int cma_init_net(struct net *net)
{
struct cma_pernet *pernet = cma_pernet(net);
- idr_init(&pernet->tcp_ps);
- idr_init(&pernet->udp_ps);
- idr_init(&pernet->ipoib_ps);
- idr_init(&pernet->ib_ps);
+ xa_init(&pernet->tcp_ps);
+ xa_init(&pernet->udp_ps);
+ xa_init(&pernet->ipoib_ps);
+ xa_init(&pernet->ib_ps);
return 0;
}
@@ -4667,10 +4668,10 @@ static void cma_exit_net(struct net *net)
{
struct cma_pernet *pernet = cma_pernet(net);
- idr_destroy(&pernet->tcp_ps);
- idr_destroy(&pernet->udp_ps);
- idr_destroy(&pernet->ipoib_ps);
- idr_destroy(&pernet->ib_ps);
+ WARN_ON(!xa_empty(&pernet->tcp_ps));
+ WARN_ON(!xa_empty(&pernet->udp_ps));
+ WARN_ON(!xa_empty(&pernet->ipoib_ps));
+ WARN_ON(!xa_empty(&pernet->ib_ps));
}
static struct pernet_operations cma_pernet_operations = {
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: WARNING in cma_exit_net
2019-04-01 17:45 ` Jason Gunthorpe
@ 2019-04-01 18:29 ` Leon Romanovsky
2019-04-02 23:39 ` Bart Van Assche
0 siblings, 1 reply; 5+ messages in thread
From: Leon Romanovsky @ 2019-04-01 18:29 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: syzbot, Matthew Wilcox, danielj, danitg, dledford, linux-kernel,
linux-rdma, parav, swise, syzkaller-bugs
[-- Attachment #1: Type: text/plain, Size: 9344 bytes --]
On Mon, Apr 01, 2019 at 02:45:54PM -0300, Jason Gunthorpe wrote:
> On Mon, Apr 01, 2019 at 10:36:05AM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit: e3ecb83e Add linux-next specific files for 20190401
> > git tree: linux-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=13bc36cd200000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=db6c9f2bfeb91a99
> > dashboard link: https://syzkaller.appspot.com/bug?extid=2e3e485d5697ea610460
> > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >
> > Unfortunately, I don't have any reproducer for this crash yet.
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+2e3e485d5697ea610460@syzkaller.appspotmail.com
> >
> > WARNING: CPU: 1 PID: 7 at drivers/infiniband/core/cma.c:4674
> > cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
> > Kernel panic - not syncing: panic_on_warn set ...
>
> Matt: This is why the WARN_ON(!xa_empty()) is so valuable. Magically
> syzkaller can find something in this code is buggy.
>
> Mellanox is also showing a different testing failure over the weekend
> (use after free or something) from your 'cma: Convert portspace IDRs
> to XArray'
This is what I see in my environment.
[ 72.725596]
==================================================================
[ 72.726017] BUG: KASAN: use-after-free in cma_check_port+0x86a/0xa20 [rdma_cm]
[ 72.726263] Read of size 8 at addr ffff888069fde998 by task ucmatose/387
[ 72.726460]
[ 72.726550] CPU: 3 PID: 387 Comm: ucmatose Not tainted 5.1.0-rc2+ #253
[ 72.726751] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[ 72.727119] Call Trace:
[ 72.727210] dump_stack+0x7c/0xc0
[ 72.727342] print_address_description+0x6c/0x23c
[ 72.727505] ? cma_check_port+0x86a/0xa20 [rdma_cm]
[ 72.727666] kasan_report.cold.3+0x1c/0x35
[ 72.727805] ? cma_check_port+0x86a/0xa20 [rdma_cm]
[ 72.727977] ? cma_check_port+0x86a/0xa20 [rdma_cm]
[ 72.728138] cma_check_port+0x86a/0xa20 [rdma_cm]
[ 72.728306] rdma_bind_addr+0x11bc/0x1b00 [rdma_cm]
[ 72.728465] ? find_held_lock+0x33/0x1c0
[ 72.728597] ? cma_ndev_work_handler+0x180/0x180 [rdma_cm]
[ 72.728756] ? wait_for_completion+0x3d0/0x3d0
[ 72.728928] ucma_bind+0x120/0x160 [rdma_ucm]
[ 72.729089] ? ucma_resolve_addr+0x1a0/0x1a0 [rdma_ucm]
[ 72.729256] ucma_write+0x1f8/0x2b0 [rdma_ucm]
[ 72.729409] ? ucma_open+0x260/0x260 [rdma_ucm]
[ 72.729571] vfs_write+0x157/0x460
[ 72.729688] ksys_write+0xb8/0x170
[ 72.729828] ? __ia32_sys_read+0xb0/0xb0
[ 72.729954] ? trace_hardirqs_off_caller+0x5b/0x160
[ 72.730107] ? do_syscall_64+0x18/0x3c0
[ 72.730243] do_syscall_64+0x95/0x3c0
[ 72.730363] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 72.730508] RIP: 0033:0x7f6f1758fff8
[ 72.730624] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00
00 f3 0f 1e fa 48 8d 05 25 77 0d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4
55
[ 72.731146] RSP: 002b:00007fff99f99088 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 72.731365] RAX: ffffffffffffffda RBX: 00007fff99f99090 RCX: 00007f6f1758fff8
[ 72.731579] RDX: 0000000000000090 RSI: 00007fff99f99090 RDI: 0000000000000003
[ 72.731814] RBP: 0000564942bd8ec0 R08: 0000564942bd9180 R09: 0000000000000000
[ 72.732043] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 72.732262] R13: 0000000000000001 R14: 0000000000000000 R15: 00005649413cc470
[ 72.732494]
[ 72.732572] Allocated by task 381:
[ 72.732692] __kasan_kmalloc.constprop.5+0xc1/0xd0
[ 72.732857] cma_alloc_port+0x4d/0x160 [rdma_cm]
[ 72.733006] rdma_bind_addr+0x14e7/0x1b00 [rdma_cm]
[ 72.733153] ucma_bind+0x120/0x160 [rdma_ucm]
[ 72.733299] ucma_write+0x1f8/0x2b0 [rdma_ucm]
[ 72.733452] vfs_write+0x157/0x460
[ 72.733569] ksys_write+0xb8/0x170
[ 72.733675] do_syscall_64+0x95/0x3c0
[ 72.733800] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 72.733956]
[ 72.734029] Freed by task 381:
[ 72.734133] __kasan_slab_free+0x12e/0x180
[ 72.734284] kfree+0xed/0x290
[ 72.734399] rdma_destroy_id+0x6b6/0x9e0 [rdma_cm]
[ 72.734559] ucma_close+0x110/0x300 [rdma_ucm]
[ 72.734701] __fput+0x25a/0x740
[ 72.734832] task_work_run+0x10e/0x190
[ 72.734959] do_exit+0x85e/0x29e0
[ 72.735071] do_group_exit+0xf0/0x2e0
[ 72.735182] get_signal+0x2e0/0x17e0
[ 72.735304] do_signal+0x94/0x1570
[ 72.735424] exit_to_usermode_loop+0xfa/0x130
[ 72.735612] do_syscall_64+0x327/0x3c0
[ 72.735756] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 72.735941]
[ 72.736033] The buggy address belongs to the object at ffff888069fde990
[ 72.736033] which belongs to the cache kmalloc-32 of size 32
[ 72.736414] The buggy address is located 8 bytes inside of
[ 72.736414] 32-byte region [ffff888069fde990, ffff888069fde9b0)
[ 72.736777] The buggy address belongs to the page:
[ 72.736940] page:ffffea0001a7f780 count:1 mapcount:0 mapping:ffff88806bc03980 index:0x0
[ 72.737171] flags: 0x4000000000000200(slab)
[ 72.737295] raw: 4000000000000200 dead000000000100 dead000000000200 ffff88806bc03980
[ 72.737525] raw: 0000000000000000 0000000000550055 00000001ffffffff 0000000000000000
[ 72.737786] page dumped because: kasan: bad access detected
[ 72.737948]
[ 72.738019] Memory state around the buggy address:
[ 72.738164] ffff888069fde880: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
[ 72.738396] ffff888069fde900: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
[ 72.738627] >ffff888069fde980: fc fc fb fb fb fb fc fc fb fb fb fb fc fc fb fb
[ 72.738869] ^
[ 72.738999] ffff888069fdea00: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
[ 72.739213] ffff888069fdea80: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
[ 72.739431]
==================================================================
[ 72.739667] Disabling lock debugging due to kernel taint
>
> If you seem something obvious wrong in that patch please speak
> up.. Below is what I applied.
>
> Thanks,
> Jason
>
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 68c997be242930..a8b9c66c85252b 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -39,7 +39,7 @@
> #include <linux/mutex.h>
> #include <linux/random.h>
> #include <linux/igmp.h>
> -#include <linux/idr.h>
> +#include <linux/xarray.h>
> #include <linux/inetdevice.h>
> #include <linux/slab.h>
> #include <linux/module.h>
> @@ -191,10 +191,10 @@ static struct workqueue_struct *cma_wq;
> static unsigned int cma_pernet_id;
>
> struct cma_pernet {
> - struct idr tcp_ps;
> - struct idr udp_ps;
> - struct idr ipoib_ps;
> - struct idr ib_ps;
> + struct xarray tcp_ps;
> + struct xarray udp_ps;
> + struct xarray ipoib_ps;
> + struct xarray ib_ps;
> };
>
> static struct cma_pernet *cma_pernet(struct net *net)
> @@ -202,7 +202,8 @@ static struct cma_pernet *cma_pernet(struct net *net)
> return net_generic(net, cma_pernet_id);
> }
>
> -static struct idr *cma_pernet_idr(struct net *net, enum rdma_ucm_port_space ps)
> +static
> +struct xarray *cma_pernet_xa(struct net *net, enum rdma_ucm_port_space ps)
> {
> struct cma_pernet *pernet = cma_pernet(net);
>
> @@ -247,25 +248,25 @@ struct class_port_info_context {
> static int cma_ps_alloc(struct net *net, enum rdma_ucm_port_space ps,
> struct rdma_bind_list *bind_list, int snum)
> {
> - struct idr *idr = cma_pernet_idr(net, ps);
> + struct xarray *xa = cma_pernet_xa(net, ps);
>
> - return idr_alloc(idr, bind_list, snum, snum + 1, GFP_KERNEL);
> + return xa_insert(xa, snum, bind_list, GFP_KERNEL);
> }
>
> static struct rdma_bind_list *cma_ps_find(struct net *net,
> enum rdma_ucm_port_space ps, int snum)
> {
> - struct idr *idr = cma_pernet_idr(net, ps);
> + struct xarray *xa = cma_pernet_xa(net, ps);
>
> - return idr_find(idr, snum);
> + return xa_load(xa, snum);
> }
>
> static void cma_ps_remove(struct net *net, enum rdma_ucm_port_space ps,
> int snum)
> {
> - struct idr *idr = cma_pernet_idr(net, ps);
> + struct xarray *xa = cma_pernet_xa(net, ps);
>
> - idr_remove(idr, snum);
> + xa_erase(xa, snum);
> }
>
> enum {
> @@ -4655,10 +4656,10 @@ static int cma_init_net(struct net *net)
> {
> struct cma_pernet *pernet = cma_pernet(net);
>
> - idr_init(&pernet->tcp_ps);
> - idr_init(&pernet->udp_ps);
> - idr_init(&pernet->ipoib_ps);
> - idr_init(&pernet->ib_ps);
> + xa_init(&pernet->tcp_ps);
> + xa_init(&pernet->udp_ps);
> + xa_init(&pernet->ipoib_ps);
> + xa_init(&pernet->ib_ps);
>
> return 0;
> }
> @@ -4667,10 +4668,10 @@ static void cma_exit_net(struct net *net)
> {
> struct cma_pernet *pernet = cma_pernet(net);
>
> - idr_destroy(&pernet->tcp_ps);
> - idr_destroy(&pernet->udp_ps);
> - idr_destroy(&pernet->ipoib_ps);
> - idr_destroy(&pernet->ib_ps);
> + WARN_ON(!xa_empty(&pernet->tcp_ps));
> + WARN_ON(!xa_empty(&pernet->udp_ps));
> + WARN_ON(!xa_empty(&pernet->ipoib_ps));
> + WARN_ON(!xa_empty(&pernet->ib_ps));
> }
>
> static struct pernet_operations cma_pernet_operations = {
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: WARNING in cma_exit_net
2019-04-01 18:29 ` Leon Romanovsky
@ 2019-04-02 23:39 ` Bart Van Assche
2019-04-03 5:11 ` Leon Romanovsky
0 siblings, 1 reply; 5+ messages in thread
From: Bart Van Assche @ 2019-04-02 23:39 UTC (permalink / raw)
To: Leon Romanovsky, Jason Gunthorpe
Cc: syzbot, Matthew Wilcox, danielj, danitg, dledford, linux-kernel,
linux-rdma, parav, swise, syzkaller-bugs
On Mon, 2019-04-01 at 21:29 +0300, Leon Romanovsky wrote:
> On Mon, Apr 01, 2019 at 02:45:54PM -0300, Jason Gunthorpe wrote:
> > On Mon, Apr 01, 2019 at 10:36:05AM -0700, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following crash on:
> > >
> > > HEAD commit: e3ecb83e Add linux-next specific files for 20190401
> > > git tree: linux-next
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=13bc36cd200000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=db6c9f2bfeb91a99
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=2e3e485d5697ea610460
> > > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > >
> > > Unfortunately, I don't have any reproducer for this crash yet.
> > >
> > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > Reported-by: syzbot+2e3e485d5697ea610460@syzkaller.appspotmail.com
> > >
> > > WARNING: CPU: 1 PID: 7 at drivers/infiniband/core/cma.c:4674
> > > cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
> > > Kernel panic - not syncing: panic_on_warn set ...
> >
> > Matt: This is why the WARN_ON(!xa_empty()) is so valuable. Magically
> > syzkaller can find something in this code is buggy.
> >
> > Mellanox is also showing a different testing failure over the weekend
> > (use after free or something) from your 'cma: Convert portspace IDRs
> > to XArray'
>
> This is what I see in my environment.
>
> [ 72.725596]
> ==================================================================
> [ 72.726017] BUG: KASAN: use-after-free in cma_check_port+0x86a/0xa20 [rdma_cm]
> [ 72.726263] Read of size 8 at addr ffff888069fde998 by task ucmatose/387
> [ 72.726460]
> [ 72.726550] CPU: 3 PID: 387 Comm: ucmatose Not tainted 5.1.0-rc2+ #253
> [ 72.726751] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
> [ 72.727119] Call Trace:
> [ 72.727210] dump_stack+0x7c/0xc0
> [ 72.727342] print_address_description+0x6c/0x23c
> [ 72.727505] ? cma_check_port+0x86a/0xa20 [rdma_cm]
> [ 72.727666] kasan_report.cold.3+0x1c/0x35
> [ 72.727805] ? cma_check_port+0x86a/0xa20 [rdma_cm]
> [ 72.727977] ? cma_check_port+0x86a/0xa20 [rdma_cm]
> [ 72.728138] cma_check_port+0x86a/0xa20 [rdma_cm]
> [ 72.728306] rdma_bind_addr+0x11bc/0x1b00 [rdma_cm]
> [ 72.728465] ? find_held_lock+0x33/0x1c0
> [ 72.728597] ? cma_ndev_work_handler+0x180/0x180 [rdma_cm]
> [ 72.728756] ? wait_for_completion+0x3d0/0x3d0
> [ 72.728928] ucma_bind+0x120/0x160 [rdma_ucm]
> [ 72.729089] ? ucma_resolve_addr+0x1a0/0x1a0 [rdma_ucm]
> [ 72.729256] ucma_write+0x1f8/0x2b0 [rdma_ucm]
> [ 72.729409] ? ucma_open+0x260/0x260 [rdma_ucm]
> [ 72.729571] vfs_write+0x157/0x460
> [ 72.729688] ksys_write+0xb8/0x170
> [ 72.729828] ? __ia32_sys_read+0xb0/0xb0
> [ 72.729954] ? trace_hardirqs_off_caller+0x5b/0x160
> [ 72.730107] ? do_syscall_64+0x18/0x3c0
> [ 72.730243] do_syscall_64+0x95/0x3c0
> [ 72.730363] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 72.730508] RIP: 0033:0x7f6f1758fff8
> [ 72.730624] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00
> 00 f3 0f 1e fa 48 8d 05 25 77 0d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f
> 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4
> 55
> [ 72.731146] RSP: 002b:00007fff99f99088 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> [ 72.731365] RAX: ffffffffffffffda RBX: 00007fff99f99090 RCX: 00007f6f1758fff8
> [ 72.731579] RDX: 0000000000000090 RSI: 00007fff99f99090 RDI: 0000000000000003
> [ 72.731814] RBP: 0000564942bd8ec0 R08: 0000564942bd9180 R09: 0000000000000000
> [ 72.732043] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> [ 72.732262] R13: 0000000000000001 R14: 0000000000000000 R15: 00005649413cc470
> [ 72.732494]
> [ 72.732572] Allocated by task 381:
> [ 72.732692] __kasan_kmalloc.constprop.5+0xc1/0xd0
> [ 72.732857] cma_alloc_port+0x4d/0x160 [rdma_cm]
> [ 72.733006] rdma_bind_addr+0x14e7/0x1b00 [rdma_cm]
> [ 72.733153] ucma_bind+0x120/0x160 [rdma_ucm]
> [ 72.733299] ucma_write+0x1f8/0x2b0 [rdma_ucm]
> [ 72.733452] vfs_write+0x157/0x460
> [ 72.733569] ksys_write+0xb8/0x170
> [ 72.733675] do_syscall_64+0x95/0x3c0
> [ 72.733800] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 72.733956]
> [ 72.734029] Freed by task 381:
> [ 72.734133] __kasan_slab_free+0x12e/0x180
> [ 72.734284] kfree+0xed/0x290
> [ 72.734399] rdma_destroy_id+0x6b6/0x9e0 [rdma_cm]
> [ 72.734559] ucma_close+0x110/0x300 [rdma_ucm]
> [ 72.734701] __fput+0x25a/0x740
> [ 72.734832] task_work_run+0x10e/0x190
> [ 72.734959] do_exit+0x85e/0x29e0
> [ 72.735071] do_group_exit+0xf0/0x2e0
> [ 72.735182] get_signal+0x2e0/0x17e0
> [ 72.735304] do_signal+0x94/0x1570
> [ 72.735424] exit_to_usermode_loop+0xfa/0x130
> [ 72.735612] do_syscall_64+0x327/0x3c0
> [ 72.735756] entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [ 72.735941]
> [ 72.736033] The buggy address belongs to the object at ffff888069fde990
> [ 72.736033] which belongs to the cache kmalloc-32 of size 32
> [ 72.736414] The buggy address is located 8 bytes inside of
> [ 72.736414] 32-byte region [ffff888069fde990, ffff888069fde9b0)
> [ 72.736777] The buggy address belongs to the page:
> [ 72.736940] page:ffffea0001a7f780 count:1 mapcount:0 mapping:ffff88806bc03980 index:0x0
> [ 72.737171] flags: 0x4000000000000200(slab)
> [ 72.737295] raw: 4000000000000200 dead000000000100 dead000000000200 ffff88806bc03980
> [ 72.737525] raw: 0000000000000000 0000000000550055 00000001ffffffff 0000000000000000
> [ 72.737786] page dumped because: kasan: bad access detected
> [ 72.737948]
> [ 72.738019] Memory state around the buggy address:
> [ 72.738164] ffff888069fde880: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
> [ 72.738396] ffff888069fde900: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
> [ 72.738627] >ffff888069fde980: fc fc fb fb fb fb fc fc fb fb fb fb fc fc fb fb
> [ 72.738869] ^
> [ 72.738999] ffff888069fdea00: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
> [ 72.739213] ffff888069fdea80: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
> [ 72.739431]
> ==================================================================
> [ 72.739667] Disabling lock debugging due to kernel taint
This is what I encountered while running blktests:
nvmet: adding nsid 1 to subsystem nvme-test
==================================================================
BUG: KASAN: use-after-free in cma_check_port+0x28/0x400 [rdma_cm]
Read of size 8 at addr ffff8880ba96f818 by task ln/10510
CPU: 5 PID: 10510 Comm: ln Not tainted 5.1.0-rc3-dbg+ #9
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
Call Trace:
dump_stack+0x86/0xca
print_address_description+0x71/0x239
? cma_check_port+0x28/0x400 [rdma_cm]
kasan_report.cold.3+0x1b/0x3e
? cma_check_port+0x28/0x400 [rdma_cm]
__asan_load8+0x54/0x90
cma_check_port+0x28/0x400 [rdma_cm]
rdma_bind_addr+0xc13/0xe80 [rdma_cm]
? cma_ndev_work_handler+0xf0/0xf0 [rdma_cm]
? lockdep_hardirqs_on+0x185/0x260
? _raw_spin_unlock_irqrestore+0x57/0x70
? trace_hardirqs_on+0x24/0x130
? preempt_count_sub+0x18/0xd0
? _raw_spin_unlock_irqrestore+0x42/0x70
nvmet_rdma_add_port+0x143/0x1a0 [nvmet_rdma]
? nvmet_rdma_remove_port+0x40/0x40 [nvmet_rdma]
nvmet_enable_port+0x85/0x180 [nvmet]
nvmet_port_subsys_allow_link+0x1bc/0x1e0 [nvmet]
? do_raw_spin_unlock+0xa8/0x140
configfs_symlink+0x2b6/0x650
? configfs_get_link+0x3e0/0x3e0
? inode_permission+0x69/0x200
vfs_symlink+0x163/0x230
do_symlinkat+0xeb/0x160
? __ia32_sys_unlink+0x40/0x40
? do_syscall_64+0x19/0x210
? entry_SYSCALL_64_after_hwframe+0x49/0xbe
__x64_sys_symlinkat+0x43/0x50
do_syscall_64+0x71/0x210
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f35b984d9e7
Code: 73 01 c3 48 8b 0d a9 84 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 0a 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 79 84 0d 00 f7 d8 64 89 01 48
RSP: 002b:00007ffcc3a88af8 EFLAGS: 00000246 ORIG_RAX: 000000000000010a
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f35b984d9e7
RDX: 000055c1855f82b0 RSI: 00000000ffffff9c RDI: 00007ffcc3a8a7a6
RBP: 00000000ffffff9c R08: 000055c1855f8010 R09: 0000000000000000
R10: fffffffffffff000 R11: 0000000000000246 R12: 000055c1855f82b0
R13: 00007ffcc3a8a7a6 R14: 0000000000000000 R15: 0000000000000000
Allocated by task 10270:
save_stack+0x43/0xd0
__kasan_kmalloc.constprop.9+0xc7/0xd0
kasan_kmalloc+0x9/0x10
kmem_cache_alloc_trace+0x143/0x350
cma_alloc_port+0x3d/0xf0 [rdma_cm]
rdma_bind_addr+0xdf9/0xe80 [rdma_cm]
nvmet_clear_ctrl+0x43/0x70 [nvmet]
rxe_opcode+0x15f5/0xfffffffffffef380 [rdma_rxe]
rxe_wr_opcode_info+0xa4c/0xfffffffffffeb360 [rdma_rxe]
configfs_symlink+0x2b6/0x650
vfs_symlink+0x163/0x230
do_symlinkat+0xeb/0x160
__x64_sys_symlinkat+0x43/0x50
do_syscall_64+0x71/0x210
entry_SYSCALL_64_after_hwframe+0x49/0xbe
Freed by task 10340:
save_stack+0x43/0xd0
__kasan_slab_free+0x139/0x190
kasan_slab_free+0xe/0x10
kfree+0x103/0x320
rdma_destroy_id+0x42c/0x460 [rdma_cm]
nvmet_ctrl_fatal_error+0x31/0x80 [nvmet]
rxe_opcode+0x175d/0xfffffffffffef380 [rdma_rxe]
rxe_wr_opcode_info+0x644/0xfffffffffffeb360 [rdma_rxe]
configfs_unlink+0x216/0x350
vfs_unlink+0x171/0x260
do_unlinkat+0x347/0x490
__x64_sys_unlinkat+0x60/0x90
do_syscall_64+0x71/0x210
entry_SYSCALL_64_after_hwframe+0x49/0xbe
The buggy address belongs to the object at ffff8880ba96f810
which belongs to the cache kmalloc-32 of size 32
The buggy address is located 8 bytes inside of
32-byte region [ffff8880ba96f810, ffff8880ba96f830)
The buggy address belongs to the page:
page:ffffea0002ea5bc0 count:1 mapcount:0 mapping:ffff88811b003800 index:0x0
flags: 0x1fff000000000200(slab)
raw: 1fff000000000200 ffffea0004319200 0000000900000009 ffff88811b003800
raw: 0000000000000000 0000000000550055 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff8880ba96f700: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
ffff8880ba96f780: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
>ffff8880ba96f800: fc fc fb fb fb fb fc fc fb fb fb fb fc fc fb fb
^
ffff8880ba96f880: fb fb fc fc fb fb fb fb fc fc fb fb fb fb fc fc
ffff8880ba96f900: fb fb fb fb fc fc fb fb fb fb fc fc fb fb fb fb
==================================================================
Disabling lock debugging due to kernel taint
nvmet_rdma: enabling port 1 (192.168.122.49:7777)
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: WARNING in cma_exit_net
2019-04-02 23:39 ` Bart Van Assche
@ 2019-04-03 5:11 ` Leon Romanovsky
0 siblings, 0 replies; 5+ messages in thread
From: Leon Romanovsky @ 2019-04-03 5:11 UTC (permalink / raw)
To: Bart Van Assche
Cc: Jason Gunthorpe, syzbot, Matthew Wilcox, danielj, danitg,
dledford, linux-kernel, linux-rdma, parav, swise, syzkaller-bugs
[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]
On Tue, Apr 02, 2019 at 04:39:12PM -0700, Bart Van Assche wrote:
> On Mon, 2019-04-01 at 21:29 +0300, Leon Romanovsky wrote:
> > On Mon, Apr 01, 2019 at 02:45:54PM -0300, Jason Gunthorpe wrote:
> > > On Mon, Apr 01, 2019 at 10:36:05AM -0700, syzbot wrote:
> > > > Hello,
> > > >
> > > > syzbot found the following crash on:
> > > >
> > > > HEAD commit: e3ecb83e Add linux-next specific files for 20190401
> > > > git tree: linux-next
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=13bc36cd200000
> > > > kernel config: https://syzkaller.appspot.com/x/.config?x=db6c9f2bfeb91a99
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=2e3e485d5697ea610460
> > > > compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> > > >
> > > > Unfortunately, I don't have any reproducer for this crash yet.
> > > >
> > > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > > Reported-by: syzbot+2e3e485d5697ea610460@syzkaller.appspotmail.com
> > > >
> > > > WARNING: CPU: 1 PID: 7 at drivers/infiniband/core/cma.c:4674
> > > > cma_exit_net+0x327/0x390 drivers/infiniband/core/cma.c:4674
> > > > Kernel panic - not syncing: panic_on_warn set ...
> > >
> > > Matt: This is why the WARN_ON(!xa_empty()) is so valuable. Magically
> > > syzkaller can find something in this code is buggy.
> > >
> > > Mellanox is also showing a different testing failure over the weekend
> > > (use after free or something) from your 'cma: Convert portspace IDRs
> > > to XArray'
> >
> > This is what I see in my environment.
Bart,
This patch [1] will fix it.
Thanks
[1] https://patchwork.kernel.org/patch/10881501/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-04-03 5:11 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-01 17:36 WARNING in cma_exit_net syzbot
2019-04-01 17:45 ` Jason Gunthorpe
2019-04-01 18:29 ` Leon Romanovsky
2019-04-02 23:39 ` Bart Van Assche
2019-04-03 5:11 ` Leon Romanovsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).