KASAN: use-after-free Read in rdma_listen (2)

KASAN: use-after-free Read in rdma_listen (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    9936328b Merge tag 'pci-v5.1-fixes-1' of git://git.kernel...
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=16380473200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8dcdce25ea72bedf
dashboard link: https://syzkaller.appspot.com/bug?extid=adb15cf8c2798e4e0db4
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=11580c3f200000

Bisection is inconclusive: the bug happens on the oldest tested release.

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14b80c3f200000
final crash:    https://syzkaller.appspot.com/x/report.txt?x=16b80c3f200000
console output: https://syzkaller.appspot.com/x/log.txt?x=12b80c3f200000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+adb15cf8c2798e4e0db4@syzkaller.appspotmail.com

==================================================================
BUG: KASAN: use-after-free in __list_add_valid+0x9a/0xa0 lib/list_debug.c:26
Read of size 8 at addr ffff8880a5d2b3e0 by task syz-executor.0/7797

CPU: 1 PID: 7797 Comm: syz-executor.0 Not tainted 5.1.0-rc2+ #43
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x172/0x1f0 lib/dump_stack.c:113
  print_address_description.cold+0x7c/0x20d mm/kasan/report.c:187
  kasan_report.cold+0x1b/0x40 mm/kasan/report.c:317
  __asan_report_load8_noabort+0x14/0x20 mm/kasan/generic_report.c:132
  __list_add_valid+0x9a/0xa0 lib/list_debug.c:26
  __list_add include/linux/list.h:60 [inline]
  list_add_tail include/linux/list.h:93 [inline]
  cma_listen_on_all drivers/infiniband/core/cma.c:2483 [inline]
  rdma_listen+0x6b7/0x970 drivers/infiniband/core/cma.c:3537
  ucma_listen+0x14d/0x1c0 drivers/infiniband/core/ucma.c:1100
  ucma_write+0x2da/0x3c0 drivers/infiniband/core/ucma.c:1696
  __vfs_write+0x8d/0x110 fs/read_write.c:485
  vfs_write+0x20c/0x580 fs/read_write.c:549
  ksys_write+0xea/0x1f0 fs/read_write.c:598
  __do_sys_write fs/read_write.c:610 [inline]
  __se_sys_write fs/read_write.c:607 [inline]
  __x64_sys_write+0x73/0xb0 fs/read_write.c:607
  do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x458209
Code: ad b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 7b b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fb9c2080c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000458209
RDX: 0000000000000010 RSI: 00000000200001c0 RDI: 0000000000000003
RBP: 000000000073bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fb9c20816d4
R13: 00000000004c77c2 R14: 00000000004dd780 R15: 00000000ffffffff

Allocated by task 7794:
  save_stack+0x45/0xd0 mm/kasan/common.c:75
  set_track mm/kasan/common.c:87 [inline]
  __kasan_kmalloc mm/kasan/common.c:497 [inline]
  __kasan_kmalloc.constprop.0+0xcf/0xe0 mm/kasan/common.c:470
  kasan_kmalloc+0x9/0x10 mm/kasan/common.c:511
  kmem_cache_alloc_trace+0x151/0x760 mm/slab.c:3621
  kmalloc include/linux/slab.h:545 [inline]
  kzalloc include/linux/slab.h:740 [inline]
  __rdma_create_id+0x5f/0x4e0 drivers/infiniband/core/cma.c:878
  ucma_create_id+0x1de/0x640 drivers/infiniband/core/ucma.c:506
  ucma_write+0x2da/0x3c0 drivers/infiniband/core/ucma.c:1696
  __vfs_write+0x8d/0x110 fs/read_write.c:485
  vfs_write+0x20c/0x580 fs/read_write.c:549
  ksys_write+0xea/0x1f0 fs/read_write.c:598
  __do_sys_write fs/read_write.c:610 [inline]
  __se_sys_write fs/read_write.c:607 [inline]
  __x64_sys_write+0x73/0xb0 fs/read_write.c:607
  do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

Freed by task 7789:
  save_stack+0x45/0xd0 mm/kasan/common.c:75
  set_track mm/kasan/common.c:87 [inline]
  __kasan_slab_free+0x102/0x150 mm/kasan/common.c:459
  kasan_slab_free+0xe/0x10 mm/kasan/common.c:467
  __cache_free mm/slab.c:3498 [inline]
  kfree+0xcf/0x230 mm/slab.c:3821
  rdma_destroy_id+0x719/0xaa0 drivers/infiniband/core/cma.c:1852
  ucma_close+0x115/0x320 drivers/infiniband/core/ucma.c:1777
  __fput+0x2e5/0x8d0 fs/file_table.c:278
  ____fput+0x16/0x20 fs/file_table.c:309
  task_work_run+0x14a/0x1c0 kernel/task_work.c:113
  tracehook_notify_resume include/linux/tracehook.h:188 [inline]
  exit_to_usermode_loop+0x273/0x2c0 arch/x86/entry/common.c:166
  prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline]
  syscall_return_slowpath arch/x86/entry/common.c:268 [inline]
  do_syscall_64+0x52d/0x610 arch/x86/entry/common.c:293
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

The buggy address belongs to the object at ffff8880a5d2b200
  which belongs to the cache kmalloc-2k of size 2048
The buggy address is located 480 bytes inside of
  2048-byte region [ffff8880a5d2b200, ffff8880a5d2ba00)
The buggy address belongs to the page:
page:ffffea0002974a80 count:1 mapcount:0 mapping:ffff88812c3f0c40 index:0x0  
compound_mapcount: 0
flags: 0x1fffc0000010200(slab|head)
raw: 01fffc0000010200 ffffea000231da88 ffff88812c3f1948 ffff88812c3f0c40
raw: 0000000000000000 ffff8880a5d2a100 0000000100000003 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
  ffff8880a5d2b280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8880a5d2b300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8880a5d2b380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                                        ^
  ffff8880a5d2b400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8880a5d2b480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: KASAN: use-after-free Read in rdma_listen (2)

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit:    c25a951c Add linux-next specific files for 20200217
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=10df082de00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c727d8fc485ff049
dashboard link: https://syzkaller.appspot.com/bug?extid=adb15cf8c2798e4e0db4
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=112b9d6ee00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=147abb11e00000

Bisection is inconclusive: the bug happens on the oldest tested release.

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14b80c3f200000
final crash:    https://syzkaller.appspot.com/x/report.txt?x=16b80c3f200000
console output: https://syzkaller.appspot.com/x/log.txt?x=12b80c3f200000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+adb15cf8c2798e4e0db4@syzkaller.appspotmail.com

==================================================================
BUG: KASAN: use-after-free in __list_add_valid+0x9a/0xa0 lib/list_debug.c:26
Read of size 8 at addr ffff888093bbb1e0 by task syz-executor570/10159

CPU: 1 PID: 10159 Comm: syz-executor570 Not tainted 5.6.0-rc2-next-20200217-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 print_address_description.constprop.0.cold+0xd4/0x30b mm/kasan/report.c:374
 __kasan_report.cold+0x1b/0x32 mm/kasan/report.c:506
 kasan_report+0x12/0x20 mm/kasan/common.c:641
 __asan_report_load8_noabort+0x14/0x20 mm/kasan/generic_report.c:135
 __list_add_valid+0x9a/0xa0 lib/list_debug.c:26
 __list_add include/linux/list.h:67 [inline]
 list_add_tail include/linux/list.h:100 [inline]
 cma_listen_on_all drivers/infiniband/core/cma.c:2517 [inline]
 rdma_listen+0x6b7/0x970 drivers/infiniband/core/cma.c:3628
 ucma_listen+0x14d/0x1c0 drivers/infiniband/core/ucma.c:1092
 ucma_write+0x2d7/0x3c0 drivers/infiniband/core/ucma.c:1684
 __vfs_write+0x8a/0x110 fs/read_write.c:494
 vfs_write+0x268/0x5d0 fs/read_write.c:558
 ksys_write+0x220/0x290 fs/read_write.c:611
 __do_sys_write fs/read_write.c:623 [inline]
 __se_sys_write fs/read_write.c:620 [inline]
 __x64_sys_write+0x73/0xb0 fs/read_write.c:620
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x446a69
Code: e8 5c b3 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 0b 08 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fdd8433eda8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00000000006dbc28 RCX: 0000000000446a69
RDX: 0000000000000010 RSI: 00000000200001c0 RDI: 0000000000000004
RBP: 00000000006dbc20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dbc2c
R13: 0000000000000000 R14: 0000000000000000 R15: 20c49ba5e353f7cf

Allocated by task 10155:
 save_stack+0x23/0x90 mm/kasan/common.c:72
 set_track mm/kasan/common.c:80 [inline]
 __kasan_kmalloc mm/kasan/common.c:515 [inline]
 __kasan_kmalloc.constprop.0+0xcf/0xe0 mm/kasan/common.c:488
 kasan_kmalloc+0x9/0x10 mm/kasan/common.c:529
 kmem_cache_alloc_trace+0x158/0x790 mm/slab.c:3551
 kmalloc include/linux/slab.h:555 [inline]
 kzalloc include/linux/slab.h:669 [inline]
 __rdma_create_id+0x5e/0x870 drivers/infiniband/core/cma.c:861
 ucma_create_id+0x1de/0x620 drivers/infiniband/core/ucma.c:501
 ucma_write+0x2d7/0x3c0 drivers/infiniband/core/ucma.c:1684
 __vfs_write+0x8a/0x110 fs/read_write.c:494
 vfs_write+0x268/0x5d0 fs/read_write.c:558
 ksys_write+0x220/0x290 fs/read_write.c:611
 __do_sys_write fs/read_write.c:623 [inline]
 __se_sys_write fs/read_write.c:620 [inline]
 __x64_sys_write+0x73/0xb0 fs/read_write.c:620
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe

Freed by task 10157:
 save_stack+0x23/0x90 mm/kasan/common.c:72
 set_track mm/kasan/common.c:80 [inline]
 kasan_set_free_info mm/kasan/common.c:337 [inline]
 __kasan_slab_free+0x102/0x150 mm/kasan/common.c:476
 kasan_slab_free+0xe/0x10 mm/kasan/common.c:485
 __cache_free mm/slab.c:3426 [inline]
 kfree+0x10a/0x2c0 mm/slab.c:3757
 rdma_destroy_id+0x7c6/0xdd0 drivers/infiniband/core/cma.c:1866
 ucma_close+0x115/0x310 drivers/infiniband/core/ucma.c:1762
 __fput+0x2ff/0x890 fs/file_table.c:280
 ____fput+0x16/0x20 fs/file_table.c:313
 task_work_run+0x145/0x1c0 kernel/task_work.c:113
 exit_task_work include/linux/task_work.h:22 [inline]
 do_exit+0xbcb/0x3030 kernel/exit.c:802
 do_group_exit+0x135/0x360 kernel/exit.c:900
 get_signal+0x47c/0x24f0 kernel/signal.c:2734
 do_signal+0x87/0x1700 arch/x86/kernel/signal.c:813
 exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
 prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
 syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
 do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
 entry_SYSCALL_64_after_hwframe+0x49/0xbe

The buggy address belongs to the object at ffff888093bbb000
 which belongs to the cache kmalloc-2k of size 2048
The buggy address is located 480 bytes inside of
 2048-byte region [ffff888093bbb000, ffff888093bbb800)
The buggy address belongs to the page:
page:ffffea00024eeec0 refcount:1 mapcount:0 mapping:00000000f8d67f88 index:0x0
flags: 0xfffe0000000200(slab)
raw: 00fffe0000000200 ffffea00026b2908 ffffea00024eee88 ffff8880aa400e00
raw: 0000000000000000 ffff888093bbb000 0000000100000001 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
 ffff888093bbb080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff888093bbb100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888093bbb180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                                       ^
 ffff888093bbb200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff888093bbb280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

Re: KASAN: use-after-free Read in rdma_listen (2)

[Jason Gunthorpe]

On Tue, Feb 18, 2020 at 08:27:17PM +0800, Hillf Danton wrote:
> Check if rdma is being reclaimed before listening on device while
> reclaimer is waiting for rdma to become quiesce.

This is the usual syzkaller bug in rdma_cm

The test causes rdma_resolve_addr() and rdma_listen() to run
concurrently.

There is no sane locking, so in turn this causes invariants to become
violated, in particular, in rdma_listen() we can have !id->device
but also !cma_any_addr(cma_src_addr(id_priv).

This causes cma_listen_on_all() to wrongly be called and because the
invariant is screwed up cma_cancel_listens() doesn't undo it.

Thus we fail to list_del id_priv->list from the listen_any_list and
the next manipulation of the list gets a use-after on the list member
which was now freed.

The fix is the same as all the others, add some kind of locking
instead of all this defective cma_comp_exch() thing..

Jason

Re: KASAN: use-after-free Read in rdma_listen (2)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger crash:

Reported-and-tested-by: syzbot+adb15cf8c2798e4e0db4@syzkaller.appspotmail.com

Tested on:

commit:         11a48a5a Linux 5.6-rc2
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-rc
kernel config:  https://syzkaller.appspot.com/x/.config?x=3e5684f9a45838bb
dashboard link: https://syzkaller.appspot.com/bug?extid=adb15cf8c2798e4e0db4
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
patch:          https://syzkaller.appspot.com/x/patch.diff?x=14709845e00000

Note: testing is done by a robot and is best-effort only.