KASAN: use-after-free Read in __rhashtable_lookup (2)

KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    ad1d69735878 Merge tag 'fuse-update-4.19' of git://git.ker..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=100e3b36400000
kernel config:  https://syzkaller.appspot.com/x/.config?x=6a24c33b9ed66aa
dashboard link: https://syzkaller.appspot.com/bug?extid=8967084bcac563795dc6
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+8967084bcac563795dc6@syzkaller.appspotmail.com

==================================================================
BUG: KASAN: use-after-free in memcmp+0xe8/0x150 lib/string.c:861
Read of size 1 at addr ffff8801b5201bf0 by task syz-executor5/14889

CPU: 0 PID: 14889 Comm: syz-executor5 Not tainted 4.18.0+ #202
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113
  print_address_description+0x6c/0x20b mm/kasan/report.c:256
  kasan_report_error mm/kasan/report.c:354 [inline]
  kasan_report.cold.7+0x242/0x30d mm/kasan/report.c:412
  __asan_report_load1_noabort+0x14/0x20 mm/kasan/report.c:430
  memcmp+0xe8/0x150 lib/string.c:861
  memcmp include/linux/string.h:386 [inline]
  rhashtable_compare include/linux/rhashtable.h:462 [inline]
  __rhashtable_lookup.isra.8.constprop.19+0x70f/0xd50  
include/linux/rhashtable.h:484
  rhashtable_lookup include/linux/rhashtable.h:516 [inline]
  rhashtable_lookup_fast include/linux/rhashtable.h:542 [inline]
  rds_add_bound net/rds/bind.c:117 [inline]
  rds_bind+0x7c9/0x1510 net/rds/bind.c:238
  __sys_bind+0x331/0x440 net/socket.c:1481
  __do_sys_bind net/socket.c:1492 [inline]
  __se_sys_bind net/socket.c:1490 [inline]
  __x64_sys_bind+0x73/0xb0 net/socket.c:1490
  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x457089
Code: fd b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 cb b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fdf40632c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 00007fdf406336d4 RCX: 0000000000457089
RDX: 0000000000000010 RSI: 0000000020000840 RDI: 0000000000000009
RBP: 00000000009300a0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000004caff8 R14: 00000000004c2de7 R15: 0000000000000000

Allocated by task 14887:
  save_stack+0x43/0xd0 mm/kasan/kasan.c:448
  set_track mm/kasan/kasan.c:460 [inline]
  kasan_kmalloc+0xc4/0xe0 mm/kasan/kasan.c:553
  kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490
  kmem_cache_alloc+0x12e/0x710 mm/slab.c:3554
  sk_prot_alloc+0x69/0x2e0 net/core/sock.c:1462
  sk_alloc+0x106/0x17b0 net/core/sock.c:1522
  rds_create+0x14f/0x770 net/rds/af_rds.c:666
  __sock_create+0x53c/0x940 net/socket.c:1275
  sock_create net/socket.c:1315 [inline]
  __sys_socket+0x106/0x260 net/socket.c:1345
  __do_sys_socket net/socket.c:1354 [inline]
  __se_sys_socket net/socket.c:1352 [inline]
  __x64_sys_socket+0x73/0xb0 net/socket.c:1352
  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

Freed by task 14886:
  save_stack+0x43/0xd0 mm/kasan/kasan.c:448
  set_track mm/kasan/kasan.c:460 [inline]
  __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521
  kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
  __cache_free mm/slab.c:3498 [inline]
  kmem_cache_free+0x86/0x280 mm/slab.c:3756
  sk_prot_free net/core/sock.c:1503 [inline]
  __sk_destruct+0x70a/0xa60 net/core/sock.c:1587
  sk_destruct+0x78/0x90 net/core/sock.c:1595
  __sk_free+0xcf/0x300 net/core/sock.c:1606
  sk_free+0x42/0x50 net/core/sock.c:1617
  sock_put include/net/sock.h:1691 [inline]
  rds_release+0x3e9/0x570 net/rds/af_rds.c:91
  __sock_release+0xd7/0x250 net/socket.c:579
  sock_close+0x19/0x20 net/socket.c:1139
  __fput+0x36e/0x8c0 fs/file_table.c:278
  ____fput+0x15/0x20 fs/file_table.c:309
  task_work_run+0x1e8/0x2a0 kernel/task_work.c:113
  tracehook_notify_resume include/linux/tracehook.h:193 [inline]
  exit_to_usermode_loop+0x318/0x380 arch/x86/entry/common.c:166
  prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline]
  syscall_return_slowpath arch/x86/entry/common.c:268 [inline]
  do_syscall_64+0x6be/0x820 arch/x86/entry/common.c:293
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

The buggy address belongs to the object at ffff8801b5201780
  which belongs to the cache RDS(76:syz7) of size 1608
The buggy address is located 1136 bytes inside of
  1608-byte region [ffff8801b5201780, ffff8801b5201dc8)
The buggy address belongs to the page:
page:ffffea0006d48040 count:1 mapcount:0 mapping:ffff8801d75e2c80 index:0x0
flags: 0x2fffc0000000100(slab)
raw: 02fffc0000000100 ffffea0006e74188 ffff8801ca20ef48 ffff8801d75e2c80
raw: 0000000000000000 ffff8801b5201080 0000000100000002 ffff8801acaa4640
page dumped because: kasan: bad access detected
page->mem_cgroup:ffff8801acaa4640

Memory state around the buggy address:
  ffff8801b5201a80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8801b5201b00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8801b5201b80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                                              ^
  ffff8801b5201c00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8801b5201c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with  
syzbot.

Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[Sowmini Varadhan]

#syz dup KASAN: use-after-free Read in rds_find_bound

The problem is a missing synchronize_net() in rds_release.
References:
  https://www.spinics.net/lists/netdev/msg475074.html
  https://www.spinics.net/lists/netdev/msg507908.html

Re: Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

> #syz dup KASAN: use-after-free Read in rds_find_bound

unknown command "dup"


> The problem is a missing synchronize_net() in rds_release.
> References:
>    https://www.spinics.net/lists/netdev/msg475074.html
>    https://www.spinics.net/lists/netdev/msg507908.html

Re: Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

> #syz dup KASAN: use-after-free Read in rds_find_bound

unknown command "dup"


> The problem is a missing synchronize_net() in rds_release.
> References:
>    https://www.spinics.net/lists/netdev/msg475074.html
>    https://www.spinics.net/lists/netdev/msg507908.html

Re: Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

> #syz dup KASAN: use-after-free Read in rds_find_bound

unknown command "dup"


> The problem is a missing synchronize_net() in rds_release.
> References:
>    https://www.spinics.net/lists/netdev/msg475074.html
>    https://www.spinics.net/lists/netdev/msg507908.html

> --
> You received this message because you are subscribed to the Google  
> Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit  
> https://groups.google.com/d/msgid/syzkaller-bugs/20180824230446.GA16891%40oracle.com.
> For more options, visit https://groups.google.com/d/optout.

Re: Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

> #syz dup KASAN: use-after-free Read in rds_find_bound

unknown command "dup"


> The problem is a missing synchronize_net() in rds_release.
> References:
>    https://www.spinics.net/lists/netdev/msg475074.html
>    https://www.spinics.net/lists/netdev/msg507908.html

> --
> You received this message because you are subscribed to the Google  
> Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit  
> https://groups.google.com/d/msgid/syzkaller-bugs/20180824230446.GA16891%40oracle.com.
> For more options, visit https://groups.google.com/d/optout.

Re: Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[Dmitry Vyukov]

On Fri, Aug 24, 2018 at 4:05 PM, syzbot
<syzbot+8967084bcac563795dc6@syzkaller.appspotmail.com> wrote:
>
>
>> #syz dup KASAN: use-after-free Read in rds_find_bound
>
>
> unknown command "dup"


You need a colon after dup.

But this can't possibly be a dup of that bug. That bug was fixed in February:
https://syzkaller.appspot.com/bug?extid=93a5839deb355537440f

And this crash happened a day ago. This is a different bug. If you dup
it to a closed one, this one will also be considered closed, and then
a new crash will immediately create a new bug.



>> The problem is a missing synchronize_net() in rds_release.
>> References:
>>    https://www.spinics.net/lists/netdev/msg475074.html
>>    https://www.spinics.net/lists/netdev/msg507908.html
>
>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/syzkaller-bugs/20180824230446.GA16891%40oracle.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000d8879a05743667a2%40google.com.
>
> For more options, visit https://groups.google.com/d/optout.

Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[Sowmini Varadhan]

On (08/24/18 16:10), Dmitry Vyukov wrote:
> You need a colon after dup.

I see.

> But this can't possibly be a dup of that bug. That bug was fixed in February:

Apologies, I unknowingly made a mistake in my 
syzbot-tiquette in that case,

I did not intend to *close* the bug as a dup, I was
merely trying to indicate that this is yet another 
manifestation of the original report with subject
"KASAN: use-after-free Read in rds_find_bound"

All of these problems are happening because we are missing
a synchronize_net() in rds_release.

A full description of the race conditon can be found in
https://www.spinics.net/lists/netdev/msg475074.html

If rds_release is going to nuke the rds_socket
it needs to sychronize_net() with other threads that may be
looking up the bind_hash_table. If we dont do that,
all of these "use-after-free" races can happen.
As long as we dont have the synchronize_net() we are
missing a big (and standard) bit of RCU synchronization.

--Sowmini

Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[Dmitry Vyukov]

On Fri, Aug 24, 2018 at 4:37 PM, Sowmini Varadhan
<sowmini.varadhan@oracle.com> wrote:
> On (08/24/18 16:10), Dmitry Vyukov wrote:
>> You need a colon after dup.
>
> I see.
>
>> But this can't possibly be a dup of that bug. That bug was fixed in February:
>
> Apologies, I unknowingly made a mistake in my
> syzbot-tiquette in that case,
>
> I did not intend to *close* the bug as a dup, I was
> merely trying to indicate that this is yet another
> manifestation of the original report with subject
> "KASAN: use-after-free Read in rds_find_bound"
>
> All of these problems are happening because we are missing
> a synchronize_net() in rds_release.
>
> A full description of the race conditon can be found in
> https://www.spinics.net/lists/netdev/msg475074.html
>
> If rds_release is going to nuke the rds_socket
> it needs to sychronize_net() with other threads that may be
> looking up the bind_hash_table. If we dont do that,
> all of these "use-after-free" races can happen.
> As long as we dont have the synchronize_net() we are
> missing a big (and standard) bit of RCU synchronization.


If it's a note for humans, then just a plain English comment will do.

Re: KASAN: use-after-free Read in __rhashtable_lookup (2)

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit:    d7b686ebf704 Merge branch 'i2c/for-current' of git://git.k..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1132d70a400000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8f59875069d721b6
dashboard link: https://syzkaller.appspot.com/bug?extid=8967084bcac563795dc6
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10b67e49400000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=11119e49400000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+8967084bcac563795dc6@syzkaller.appspotmail.com

==================================================================
BUG: KASAN: use-after-free in memcmp+0xe3/0x160 lib/string.c:861
Read of size 1 at addr ffff8801ce73eb70 by task syz-executor383/11736

CPU: 1 PID: 11736 Comm: syz-executor383 Not tainted 4.19.0-rc2+ #228
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
  print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
  kasan_report_error mm/kasan/report.c:354 [inline]
  kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
  __asan_report_load1_noabort+0x14/0x20 mm/kasan/report.c:430
  memcmp+0xe3/0x160 lib/string.c:861
  memcmp include/linux/string.h:386 [inline]
  rhashtable_compare include/linux/rhashtable.h:462 [inline]
  __rhashtable_lookup.isra.8.constprop.20+0x73a/0xd00  
include/linux/rhashtable.h:484
  rhashtable_lookup include/linux/rhashtable.h:516 [inline]
  rhashtable_lookup_fast include/linux/rhashtable.h:542 [inline]
  rds_add_bound net/rds/bind.c:117 [inline]
  rds_bind+0x7d2/0x1520 net/rds/bind.c:238
  __sys_bind+0x331/0x440 net/socket.c:1481
  __do_sys_bind net/socket.c:1492 [inline]
  __se_sys_bind net/socket.c:1490 [inline]
  __x64_sys_bind+0x73/0xb0 net/socket.c:1490
  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x444e29
Code: e8 ac e8 ff ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 2b ce fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffec4b12988 EFLAGS: 00000217 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 0000000000444e29
RDX: 0000000000000010 RSI: 00000000200002c0 RDI: 0000000000000006
RBP: 0000000000000000 R08: 00000000004002e0 R09: 00000000004002e0
R10: 0000000000000004 R11: 0000000000000217 R12: 00000000000333a3
R13: 0000000000402170 R14: 0000000000000000 R15: 0000000000000000

Allocated by task 11738:
  save_stack+0x43/0xd0 mm/kasan/kasan.c:448
  set_track mm/kasan/kasan.c:460 [inline]
  kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
  kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490
  kmem_cache_alloc+0x12e/0x730 mm/slab.c:3554
  sk_prot_alloc+0x69/0x2e0 net/core/sock.c:1462
  sk_alloc+0x10d/0x1690 net/core/sock.c:1522
  rds_create+0x14f/0x740 net/rds/af_rds.c:666
  __sock_create+0x536/0x930 net/socket.c:1275
  sock_create net/socket.c:1315 [inline]
  __sys_socket+0x106/0x260 net/socket.c:1345
  __do_sys_socket net/socket.c:1354 [inline]
  __se_sys_socket net/socket.c:1352 [inline]
  __x64_sys_socket+0x73/0xb0 net/socket.c:1352
  do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

Freed by task 11738:
  save_stack+0x43/0xd0 mm/kasan/kasan.c:448
  set_track mm/kasan/kasan.c:460 [inline]
  __kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
  kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
  __cache_free mm/slab.c:3498 [inline]
  kmem_cache_free+0x83/0x290 mm/slab.c:3756
  sk_prot_free net/core/sock.c:1503 [inline]
  __sk_destruct+0x766/0xbd0 net/core/sock.c:1587
  sk_destruct+0x78/0x90 net/core/sock.c:1595
  __sk_free+0xcf/0x300 net/core/sock.c:1606
  sk_free+0x42/0x50 net/core/sock.c:1617
  sock_put include/net/sock.h:1691 [inline]
  rds_release+0x3e8/0x570 net/rds/af_rds.c:91
  __sock_release+0xd7/0x250 net/socket.c:579
  sock_close+0x19/0x20 net/socket.c:1139
  __fput+0x385/0xa30 fs/file_table.c:278
  ____fput+0x15/0x20 fs/file_table.c:309
  task_work_run+0x1e8/0x2a0 kernel/task_work.c:113
  tracehook_notify_resume include/linux/tracehook.h:193 [inline]
  exit_to_usermode_loop+0x318/0x380 arch/x86/entry/common.c:166
  prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline]
  syscall_return_slowpath arch/x86/entry/common.c:268 [inline]
  do_syscall_64+0x6be/0x820 arch/x86/entry/common.c:293
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

The buggy address belongs to the object at ffff8801ce73e700
  which belongs to the cache RDS of size 1608
The buggy address is located 1136 bytes inside of
  1608-byte region [ffff8801ce73e700, ffff8801ce73ed48)
The buggy address belongs to the page:
page:ffffea000739cf80 count:1 mapcount:0 mapping:ffff8801cb1c0c40 index:0x0
flags: 0x2fffc0000000100(slab)
raw: 02fffc0000000100 ffffea00073a5348 ffffea000738ea48 ffff8801cb1c0c40
raw: 0000000000000000 ffff8801ce73e000 0000000100000002 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
  ffff8801ce73ea00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8801ce73ea80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8801ce73eb00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                                              ^
  ffff8801ce73eb80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff8801ce73ec00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================