* KASAN: use-after-free Read in tcp_write_timer_handler
@ 2021-09-22 1:43 Hao Sun
2021-09-22 1:56 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Hao Sun @ 2021-09-22 1:43 UTC (permalink / raw)
To: davem, dsahern, Eric Dumazet, kuba, netdev, yoshfuji
Cc: andrii, ast, bpf, daniel, john.fastabend, kafai,
Linux Kernel Mailing List, songliubraving, yhs
Hello,
When using Healer to fuzz the latest Linux kernel, the following crash
was triggered.
HEAD commit: 4357f03d6611 Merge tag 'pm-5.15-rc2
git tree: upstream
console output:
https://drive.google.com/file/d/1TvIf-dvfzbm8RKtYz9QHfPWixHnjIcFW/view?usp=sharing
kernel config: https://drive.google.com/file/d/1HKZtF_s3l6PL3OoQbNq_ei9CdBus-Tz0/view?usp=sharing
Similar report:
https://syzkaller.appspot.com/bug?id=83d75b561d8b1b2529c635338ecfb7136261db11
Sorry, I don't have a reproducer for this crash, hope the symbolized
report can help.
If you fix this issue, please add the following tag to the commit:
Reported-by: Hao Sun <sunhao.th@gmail.com>
==================================================================
BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:383 [inline]
BUG: KASAN: use-after-free in tcp_write_timer_handler+0x8fd/0x940
net/ipv4/tcp_timer.c:626
Read of size 1 at addr ffff88802272a0d5 by task syz-executor/12060
CPU: 2 PID: 12060 Comm: syz-executor Not tainted 5.15.0-rc1+ #10
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
print_address_description.constprop.0.cold+0x93/0x334 mm/kasan/report.c:256
__kasan_report mm/kasan/report.c:442 [inline]
kasan_report.cold+0x83/0xdf mm/kasan/report.c:459
tcp_probe_timer net/ipv4/tcp_timer.c:383 [inline]
tcp_write_timer_handler+0x8fd/0x940 net/ipv4/tcp_timer.c:626
tcp_write_timer+0xa2/0x2b0 net/ipv4/tcp_timer.c:642
call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1421
expire_timers kernel/time/timer.c:1466 [inline]
__run_timers.part.0+0x6b0/0xa90 kernel/time/timer.c:1734
__run_timers kernel/time/timer.c:1715 [inline]
run_timer_softirq+0xb6/0x1d0 kernel/time/timer.c:1747
__do_softirq+0x1d7/0x93b kernel/softirq.c:558
invoke_softirq kernel/softirq.c:432 [inline]
__irq_exit_rcu kernel/softirq.c:636 [inline]
irq_exit_rcu+0xf2/0x130 kernel/softirq.c:648
sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1097
</IRQ>
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:zap_pte_range mm/memory.c:1335 [inline]
RIP: 0010:zap_pmd_range mm/memory.c:1481 [inline]
RIP: 0010:zap_pud_range mm/memory.c:1510 [inline]
RIP: 0010:zap_p4d_range mm/memory.c:1531 [inline]
RIP: 0010:unmap_page_range+0xbdd/0x2da0 mm/memory.c:1552
Code: 00 00 00 48 89 44 24 08 e9 8a f5 ff ff e8 4b 97 ca ff 48 8b 74
24 08 4c 89 ea 48 8b 7c 24 48 e8 69 ec ff ff 48 83 7c 24 38 00 <49> 89
c7 0f 85 02 15 00 00 e8 25 97 ca ff 48 8b 44 24 10 48 8d 68
RSP: 0018:ffffc900092cf7a0 EFLAGS: 00000246
RAX: ffffea000080c780 RBX: 0000000000000025 RCX: ffff888104079c80
RDX: 0000000000000000 RSI: ffff888104079c80 RDI: 0000000000000002
RBP: 0000000000000001 R08: ffffffff81aba2c6 R09: 000000000013ffff
R10: 0000000000000006 R11: ffffed102080f390 R12: 0000000000000025
R13: 000000002031e025 R14: dffffc0000000000 R15: 00000000004db000
unmap_single_vma+0x198/0x310 mm/memory.c:1597
unmap_vmas+0x16d/0x2f0 mm/memory.c:1629
exit_mmap+0x1d0/0x650 mm/mmap.c:3171
__mmput kernel/fork.c:1115 [inline]
mmput+0x16d/0x440 kernel/fork.c:1136
exit_mm kernel/exit.c:501 [inline]
do_exit+0xad6/0x2dd0 kernel/exit.c:812
do_group_exit+0x125/0x340 kernel/exit.c:922
get_signal+0x4d5/0x25a0 kernel/signal.c:2868
arch_do_signal_or_restart+0x2ed/0x1c40 arch/x86/kernel/signal.c:865
handle_signal_work kernel/entry/common.c:148 [inline]
exit_to_user_mode_loop kernel/entry/common.c:172 [inline]
exit_to_user_mode_prepare+0x192/0x2a0 kernel/entry/common.c:209
__syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:302
do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4739cd
Code: Unable to access opcode bytes at RIP 0x4739a3.
RSP: 002b:00007f4ef3c1bcd8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 000000000059c0a0 RCX: 00000000004739cd
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000059c0a8
RBP: 000000000059c0a8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000059c0ac
R13: 00007ffc59e0995f R14: 00007ffc59e09b00 R15: 00007f4ef3c1bdc0
Allocated by task 10273:
kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
kasan_set_track mm/kasan/common.c:46 [inline]
set_alloc_info mm/kasan/common.c:434 [inline]
__kasan_slab_alloc+0x83/0xb0 mm/kasan/common.c:467
kasan_slab_alloc include/linux/kasan.h:254 [inline]
slab_post_alloc_hook+0x4d/0x4b0 mm/slab.h:519
slab_alloc_node mm/slub.c:3206 [inline]
slab_alloc mm/slub.c:3214 [inline]
kmem_cache_alloc+0x150/0x340 mm/slub.c:3219
kmem_cache_zalloc include/linux/slab.h:711 [inline]
net_alloc net/core/net_namespace.c:402 [inline]
copy_net_ns+0xea/0x660 net/core/net_namespace.c:457
create_new_namespaces.isra.0+0x3cb/0xae0 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xc8/0x1f0 kernel/nsproxy.c:226
ksys_unshare+0x445/0x920 kernel/fork.c:3077
__do_sys_unshare kernel/fork.c:3151 [inline]
__se_sys_unshare kernel/fork.c:3149 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3149
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
The buggy address belongs to the object at ffff888022729a40
which belongs to the cache net_namespace of size 6464
The buggy address is located 1685 bytes inside of
6464-byte region [ffff888022729a40, ffff88802272b380)
The buggy address belongs to the page:
page:ffffea000089ca00 refcount:1 mapcount:0 mapping:0000000000000000
index:0xffff888022729a40 pfn:0x22728
head:ffffea000089ca00 order:3 compound_mapcount:0 compound_pincount:0
flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888010e0c500
raw: ffff888022729a40 0000000080040002 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 3, migratetype Unmovable, gfp_mask
0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC),
pid 9342, ts 177339275214, free_ts 174923780601
set_page_owner include/linux/page_owner.h:31 [inline]
post_alloc_hook mm/page_alloc.c:2418 [inline]
prep_new_page+0x1a5/0x240 mm/page_alloc.c:2424
get_page_from_freelist+0x1f10/0x3b70 mm/page_alloc.c:4153
__alloc_pages+0x306/0x6e0 mm/page_alloc.c:5375
alloc_pages+0x115/0x240 mm/mempolicy.c:2197
alloc_slab_page mm/slub.c:1763 [inline]
allocate_slab mm/slub.c:1900 [inline]
new_slab+0x34a/0x480 mm/slub.c:1963
___slab_alloc+0xa9f/0x10d0 mm/slub.c:2994
__slab_alloc.isra.0+0x4d/0xa0 mm/slub.c:3081
slab_alloc_node mm/slub.c:3172 [inline]
slab_alloc mm/slub.c:3214 [inline]
kmem_cache_alloc+0x31e/0x340 mm/slub.c:3219
kmem_cache_zalloc include/linux/slab.h:711 [inline]
net_alloc net/core/net_namespace.c:402 [inline]
copy_net_ns+0xea/0x660 net/core/net_namespace.c:457
create_new_namespaces.isra.0+0x3cb/0xae0 kernel/nsproxy.c:110
copy_namespaces+0x391/0x450 kernel/nsproxy.c:178
copy_process+0x2d37/0x73d0 kernel/fork.c:2197
kernel_clone+0xe7/0x10d0 kernel/fork.c:2584
__do_sys_clone+0xc8/0x110 kernel/fork.c:2701
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
page last free stack trace:
reset_page_owner include/linux/page_owner.h:24 [inline]
free_pages_prepare mm/page_alloc.c:1338 [inline]
free_pcp_prepare+0x412/0x900 mm/page_alloc.c:1389
free_unref_page_prepare mm/page_alloc.c:3315 [inline]
free_unref_page+0x19/0x580 mm/page_alloc.c:3394
__unfreeze_partials+0x340/0x360 mm/slub.c:2495
qlink_free mm/kasan/quarantine.c:146 [inline]
qlist_free_all+0x5a/0xc0 mm/kasan/quarantine.c:165
kasan_quarantine_reduce+0x13d/0x180 mm/kasan/quarantine.c:272
__kasan_slab_alloc+0x95/0xb0 mm/kasan/common.c:444
kasan_slab_alloc include/linux/kasan.h:254 [inline]
slab_post_alloc_hook+0x4d/0x4b0 mm/slab.h:519
slab_alloc_node mm/slub.c:3206 [inline]
slab_alloc mm/slub.c:3214 [inline]
kmem_cache_alloc+0x150/0x340 mm/slub.c:3219
kmem_cache_zalloc include/linux/slab.h:711 [inline]
jbd2_alloc_handle include/linux/jbd2.h:1603 [inline]
new_handle fs/jbd2/transaction.c:481 [inline]
jbd2__journal_start fs/jbd2/transaction.c:508 [inline]
jbd2__journal_start+0x191/0x920 fs/jbd2/transaction.c:490
__ext4_journal_start_sb+0x3a8/0x4a0 fs/ext4/ext4_jbd2.c:105
__ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
ext4_da_write_begin+0x4c5/0x1180 fs/ext4/inode.c:3002
generic_perform_write+0x1fe/0x510 mm/filemap.c:3770
ext4_buffered_write_iter+0x206/0x4c0 fs/ext4/file.c:269
ext4_file_write_iter+0x42e/0x14a0 fs/ext4/file.c:680
call_write_iter include/linux/fs.h:2163 [inline]
do_iter_readv_writev+0x47b/0x750 fs/read_write.c:729
do_iter_write fs/read_write.c:855 [inline]
do_iter_write+0x18b/0x700 fs/read_write.c:836
Memory state around the buggy address:
ffff888022729f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88802272a000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88802272a080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff88802272a100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88802272a180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
----------------
Code disassembly (best guess):
0: 00 00 add %al,(%rax)
2: 00 48 89 add %cl,-0x77(%rax)
5: 44 24 08 rex.R and $0x8,%al
8: e9 8a f5 ff ff jmpq 0xfffff597
d: e8 4b 97 ca ff callq 0xffca975d
12: 48 8b 74 24 08 mov 0x8(%rsp),%rsi
17: 4c 89 ea mov %r13,%rdx
1a: 48 8b 7c 24 48 mov 0x48(%rsp),%rdi
1f: e8 69 ec ff ff callq 0xffffec8d
24: 48 83 7c 24 38 00 cmpq $0x0,0x38(%rsp)
* 2a: 49 89 c7 mov %rax,%r15 <-- trapping instruction
2d: 0f 85 02 15 00 00 jne 0x1535
33: e8 25 97 ca ff callq 0xffca975d
38: 48 8b 44 24 10 mov 0x10(%rsp),%rax
3d: 48 rex.W
3e: 8d .byte 0x8d
3f: 68 .byte 0x68
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KASAN: use-after-free Read in tcp_write_timer_handler
2021-09-22 1:43 KASAN: use-after-free Read in tcp_write_timer_handler Hao Sun
@ 2021-09-22 1:56 ` Eric Dumazet
0 siblings, 0 replies; 7+ messages in thread
From: Eric Dumazet @ 2021-09-22 1:56 UTC (permalink / raw)
To: Hao Sun
Cc: David Miller, David Ahern, Jakub Kicinski, netdev,
Hideaki YOSHIFUJI, Andrii Nakryiko, Alexei Starovoitov, bpf,
Daniel Borkmann, John Fastabend, Martin KaFai Lau,
Linux Kernel Mailing List, Song Liu, Yonghong Song
On Tue, Sep 21, 2021 at 6:43 PM Hao Sun <sunhao.th@gmail.com> wrote:
>
> Hello,
>
> When using Healer to fuzz the latest Linux kernel, the following crash
> was triggered.
>
We have dozens of such reports provided already by syzbot.
If you do not provide a repro, there is little hope.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KASAN: use-after-free Read in tcp_write_timer_handler
2023-04-05 19:47 ` Eric Dumazet
@ 2023-04-05 22:17 ` Dae R. Jeong
0 siblings, 0 replies; 7+ messages in thread
From: Dae R. Jeong @ 2023-04-05 22:17 UTC (permalink / raw)
To: Eric Dumazet
Cc: Kuniyuki Iwashima, bpf, davem, dsahern, kuba, linux-kernel,
netdev, pabeni, yoshfuji
On Thu, Apr 6, 2023 at 4:48 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Wed, Apr 5, 2023 at 9:42 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
> >
> > From: Eric Dumazet <edumazet@google.com>
> > Date: Wed, 5 Apr 2023 13:28:16 +0200
> > > On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> > > >
> > > > Unfortunately, we have not found a reproducer for the crash yet. We
> > > > will inform you if we have any update on this crash. Detailed crash
> > > > information is attached below.
> > > >
> > >
> > > Thanks for the report.
> > >
> > > I have dozens of similar syzbot reports, with no repro.
> > >
> > > I usually hold them, because otherwise it is just noise to mailing lists.
> > >
> > > Normally, all user TCP sockets hold a reference on the netns
> > >
> > > In all these cases, we see a netns being dismantled while there is at
> > > least one socket with a live timer.
> > >
> > > This is therefore a kernel TCP socket, for which we do not have yet
> > > debugging infra ( REF_TRACKER )
> > >
> > > CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> > > we need something tracking the "kernel sockets" as well.
> >
> > Maybe I missed something, but we track kernel sockets with netns
> > by notrefcnt_tracker ?
>
> Oh right, I forgot I did this already :)
>
> commit 0cafd77dcd032d1687efaba5598cf07bce85997f
> Author: Eric Dumazet <edumazet@google.com>
> Date: Thu Oct 20 23:20:18 2022 +0000
>
> net: add a refcount tracker for kernel sockets
>
> Dae, make sure to not send reports based on old kernels.
>
> Using 6.0-rc7 is a waste of your time, and everyone else reading this thread.
>
> I confess I did not check this, and I really should do that all the time.
I'm sorry and I understand your time is valuable.
I will let you know when I observe this issue again.
>
> >
> > I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.
> >
> >
> > >
> > > Otherwise bugs in subsystems not properly dismantling their kernel
> > > socket at netns dismantle are next to impossible to track and fix.
> > >
> > > If anyone has time to implement this, feel free to submit patches.
> > >
> > > Thanks.
Best regards,
Dae R. Jeong.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KASAN: use-after-free Read in tcp_write_timer_handler
2023-04-05 19:41 ` Kuniyuki Iwashima
@ 2023-04-05 19:47 ` Eric Dumazet
2023-04-05 22:17 ` Dae R. Jeong
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2023-04-05 19:47 UTC (permalink / raw)
To: Kuniyuki Iwashima
Cc: bpf, davem, dsahern, kuba, linux-kernel, netdev, pabeni,
threeearcat, yoshfuji
On Wed, Apr 5, 2023 at 9:42 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
>
> From: Eric Dumazet <edumazet@google.com>
> Date: Wed, 5 Apr 2023 13:28:16 +0200
> > On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> > >
> > > Unfortunately, we have not found a reproducer for the crash yet. We
> > > will inform you if we have any update on this crash. Detailed crash
> > > information is attached below.
> > >
> >
> > Thanks for the report.
> >
> > I have dozens of similar syzbot reports, with no repro.
> >
> > I usually hold them, because otherwise it is just noise to mailing lists.
> >
> > Normally, all user TCP sockets hold a reference on the netns
> >
> > In all these cases, we see a netns being dismantled while there is at
> > least one socket with a live timer.
> >
> > This is therefore a kernel TCP socket, for which we do not have yet
> > debugging infra ( REF_TRACKER )
> >
> > CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> > we need something tracking the "kernel sockets" as well.
>
> Maybe I missed something, but we track kernel sockets with netns
> by notrefcnt_tracker ?
Oh right, I forgot I did this already :)
commit 0cafd77dcd032d1687efaba5598cf07bce85997f
Author: Eric Dumazet <edumazet@google.com>
Date: Thu Oct 20 23:20:18 2022 +0000
net: add a refcount tracker for kernel sockets
Dae, make sure to not send reports based on old kernels.
Using 6.0-rc7 is a waste of your time, and everyone else reading this thread.
I confess I did not check this, and I really should do that all the time.
>
> I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.
>
>
> >
> > Otherwise bugs in subsystems not properly dismantling their kernel
> > socket at netns dismantle are next to impossible to track and fix.
> >
> > If anyone has time to implement this, feel free to submit patches.
> >
> > Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KASAN: use-after-free Read in tcp_write_timer_handler
2023-04-05 11:28 ` Eric Dumazet
@ 2023-04-05 19:41 ` Kuniyuki Iwashima
2023-04-05 19:47 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Kuniyuki Iwashima @ 2023-04-05 19:41 UTC (permalink / raw)
To: edumazet
Cc: bpf, davem, dsahern, kuba, linux-kernel, netdev, pabeni,
threeearcat, yoshfuji, kuniyu
From: Eric Dumazet <edumazet@google.com>
Date: Wed, 5 Apr 2023 13:28:16 +0200
> On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> >
> > Hi,
> >
> > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> >
> > Unfortunately, we have not found a reproducer for the crash yet. We
> > will inform you if we have any update on this crash. Detailed crash
> > information is attached below.
> >
>
> Thanks for the report.
>
> I have dozens of similar syzbot reports, with no repro.
>
> I usually hold them, because otherwise it is just noise to mailing lists.
>
> Normally, all user TCP sockets hold a reference on the netns
>
> In all these cases, we see a netns being dismantled while there is at
> least one socket with a live timer.
>
> This is therefore a kernel TCP socket, for which we do not have yet
> debugging infra ( REF_TRACKER )
>
> CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> we need something tracking the "kernel sockets" as well.
Maybe I missed something, but we track kernel sockets with netns
by notrefcnt_tracker ?
I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.
>
> Otherwise bugs in subsystems not properly dismantling their kernel
> socket at netns dismantle are next to impossible to track and fix.
>
> If anyone has time to implement this, feel free to submit patches.
>
> Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KASAN: use-after-free Read in tcp_write_timer_handler
2023-04-05 10:41 Dae R. Jeong
@ 2023-04-05 11:28 ` Eric Dumazet
2023-04-05 19:41 ` Kuniyuki Iwashima
0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2023-04-05 11:28 UTC (permalink / raw)
To: Dae R. Jeong
Cc: davem, yoshfuji, dsahern, kuba, pabeni, netdev, linux-kernel, bpf
On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
>
> Hi,
>
> We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
>
> Unfortunately, we have not found a reproducer for the crash yet. We
> will inform you if we have any update on this crash. Detailed crash
> information is attached below.
>
Thanks for the report.
I have dozens of similar syzbot reports, with no repro.
I usually hold them, because otherwise it is just noise to mailing lists.
Normally, all user TCP sockets hold a reference on the netns
In all these cases, we see a netns being dismantled while there is at
least one socket with a live timer.
This is therefore a kernel TCP socket, for which we do not have yet
debugging infra ( REF_TRACKER )
CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
we need something tracking the "kernel sockets" as well.
Otherwise bugs in subsystems not properly dismantling their kernel
socket at netns dismantle are next to impossible to track and fix.
If anyone has time to implement this, feel free to submit patches.
Thanks.
> Best regards,
> Dae R. Jeong
>
> -----
> - Kernel version:
> 6.0-rc7
>
> - Crash report:
> ==================================================================
> BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
> BUG: KASAN: use-after-free in tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
> Read of size 1 at addr ffff888046bc86a5 by task syz-fuzzer/6625
>
> CPU: 0 PID: 6625 Comm: syz-fuzzer Not tainted 6.0.0-rc7-00167-g92162e4a9862 #2
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x1cf/0x2b7 lib/dump_stack.c:106
> print_address_description+0x21/0x470 mm/kasan/report.c:317
> print_report+0x108/0x1f0 mm/kasan/report.c:433
> kasan_report+0xe5/0x110 mm/kasan/report.c:495
> tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
> tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
> tcp_write_timer+0x1a5/0x2c0 net/ipv4/tcp_timer.c:637
> call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
> expire_timers kernel/time/timer.c:1519 [inline]
> __run_timers+0x76f/0x980 kernel/time/timer.c:1790
> run_timer_softirq+0x63/0xf0 kernel/time/timer.c:1803
> __do_softirq+0x372/0x783 kernel/softirq.c:571
> __irq_exit_rcu+0xcf/0x160 kernel/softirq.c:650
> irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
> sysvec_apic_timer_interrupt+0x43/0xb0 arch/x86/kernel/apic/apic.c:1106
> asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
> RIP: 0033:0x421fb1
> Code: 90 48 8b 4f 18 90 49 b8 00 00 00 00 00 80 00 00 49 01 c8 49 c1 e8 1a 66 90 49 81 f8 00 00 40 00 0f 83 e2 00 00 00 4a 8b 14 c2 <84> 02 49 89 c8 48 c1 e9 10 81 e1 ff 03 00 00 44 0f b6 8c 0a 00 04
> RSP: 002b:00007f3d50bebd38 EFLAGS: 00000287
> RAX: 000000c001a0d140 RBX: 000000c001bba000 RCX: 000000c001a0c000
> RDX: 00007f3d51bef000 RSI: 000000c000025240 RDI: 00007f3d791daa28
> RBP: 00007f3d50bebd78 R08: 0000000000203000 R09: 00007f3d4c2f6001
> R10: 000000000000008a R11: 0000000000004048 R12: 0000000000000004
> R13: 000000c001a0d140 R14: 000000c000007520 R15: 0000000000000180
> </TASK>
>
> Allocated by task 6664:
> kasan_save_stack mm/kasan/common.c:38 [inline]
> kasan_set_track mm/kasan/common.c:45 [inline]
> set_alloc_info mm/kasan/common.c:437 [inline]
> __kasan_slab_alloc+0xa3/0xd0 mm/kasan/common.c:470
> kasan_slab_alloc include/linux/kasan.h:224 [inline]
> slab_post_alloc_hook mm/slab.h:727 [inline]
> slab_alloc_node mm/slub.c:3248 [inline]
> slab_alloc mm/slub.c:3256 [inline]
> __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
> kmem_cache_alloc+0x2e6/0x450 mm/slub.c:3273
> kmem_cache_zalloc include/linux/slab.h:723 [inline]
> net_alloc net/core/net_namespace.c:404 [inline]
> copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
> create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
> unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
> ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
> __do_sys_unshare kernel/fork.c:3254 [inline]
> __se_sys_unshare kernel/fork.c:3252 [inline]
> __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
> do_syscall_x64 arch/x86/entry/common.c:51 [inline]
> do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> Freed by task 6874:
> kasan_save_stack mm/kasan/common.c:38 [inline]
> kasan_set_track+0x3d/0x60 mm/kasan/common.c:45
> kasan_set_free_info+0x1f/0x40 mm/kasan/generic.c:370
> ____kasan_slab_free+0x134/0x1c0 mm/kasan/common.c:367
> kasan_slab_free include/linux/kasan.h:200 [inline]
> slab_free_hook mm/slub.c:1759 [inline]
> slab_free_freelist_hook+0x278/0x370 mm/slub.c:1785
> slab_free mm/slub.c:3539 [inline]
> kmem_cache_free+0x11a/0x310 mm/slub.c:3556
> net_free net/core/net_namespace.c:433 [inline]
> cleanup_net+0xd68/0xe20 net/core/net_namespace.c:616
> process_one_work+0x83f/0x11a0 kernel/workqueue.c:2289
> worker_thread+0xa6c/0x1290 kernel/workqueue.c:2436
> kthread+0x28a/0x320 kernel/kthread.c:376
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
>
> Last potentially related work creation:
> kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
> __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
> insert_work+0x54/0x400 kernel/workqueue.c:1358
> __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
> call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
> expire_timers kernel/time/timer.c:1514 [inline]
> __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
> __do_softirq+0x372/0x783 kernel/softirq.c:571
>
> Second to last potentially related work creation:
> kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
> __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
> insert_work+0x54/0x400 kernel/workqueue.c:1358
> __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
> call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
> expire_timers kernel/time/timer.c:1514 [inline]
> __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
> __do_softirq+0x372/0x783 kernel/softirq.c:571
>
> The buggy address belongs to the object at ffff888046bc8000
> which belongs to the cache net_namespace of size 6784
> The buggy address is located 1701 bytes inside of
> 6784-byte region [ffff888046bc8000, ffff888046bc9a80)
>
> The buggy address belongs to the physical page:
> page:ffffea00011af200 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x46bc8
> head:ffffea00011af200 order:3 compound_mapcount:0 compound_pincount:0
> flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
> raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888013618f00
> raw: 0000000000000000 0000000080040004 00000001ffffffff 0000000000000000
> page dumped because: kasan: bad access detected
> page_owner tracks the page as allocated
> page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 6664, tgid 6664 (syz-executor.0), ts 88505587135, free_ts 0
> prep_new_page mm/page_alloc.c:2532 [inline]
> get_page_from_freelist+0x800/0xc10 mm/page_alloc.c:4283
> __alloc_pages+0x2f0/0x650 mm/page_alloc.c:5549
> alloc_slab_page mm/slub.c:1829 [inline]
> allocate_slab+0x1eb/0xc00 mm/slub.c:1974
> new_slab mm/slub.c:2034 [inline]
> ___slab_alloc+0x581/0xff0 mm/slub.c:3036
> __slab_alloc mm/slub.c:3123 [inline]
> slab_alloc_node mm/slub.c:3214 [inline]
> slab_alloc mm/slub.c:3256 [inline]
> __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
> kmem_cache_alloc+0x386/0x450 mm/slub.c:3273
> kmem_cache_zalloc include/linux/slab.h:723 [inline]
> net_alloc net/core/net_namespace.c:404 [inline]
> copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
> create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
> unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
> ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
> __do_sys_unshare kernel/fork.c:3254 [inline]
> __se_sys_unshare kernel/fork.c:3252 [inline]
> __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
> do_syscall_x64 arch/x86/entry/common.c:51 [inline]
> do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
> page_owner free stack trace missing
>
> Memory state around the buggy address:
> ffff888046bc8580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff888046bc8600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >ffff888046bc8680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ^
> ffff888046bc8700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff888046bc8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ==================================================================
^ permalink raw reply [flat|nested] 7+ messages in thread
* KASAN: use-after-free Read in tcp_write_timer_handler
@ 2023-04-05 10:41 Dae R. Jeong
2023-04-05 11:28 ` Eric Dumazet
0 siblings, 1 reply; 7+ messages in thread
From: Dae R. Jeong @ 2023-04-05 10:41 UTC (permalink / raw)
To: edumazet, davem, yoshfuji, dsahern, kuba, pabeni, netdev,
linux-kernel, bpf
Hi,
We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
Unfortunately, we have not found a reproducer for the crash yet. We
will inform you if we have any update on this crash. Detailed crash
information is attached below.
Best regards,
Dae R. Jeong
-----
- Kernel version:
6.0-rc7
- Crash report:
==================================================================
BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
BUG: KASAN: use-after-free in tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
Read of size 1 at addr ffff888046bc86a5 by task syz-fuzzer/6625
CPU: 0 PID: 6625 Comm: syz-fuzzer Not tainted 6.0.0-rc7-00167-g92162e4a9862 #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1cf/0x2b7 lib/dump_stack.c:106
print_address_description+0x21/0x470 mm/kasan/report.c:317
print_report+0x108/0x1f0 mm/kasan/report.c:433
kasan_report+0xe5/0x110 mm/kasan/report.c:495
tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
tcp_write_timer+0x1a5/0x2c0 net/ipv4/tcp_timer.c:637
call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1519 [inline]
__run_timers+0x76f/0x980 kernel/time/timer.c:1790
run_timer_softirq+0x63/0xf0 kernel/time/timer.c:1803
__do_softirq+0x372/0x783 kernel/softirq.c:571
__irq_exit_rcu+0xcf/0x160 kernel/softirq.c:650
irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
sysvec_apic_timer_interrupt+0x43/0xb0 arch/x86/kernel/apic/apic.c:1106
asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0033:0x421fb1
Code: 90 48 8b 4f 18 90 49 b8 00 00 00 00 00 80 00 00 49 01 c8 49 c1 e8 1a 66 90 49 81 f8 00 00 40 00 0f 83 e2 00 00 00 4a 8b 14 c2 <84> 02 49 89 c8 48 c1 e9 10 81 e1 ff 03 00 00 44 0f b6 8c 0a 00 04
RSP: 002b:00007f3d50bebd38 EFLAGS: 00000287
RAX: 000000c001a0d140 RBX: 000000c001bba000 RCX: 000000c001a0c000
RDX: 00007f3d51bef000 RSI: 000000c000025240 RDI: 00007f3d791daa28
RBP: 00007f3d50bebd78 R08: 0000000000203000 R09: 00007f3d4c2f6001
R10: 000000000000008a R11: 0000000000004048 R12: 0000000000000004
R13: 000000c001a0d140 R14: 000000c000007520 R15: 0000000000000180
</TASK>
Allocated by task 6664:
kasan_save_stack mm/kasan/common.c:38 [inline]
kasan_set_track mm/kasan/common.c:45 [inline]
set_alloc_info mm/kasan/common.c:437 [inline]
__kasan_slab_alloc+0xa3/0xd0 mm/kasan/common.c:470
kasan_slab_alloc include/linux/kasan.h:224 [inline]
slab_post_alloc_hook mm/slab.h:727 [inline]
slab_alloc_node mm/slub.c:3248 [inline]
slab_alloc mm/slub.c:3256 [inline]
__kmem_cache_alloc_lru mm/slub.c:3263 [inline]
kmem_cache_alloc+0x2e6/0x450 mm/slub.c:3273
kmem_cache_zalloc include/linux/slab.h:723 [inline]
net_alloc net/core/net_namespace.c:404 [inline]
copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
__do_sys_unshare kernel/fork.c:3254 [inline]
__se_sys_unshare kernel/fork.c:3252 [inline]
__x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0xcd
Freed by task 6874:
kasan_save_stack mm/kasan/common.c:38 [inline]
kasan_set_track+0x3d/0x60 mm/kasan/common.c:45
kasan_set_free_info+0x1f/0x40 mm/kasan/generic.c:370
____kasan_slab_free+0x134/0x1c0 mm/kasan/common.c:367
kasan_slab_free include/linux/kasan.h:200 [inline]
slab_free_hook mm/slub.c:1759 [inline]
slab_free_freelist_hook+0x278/0x370 mm/slub.c:1785
slab_free mm/slub.c:3539 [inline]
kmem_cache_free+0x11a/0x310 mm/slub.c:3556
net_free net/core/net_namespace.c:433 [inline]
cleanup_net+0xd68/0xe20 net/core/net_namespace.c:616
process_one_work+0x83f/0x11a0 kernel/workqueue.c:2289
worker_thread+0xa6c/0x1290 kernel/workqueue.c:2436
kthread+0x28a/0x320 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
Last potentially related work creation:
kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
__kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
insert_work+0x54/0x400 kernel/workqueue.c:1358
__queue_work+0xa95/0xe00 kernel/workqueue.c:1517
call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1514 [inline]
__run_timers+0x7a2/0x980 kernel/time/timer.c:1790
__do_softirq+0x372/0x783 kernel/softirq.c:571
Second to last potentially related work creation:
kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
__kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
insert_work+0x54/0x400 kernel/workqueue.c:1358
__queue_work+0xa95/0xe00 kernel/workqueue.c:1517
call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
expire_timers kernel/time/timer.c:1514 [inline]
__run_timers+0x7a2/0x980 kernel/time/timer.c:1790
__do_softirq+0x372/0x783 kernel/softirq.c:571
The buggy address belongs to the object at ffff888046bc8000
which belongs to the cache net_namespace of size 6784
The buggy address is located 1701 bytes inside of
6784-byte region [ffff888046bc8000, ffff888046bc9a80)
The buggy address belongs to the physical page:
page:ffffea00011af200 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x46bc8
head:ffffea00011af200 order:3 compound_mapcount:0 compound_pincount:0
flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888013618f00
raw: 0000000000000000 0000000080040004 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 6664, tgid 6664 (syz-executor.0), ts 88505587135, free_ts 0
prep_new_page mm/page_alloc.c:2532 [inline]
get_page_from_freelist+0x800/0xc10 mm/page_alloc.c:4283
__alloc_pages+0x2f0/0x650 mm/page_alloc.c:5549
alloc_slab_page mm/slub.c:1829 [inline]
allocate_slab+0x1eb/0xc00 mm/slub.c:1974
new_slab mm/slub.c:2034 [inline]
___slab_alloc+0x581/0xff0 mm/slub.c:3036
__slab_alloc mm/slub.c:3123 [inline]
slab_alloc_node mm/slub.c:3214 [inline]
slab_alloc mm/slub.c:3256 [inline]
__kmem_cache_alloc_lru mm/slub.c:3263 [inline]
kmem_cache_alloc+0x386/0x450 mm/slub.c:3273
kmem_cache_zalloc include/linux/slab.h:723 [inline]
net_alloc net/core/net_namespace.c:404 [inline]
copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
__do_sys_unshare kernel/fork.c:3254 [inline]
__se_sys_unshare kernel/fork.c:3252 [inline]
__x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0xcd
page_owner free stack trace missing
Memory state around the buggy address:
ffff888046bc8580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888046bc8600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888046bc8680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff888046bc8700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888046bc8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-04-05 22:17 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-22 1:43 KASAN: use-after-free Read in tcp_write_timer_handler Hao Sun
2021-09-22 1:56 ` Eric Dumazet
2023-04-05 10:41 Dae R. Jeong
2023-04-05 11:28 ` Eric Dumazet
2023-04-05 19:41 ` Kuniyuki Iwashima
2023-04-05 19:47 ` Eric Dumazet
2023-04-05 22:17 ` Dae R. Jeong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).