linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* KASAN: use-after-free Read in tcp_write_timer_handler
@ 2021-09-22  1:43 Hao Sun
  2021-09-22  1:56 ` Eric Dumazet
  0 siblings, 1 reply; 7+ messages in thread
From: Hao Sun @ 2021-09-22  1:43 UTC (permalink / raw)
  To: davem, dsahern, Eric Dumazet, kuba, netdev, yoshfuji
  Cc: andrii, ast, bpf, daniel, john.fastabend, kafai,
	Linux Kernel Mailing List, songliubraving, yhs

Hello,

When using Healer to fuzz the latest Linux kernel, the following crash
was triggered.

HEAD commit: 4357f03d6611 Merge tag 'pm-5.15-rc2
git tree: upstream
console output:
https://drive.google.com/file/d/1TvIf-dvfzbm8RKtYz9QHfPWixHnjIcFW/view?usp=sharing
kernel config: https://drive.google.com/file/d/1HKZtF_s3l6PL3OoQbNq_ei9CdBus-Tz0/view?usp=sharing
Similar report:
https://syzkaller.appspot.com/bug?id=83d75b561d8b1b2529c635338ecfb7136261db11

Sorry, I don't have a reproducer for this crash, hope the symbolized
report can help.
If you fix this issue, please add the following tag to the commit:
Reported-by: Hao Sun <sunhao.th@gmail.com>

==================================================================
BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:383 [inline]
BUG: KASAN: use-after-free in tcp_write_timer_handler+0x8fd/0x940
net/ipv4/tcp_timer.c:626
Read of size 1 at addr ffff88802272a0d5 by task syz-executor/12060

CPU: 2 PID: 12060 Comm: syz-executor Not tainted 5.15.0-rc1+ #10
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 print_address_description.constprop.0.cold+0x93/0x334 mm/kasan/report.c:256
 __kasan_report mm/kasan/report.c:442 [inline]
 kasan_report.cold+0x83/0xdf mm/kasan/report.c:459
 tcp_probe_timer net/ipv4/tcp_timer.c:383 [inline]
 tcp_write_timer_handler+0x8fd/0x940 net/ipv4/tcp_timer.c:626
 tcp_write_timer+0xa2/0x2b0 net/ipv4/tcp_timer.c:642
 call_timer_fn+0x1a5/0x6b0 kernel/time/timer.c:1421
 expire_timers kernel/time/timer.c:1466 [inline]
 __run_timers.part.0+0x6b0/0xa90 kernel/time/timer.c:1734
 __run_timers kernel/time/timer.c:1715 [inline]
 run_timer_softirq+0xb6/0x1d0 kernel/time/timer.c:1747
 __do_softirq+0x1d7/0x93b kernel/softirq.c:558
 invoke_softirq kernel/softirq.c:432 [inline]
 __irq_exit_rcu kernel/softirq.c:636 [inline]
 irq_exit_rcu+0xf2/0x130 kernel/softirq.c:648
 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1097
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638
RIP: 0010:zap_pte_range mm/memory.c:1335 [inline]
RIP: 0010:zap_pmd_range mm/memory.c:1481 [inline]
RIP: 0010:zap_pud_range mm/memory.c:1510 [inline]
RIP: 0010:zap_p4d_range mm/memory.c:1531 [inline]
RIP: 0010:unmap_page_range+0xbdd/0x2da0 mm/memory.c:1552
Code: 00 00 00 48 89 44 24 08 e9 8a f5 ff ff e8 4b 97 ca ff 48 8b 74
24 08 4c 89 ea 48 8b 7c 24 48 e8 69 ec ff ff 48 83 7c 24 38 00 <49> 89
c7 0f 85 02 15 00 00 e8 25 97 ca ff 48 8b 44 24 10 48 8d 68
RSP: 0018:ffffc900092cf7a0 EFLAGS: 00000246
RAX: ffffea000080c780 RBX: 0000000000000025 RCX: ffff888104079c80
RDX: 0000000000000000 RSI: ffff888104079c80 RDI: 0000000000000002
RBP: 0000000000000001 R08: ffffffff81aba2c6 R09: 000000000013ffff
R10: 0000000000000006 R11: ffffed102080f390 R12: 0000000000000025
R13: 000000002031e025 R14: dffffc0000000000 R15: 00000000004db000
 unmap_single_vma+0x198/0x310 mm/memory.c:1597
 unmap_vmas+0x16d/0x2f0 mm/memory.c:1629
 exit_mmap+0x1d0/0x650 mm/mmap.c:3171
 __mmput kernel/fork.c:1115 [inline]
 mmput+0x16d/0x440 kernel/fork.c:1136
 exit_mm kernel/exit.c:501 [inline]
 do_exit+0xad6/0x2dd0 kernel/exit.c:812
 do_group_exit+0x125/0x340 kernel/exit.c:922
 get_signal+0x4d5/0x25a0 kernel/signal.c:2868
 arch_do_signal_or_restart+0x2ed/0x1c40 arch/x86/kernel/signal.c:865
 handle_signal_work kernel/entry/common.c:148 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:172 [inline]
 exit_to_user_mode_prepare+0x192/0x2a0 kernel/entry/common.c:209
 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
 syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:302
 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4739cd
Code: Unable to access opcode bytes at RIP 0x4739a3.
RSP: 002b:00007f4ef3c1bcd8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 000000000059c0a0 RCX: 00000000004739cd
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000059c0a8
RBP: 000000000059c0a8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000059c0ac
R13: 00007ffc59e0995f R14: 00007ffc59e09b00 R15: 00007f4ef3c1bdc0

Allocated by task 10273:
 kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
 kasan_set_track mm/kasan/common.c:46 [inline]
 set_alloc_info mm/kasan/common.c:434 [inline]
 __kasan_slab_alloc+0x83/0xb0 mm/kasan/common.c:467
 kasan_slab_alloc include/linux/kasan.h:254 [inline]
 slab_post_alloc_hook+0x4d/0x4b0 mm/slab.h:519
 slab_alloc_node mm/slub.c:3206 [inline]
 slab_alloc mm/slub.c:3214 [inline]
 kmem_cache_alloc+0x150/0x340 mm/slub.c:3219
 kmem_cache_zalloc include/linux/slab.h:711 [inline]
 net_alloc net/core/net_namespace.c:402 [inline]
 copy_net_ns+0xea/0x660 net/core/net_namespace.c:457
 create_new_namespaces.isra.0+0x3cb/0xae0 kernel/nsproxy.c:110
 unshare_nsproxy_namespaces+0xc8/0x1f0 kernel/nsproxy.c:226
 ksys_unshare+0x445/0x920 kernel/fork.c:3077
 __do_sys_unshare kernel/fork.c:3151 [inline]
 __se_sys_unshare kernel/fork.c:3149 [inline]
 __x64_sys_unshare+0x2d/0x40 kernel/fork.c:3149
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae

The buggy address belongs to the object at ffff888022729a40
 which belongs to the cache net_namespace of size 6464
The buggy address is located 1685 bytes inside of
 6464-byte region [ffff888022729a40, ffff88802272b380)
The buggy address belongs to the page:
page:ffffea000089ca00 refcount:1 mapcount:0 mapping:0000000000000000
index:0xffff888022729a40 pfn:0x22728
head:ffffea000089ca00 order:3 compound_mapcount:0 compound_pincount:0
flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888010e0c500
raw: ffff888022729a40 0000000080040002 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 3, migratetype Unmovable, gfp_mask
0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC),
pid 9342, ts 177339275214, free_ts 174923780601
 set_page_owner include/linux/page_owner.h:31 [inline]
 post_alloc_hook mm/page_alloc.c:2418 [inline]
 prep_new_page+0x1a5/0x240 mm/page_alloc.c:2424
 get_page_from_freelist+0x1f10/0x3b70 mm/page_alloc.c:4153
 __alloc_pages+0x306/0x6e0 mm/page_alloc.c:5375
 alloc_pages+0x115/0x240 mm/mempolicy.c:2197
 alloc_slab_page mm/slub.c:1763 [inline]
 allocate_slab mm/slub.c:1900 [inline]
 new_slab+0x34a/0x480 mm/slub.c:1963
 ___slab_alloc+0xa9f/0x10d0 mm/slub.c:2994
 __slab_alloc.isra.0+0x4d/0xa0 mm/slub.c:3081
 slab_alloc_node mm/slub.c:3172 [inline]
 slab_alloc mm/slub.c:3214 [inline]
 kmem_cache_alloc+0x31e/0x340 mm/slub.c:3219
 kmem_cache_zalloc include/linux/slab.h:711 [inline]
 net_alloc net/core/net_namespace.c:402 [inline]
 copy_net_ns+0xea/0x660 net/core/net_namespace.c:457
 create_new_namespaces.isra.0+0x3cb/0xae0 kernel/nsproxy.c:110
 copy_namespaces+0x391/0x450 kernel/nsproxy.c:178
 copy_process+0x2d37/0x73d0 kernel/fork.c:2197
 kernel_clone+0xe7/0x10d0 kernel/fork.c:2584
 __do_sys_clone+0xc8/0x110 kernel/fork.c:2701
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
page last free stack trace:
 reset_page_owner include/linux/page_owner.h:24 [inline]
 free_pages_prepare mm/page_alloc.c:1338 [inline]
 free_pcp_prepare+0x412/0x900 mm/page_alloc.c:1389
 free_unref_page_prepare mm/page_alloc.c:3315 [inline]
 free_unref_page+0x19/0x580 mm/page_alloc.c:3394
 __unfreeze_partials+0x340/0x360 mm/slub.c:2495
 qlink_free mm/kasan/quarantine.c:146 [inline]
 qlist_free_all+0x5a/0xc0 mm/kasan/quarantine.c:165
 kasan_quarantine_reduce+0x13d/0x180 mm/kasan/quarantine.c:272
 __kasan_slab_alloc+0x95/0xb0 mm/kasan/common.c:444
 kasan_slab_alloc include/linux/kasan.h:254 [inline]
 slab_post_alloc_hook+0x4d/0x4b0 mm/slab.h:519
 slab_alloc_node mm/slub.c:3206 [inline]
 slab_alloc mm/slub.c:3214 [inline]
 kmem_cache_alloc+0x150/0x340 mm/slub.c:3219
 kmem_cache_zalloc include/linux/slab.h:711 [inline]
 jbd2_alloc_handle include/linux/jbd2.h:1603 [inline]
 new_handle fs/jbd2/transaction.c:481 [inline]
 jbd2__journal_start fs/jbd2/transaction.c:508 [inline]
 jbd2__journal_start+0x191/0x920 fs/jbd2/transaction.c:490
 __ext4_journal_start_sb+0x3a8/0x4a0 fs/ext4/ext4_jbd2.c:105
 __ext4_journal_start fs/ext4/ext4_jbd2.h:326 [inline]
 ext4_da_write_begin+0x4c5/0x1180 fs/ext4/inode.c:3002
 generic_perform_write+0x1fe/0x510 mm/filemap.c:3770
 ext4_buffered_write_iter+0x206/0x4c0 fs/ext4/file.c:269
 ext4_file_write_iter+0x42e/0x14a0 fs/ext4/file.c:680
 call_write_iter include/linux/fs.h:2163 [inline]
 do_iter_readv_writev+0x47b/0x750 fs/read_write.c:729
 do_iter_write fs/read_write.c:855 [inline]
 do_iter_write+0x18b/0x700 fs/read_write.c:836

Memory state around the buggy address:
 ffff888022729f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88802272a000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88802272a080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                                 ^
 ffff88802272a100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88802272a180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
----------------
Code disassembly (best guess):
   0: 00 00                add    %al,(%rax)
   2: 00 48 89              add    %cl,-0x77(%rax)
   5: 44 24 08              rex.R and $0x8,%al
   8: e9 8a f5 ff ff        jmpq   0xfffff597
   d: e8 4b 97 ca ff        callq  0xffca975d
  12: 48 8b 74 24 08        mov    0x8(%rsp),%rsi
  17: 4c 89 ea              mov    %r13,%rdx
  1a: 48 8b 7c 24 48        mov    0x48(%rsp),%rdi
  1f: e8 69 ec ff ff        callq  0xffffec8d
  24: 48 83 7c 24 38 00    cmpq   $0x0,0x38(%rsp)
* 2a: 49 89 c7              mov    %rax,%r15 <-- trapping instruction
  2d: 0f 85 02 15 00 00    jne    0x1535
  33: e8 25 97 ca ff        callq  0xffca975d
  38: 48 8b 44 24 10        mov    0x10(%rsp),%rax
  3d: 48                    rex.W
  3e: 8d                    .byte 0x8d
  3f: 68                    .byte 0x68

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: KASAN: use-after-free Read in tcp_write_timer_handler
  2021-09-22  1:43 KASAN: use-after-free Read in tcp_write_timer_handler Hao Sun
@ 2021-09-22  1:56 ` Eric Dumazet
  0 siblings, 0 replies; 7+ messages in thread
From: Eric Dumazet @ 2021-09-22  1:56 UTC (permalink / raw)
  To: Hao Sun
  Cc: David Miller, David Ahern, Jakub Kicinski, netdev,
	Hideaki YOSHIFUJI, Andrii Nakryiko, Alexei Starovoitov, bpf,
	Daniel Borkmann, John Fastabend, Martin KaFai Lau,
	Linux Kernel Mailing List, Song Liu, Yonghong Song

On Tue, Sep 21, 2021 at 6:43 PM Hao Sun <sunhao.th@gmail.com> wrote:
>
> Hello,
>
> When using Healer to fuzz the latest Linux kernel, the following crash
> was triggered.
>

We have dozens of such reports provided already by syzbot.

If you do not provide a repro, there is little hope.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: KASAN: use-after-free Read in tcp_write_timer_handler
  2023-04-05 19:47     ` Eric Dumazet
@ 2023-04-05 22:17       ` Dae R. Jeong
  0 siblings, 0 replies; 7+ messages in thread
From: Dae R. Jeong @ 2023-04-05 22:17 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Kuniyuki Iwashima, bpf, davem, dsahern, kuba, linux-kernel,
	netdev, pabeni, yoshfuji

On Thu, Apr 6, 2023 at 4:48 AM Eric Dumazet <edumazet@google.com> wrote:
>
> On Wed, Apr 5, 2023 at 9:42 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
> >
> > From:   Eric Dumazet <edumazet@google.com>
> > Date:   Wed, 5 Apr 2023 13:28:16 +0200
> > > On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> > > >
> > > > Unfortunately, we have not found a reproducer for the crash yet. We
> > > > will inform you if we have any update on this crash.  Detailed crash
> > > > information is attached below.
> > > >
> > >
> > > Thanks for the report.
> > >
> > > I have dozens of similar syzbot reports, with no repro.
> > >
> > > I usually hold them, because otherwise it is just noise to mailing lists.
> > >
> > > Normally, all user TCP sockets hold a reference on the netns
> > >
> > > In all these cases, we see a netns being dismantled while there is at
> > > least one socket with a live timer.
> > >
> > > This is therefore a kernel TCP socket, for which we do not have yet
> > > debugging infra ( REF_TRACKER )
> > >
> > > CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> > > we need something tracking the "kernel sockets" as well.
> >
> > Maybe I missed something, but we track kernel sockets with netns
> > by notrefcnt_tracker ?
>
> Oh right, I forgot I did this already :)
>
> commit 0cafd77dcd032d1687efaba5598cf07bce85997f
> Author: Eric Dumazet <edumazet@google.com>
> Date:   Thu Oct 20 23:20:18 2022 +0000
>
>     net: add a refcount tracker for kernel sockets
>
> Dae, make sure to not send reports based on old kernels.
>
> Using 6.0-rc7 is a waste of your time, and everyone else reading this thread.
>
> I confess I did not check this, and I really should do that all the time.

I'm sorry and I understand your time is valuable.
I will let you know when I observe this issue again.

>
> >
> > I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.
> >
> >
> > >
> > > Otherwise bugs in subsystems not properly dismantling their kernel
> > > socket at netns dismantle are next to impossible to track and fix.
> > >
> > > If anyone has time to implement this, feel free to submit patches.
> > >
> > > Thanks.

Best regards,
Dae R. Jeong.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: KASAN: use-after-free Read in tcp_write_timer_handler
  2023-04-05 19:41   ` Kuniyuki Iwashima
@ 2023-04-05 19:47     ` Eric Dumazet
  2023-04-05 22:17       ` Dae R. Jeong
  0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2023-04-05 19:47 UTC (permalink / raw)
  To: Kuniyuki Iwashima
  Cc: bpf, davem, dsahern, kuba, linux-kernel, netdev, pabeni,
	threeearcat, yoshfuji

On Wed, Apr 5, 2023 at 9:42 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
>
> From:   Eric Dumazet <edumazet@google.com>
> Date:   Wed, 5 Apr 2023 13:28:16 +0200
> > On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> > >
> > > Unfortunately, we have not found a reproducer for the crash yet. We
> > > will inform you if we have any update on this crash.  Detailed crash
> > > information is attached below.
> > >
> >
> > Thanks for the report.
> >
> > I have dozens of similar syzbot reports, with no repro.
> >
> > I usually hold them, because otherwise it is just noise to mailing lists.
> >
> > Normally, all user TCP sockets hold a reference on the netns
> >
> > In all these cases, we see a netns being dismantled while there is at
> > least one socket with a live timer.
> >
> > This is therefore a kernel TCP socket, for which we do not have yet
> > debugging infra ( REF_TRACKER )
> >
> > CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> > we need something tracking the "kernel sockets" as well.
>
> Maybe I missed something, but we track kernel sockets with netns
> by notrefcnt_tracker ?

Oh right, I forgot I did this already :)

commit 0cafd77dcd032d1687efaba5598cf07bce85997f
Author: Eric Dumazet <edumazet@google.com>
Date:   Thu Oct 20 23:20:18 2022 +0000

    net: add a refcount tracker for kernel sockets

Dae, make sure to not send reports based on old kernels.

Using 6.0-rc7 is a waste of your time, and everyone else reading this thread.

I confess I did not check this, and I really should do that all the time.



>
> I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.
>
>
> >
> > Otherwise bugs in subsystems not properly dismantling their kernel
> > socket at netns dismantle are next to impossible to track and fix.
> >
> > If anyone has time to implement this, feel free to submit patches.
> >
> > Thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: KASAN: use-after-free Read in tcp_write_timer_handler
  2023-04-05 11:28 ` Eric Dumazet
@ 2023-04-05 19:41   ` Kuniyuki Iwashima
  2023-04-05 19:47     ` Eric Dumazet
  0 siblings, 1 reply; 7+ messages in thread
From: Kuniyuki Iwashima @ 2023-04-05 19:41 UTC (permalink / raw)
  To: edumazet
  Cc: bpf, davem, dsahern, kuba, linux-kernel, netdev, pabeni,
	threeearcat, yoshfuji, kuniyu

From:   Eric Dumazet <edumazet@google.com>
Date:   Wed, 5 Apr 2023 13:28:16 +0200
> On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
> >
> > Hi,
> >
> > We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
> >
> > Unfortunately, we have not found a reproducer for the crash yet. We
> > will inform you if we have any update on this crash.  Detailed crash
> > information is attached below.
> >
> 
> Thanks for the report.
> 
> I have dozens of similar syzbot reports, with no repro.
> 
> I usually hold them, because otherwise it is just noise to mailing lists.
> 
> Normally, all user TCP sockets hold a reference on the netns
> 
> In all these cases, we see a netns being dismantled while there is at
> least one socket with a live timer.
> 
> This is therefore a kernel TCP socket, for which we do not have yet
> debugging infra ( REF_TRACKER )
> 
> CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
> we need something tracking the "kernel sockets" as well.

Maybe I missed something, but we track kernel sockets with netns
by notrefcnt_tracker ?

I thought now CONFIG_NET_NS_REFCNT_TRACKER can catch the case.


> 
> Otherwise bugs in subsystems not properly dismantling their kernel
> socket at netns dismantle are next to impossible to track and fix.
> 
> If anyone has time to implement this, feel free to submit patches.
> 
> Thanks.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: KASAN: use-after-free Read in tcp_write_timer_handler
  2023-04-05 10:41 Dae R. Jeong
@ 2023-04-05 11:28 ` Eric Dumazet
  2023-04-05 19:41   ` Kuniyuki Iwashima
  0 siblings, 1 reply; 7+ messages in thread
From: Eric Dumazet @ 2023-04-05 11:28 UTC (permalink / raw)
  To: Dae R. Jeong
  Cc: davem, yoshfuji, dsahern, kuba, pabeni, netdev, linux-kernel, bpf

On Wed, Apr 5, 2023 at 12:41 PM Dae R. Jeong <threeearcat@gmail.com> wrote:
>
> Hi,
>
> We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.
>
> Unfortunately, we have not found a reproducer for the crash yet. We
> will inform you if we have any update on this crash.  Detailed crash
> information is attached below.
>

Thanks for the report.

I have dozens of similar syzbot reports, with no repro.

I usually hold them, because otherwise it is just noise to mailing lists.

Normally, all user TCP sockets hold a reference on the netns

In all these cases, we see a netns being dismantled while there is at
least one socket with a live timer.

This is therefore a kernel TCP socket, for which we do not have yet
debugging infra ( REF_TRACKER )

CONFIG_NET_DEV_REFCNT_TRACKER=y is helping to detect too many dev_put(),
we need something tracking the "kernel sockets" as well.

Otherwise bugs in subsystems not properly dismantling their kernel
socket at netns dismantle are next to impossible to track and fix.

If anyone has time to implement this, feel free to submit patches.

Thanks.



> Best regards,
> Dae R. Jeong
>
> -----
> - Kernel version:
> 6.0-rc7
>
> - Crash report:
> ==================================================================
> BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
> BUG: KASAN: use-after-free in tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
> Read of size 1 at addr ffff888046bc86a5 by task syz-fuzzer/6625
>
> CPU: 0 PID: 6625 Comm: syz-fuzzer Not tainted 6.0.0-rc7-00167-g92162e4a9862 #2
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> Call Trace:
>  <TASK>
>  __dump_stack lib/dump_stack.c:88 [inline]
>  dump_stack_lvl+0x1cf/0x2b7 lib/dump_stack.c:106
>  print_address_description+0x21/0x470 mm/kasan/report.c:317
>  print_report+0x108/0x1f0 mm/kasan/report.c:433
>  kasan_report+0xe5/0x110 mm/kasan/report.c:495
>  tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
>  tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
>  tcp_write_timer+0x1a5/0x2c0 net/ipv4/tcp_timer.c:637
>  call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
>  expire_timers kernel/time/timer.c:1519 [inline]
>  __run_timers+0x76f/0x980 kernel/time/timer.c:1790
>  run_timer_softirq+0x63/0xf0 kernel/time/timer.c:1803
>  __do_softirq+0x372/0x783 kernel/softirq.c:571
>  __irq_exit_rcu+0xcf/0x160 kernel/softirq.c:650
>  irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
>  sysvec_apic_timer_interrupt+0x43/0xb0 arch/x86/kernel/apic/apic.c:1106
>  asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
> RIP: 0033:0x421fb1
> Code: 90 48 8b 4f 18 90 49 b8 00 00 00 00 00 80 00 00 49 01 c8 49 c1 e8 1a 66 90 49 81 f8 00 00 40 00 0f 83 e2 00 00 00 4a 8b 14 c2 <84> 02 49 89 c8 48 c1 e9 10 81 e1 ff 03 00 00 44 0f b6 8c 0a 00 04
> RSP: 002b:00007f3d50bebd38 EFLAGS: 00000287
> RAX: 000000c001a0d140 RBX: 000000c001bba000 RCX: 000000c001a0c000
> RDX: 00007f3d51bef000 RSI: 000000c000025240 RDI: 00007f3d791daa28
> RBP: 00007f3d50bebd78 R08: 0000000000203000 R09: 00007f3d4c2f6001
> R10: 000000000000008a R11: 0000000000004048 R12: 0000000000000004
> R13: 000000c001a0d140 R14: 000000c000007520 R15: 0000000000000180
>  </TASK>
>
> Allocated by task 6664:
>  kasan_save_stack mm/kasan/common.c:38 [inline]
>  kasan_set_track mm/kasan/common.c:45 [inline]
>  set_alloc_info mm/kasan/common.c:437 [inline]
>  __kasan_slab_alloc+0xa3/0xd0 mm/kasan/common.c:470
>  kasan_slab_alloc include/linux/kasan.h:224 [inline]
>  slab_post_alloc_hook mm/slab.h:727 [inline]
>  slab_alloc_node mm/slub.c:3248 [inline]
>  slab_alloc mm/slub.c:3256 [inline]
>  __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
>  kmem_cache_alloc+0x2e6/0x450 mm/slub.c:3273
>  kmem_cache_zalloc include/linux/slab.h:723 [inline]
>  net_alloc net/core/net_namespace.c:404 [inline]
>  copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
>  create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
>  unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
>  ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
>  __do_sys_unshare kernel/fork.c:3254 [inline]
>  __se_sys_unshare kernel/fork.c:3252 [inline]
>  __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
>  do_syscall_x64 arch/x86/entry/common.c:51 [inline]
>  do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
>  entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> Freed by task 6874:
>  kasan_save_stack mm/kasan/common.c:38 [inline]
>  kasan_set_track+0x3d/0x60 mm/kasan/common.c:45
>  kasan_set_free_info+0x1f/0x40 mm/kasan/generic.c:370
>  ____kasan_slab_free+0x134/0x1c0 mm/kasan/common.c:367
>  kasan_slab_free include/linux/kasan.h:200 [inline]
>  slab_free_hook mm/slub.c:1759 [inline]
>  slab_free_freelist_hook+0x278/0x370 mm/slub.c:1785
>  slab_free mm/slub.c:3539 [inline]
>  kmem_cache_free+0x11a/0x310 mm/slub.c:3556
>  net_free net/core/net_namespace.c:433 [inline]
>  cleanup_net+0xd68/0xe20 net/core/net_namespace.c:616
>  process_one_work+0x83f/0x11a0 kernel/workqueue.c:2289
>  worker_thread+0xa6c/0x1290 kernel/workqueue.c:2436
>  kthread+0x28a/0x320 kernel/kthread.c:376
>  ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
>
> Last potentially related work creation:
>  kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
>  __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
>  insert_work+0x54/0x400 kernel/workqueue.c:1358
>  __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
>  call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
>  expire_timers kernel/time/timer.c:1514 [inline]
>  __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
>  __do_softirq+0x372/0x783 kernel/softirq.c:571
>
> Second to last potentially related work creation:
>  kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
>  __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
>  insert_work+0x54/0x400 kernel/workqueue.c:1358
>  __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
>  call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
>  expire_timers kernel/time/timer.c:1514 [inline]
>  __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
>  __do_softirq+0x372/0x783 kernel/softirq.c:571
>
> The buggy address belongs to the object at ffff888046bc8000
>  which belongs to the cache net_namespace of size 6784
> The buggy address is located 1701 bytes inside of
>  6784-byte region [ffff888046bc8000, ffff888046bc9a80)
>
> The buggy address belongs to the physical page:
> page:ffffea00011af200 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x46bc8
> head:ffffea00011af200 order:3 compound_mapcount:0 compound_pincount:0
> flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
> raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888013618f00
> raw: 0000000000000000 0000000080040004 00000001ffffffff 0000000000000000
> page dumped because: kasan: bad access detected
> page_owner tracks the page as allocated
> page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 6664, tgid 6664 (syz-executor.0), ts 88505587135, free_ts 0
>  prep_new_page mm/page_alloc.c:2532 [inline]
>  get_page_from_freelist+0x800/0xc10 mm/page_alloc.c:4283
>  __alloc_pages+0x2f0/0x650 mm/page_alloc.c:5549
>  alloc_slab_page mm/slub.c:1829 [inline]
>  allocate_slab+0x1eb/0xc00 mm/slub.c:1974
>  new_slab mm/slub.c:2034 [inline]
>  ___slab_alloc+0x581/0xff0 mm/slub.c:3036
>  __slab_alloc mm/slub.c:3123 [inline]
>  slab_alloc_node mm/slub.c:3214 [inline]
>  slab_alloc mm/slub.c:3256 [inline]
>  __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
>  kmem_cache_alloc+0x386/0x450 mm/slub.c:3273
>  kmem_cache_zalloc include/linux/slab.h:723 [inline]
>  net_alloc net/core/net_namespace.c:404 [inline]
>  copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
>  create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
>  unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
>  ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
>  __do_sys_unshare kernel/fork.c:3254 [inline]
>  __se_sys_unshare kernel/fork.c:3252 [inline]
>  __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
>  do_syscall_x64 arch/x86/entry/common.c:51 [inline]
>  do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
>  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> page_owner free stack trace missing
>
> Memory state around the buggy address:
>  ffff888046bc8580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>  ffff888046bc8600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> >ffff888046bc8680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>                                ^
>  ffff888046bc8700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>  ffff888046bc8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ==================================================================

^ permalink raw reply	[flat|nested] 7+ messages in thread

* KASAN: use-after-free Read in tcp_write_timer_handler
@ 2023-04-05 10:41 Dae R. Jeong
  2023-04-05 11:28 ` Eric Dumazet
  0 siblings, 1 reply; 7+ messages in thread
From: Dae R. Jeong @ 2023-04-05 10:41 UTC (permalink / raw)
  To: edumazet, davem, yoshfuji, dsahern, kuba, pabeni, netdev,
	linux-kernel, bpf

Hi,

We observed an issue "KASAN: use-after-free Read in tcp_write_timer_handler" during fuzzing.

Unfortunately, we have not found a reproducer for the crash yet. We
will inform you if we have any update on this crash.  Detailed crash
information is attached below.

Best regards,
Dae R. Jeong

-----
- Kernel version:
6.0-rc7

- Crash report:
==================================================================
BUG: KASAN: use-after-free in tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
BUG: KASAN: use-after-free in tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
Read of size 1 at addr ffff888046bc86a5 by task syz-fuzzer/6625

CPU: 0 PID: 6625 Comm: syz-fuzzer Not tainted 6.0.0-rc7-00167-g92162e4a9862 #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1cf/0x2b7 lib/dump_stack.c:106
 print_address_description+0x21/0x470 mm/kasan/report.c:317
 print_report+0x108/0x1f0 mm/kasan/report.c:433
 kasan_report+0xe5/0x110 mm/kasan/report.c:495
 tcp_probe_timer net/ipv4/tcp_timer.c:378 [inline]
 tcp_write_timer_handler+0x921/0xa60 net/ipv4/tcp_timer.c:624
 tcp_write_timer+0x1a5/0x2c0 net/ipv4/tcp_timer.c:637
 call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
 expire_timers kernel/time/timer.c:1519 [inline]
 __run_timers+0x76f/0x980 kernel/time/timer.c:1790
 run_timer_softirq+0x63/0xf0 kernel/time/timer.c:1803
 __do_softirq+0x372/0x783 kernel/softirq.c:571
 __irq_exit_rcu+0xcf/0x160 kernel/softirq.c:650
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
 sysvec_apic_timer_interrupt+0x43/0xb0 arch/x86/kernel/apic/apic.c:1106
 asm_sysvec_apic_timer_interrupt+0x16/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0033:0x421fb1
Code: 90 48 8b 4f 18 90 49 b8 00 00 00 00 00 80 00 00 49 01 c8 49 c1 e8 1a 66 90 49 81 f8 00 00 40 00 0f 83 e2 00 00 00 4a 8b 14 c2 <84> 02 49 89 c8 48 c1 e9 10 81 e1 ff 03 00 00 44 0f b6 8c 0a 00 04
RSP: 002b:00007f3d50bebd38 EFLAGS: 00000287
RAX: 000000c001a0d140 RBX: 000000c001bba000 RCX: 000000c001a0c000
RDX: 00007f3d51bef000 RSI: 000000c000025240 RDI: 00007f3d791daa28
RBP: 00007f3d50bebd78 R08: 0000000000203000 R09: 00007f3d4c2f6001
R10: 000000000000008a R11: 0000000000004048 R12: 0000000000000004
R13: 000000c001a0d140 R14: 000000c000007520 R15: 0000000000000180
 </TASK>

Allocated by task 6664:
 kasan_save_stack mm/kasan/common.c:38 [inline]
 kasan_set_track mm/kasan/common.c:45 [inline]
 set_alloc_info mm/kasan/common.c:437 [inline]
 __kasan_slab_alloc+0xa3/0xd0 mm/kasan/common.c:470
 kasan_slab_alloc include/linux/kasan.h:224 [inline]
 slab_post_alloc_hook mm/slab.h:727 [inline]
 slab_alloc_node mm/slub.c:3248 [inline]
 slab_alloc mm/slub.c:3256 [inline]
 __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
 kmem_cache_alloc+0x2e6/0x450 mm/slub.c:3273
 kmem_cache_zalloc include/linux/slab.h:723 [inline]
 net_alloc net/core/net_namespace.c:404 [inline]
 copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
 create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
 unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
 ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
 __do_sys_unshare kernel/fork.c:3254 [inline]
 __se_sys_unshare kernel/fork.c:3252 [inline]
 __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
 entry_SYSCALL_64_after_hwframe+0x63/0xcd

Freed by task 6874:
 kasan_save_stack mm/kasan/common.c:38 [inline]
 kasan_set_track+0x3d/0x60 mm/kasan/common.c:45
 kasan_set_free_info+0x1f/0x40 mm/kasan/generic.c:370
 ____kasan_slab_free+0x134/0x1c0 mm/kasan/common.c:367
 kasan_slab_free include/linux/kasan.h:200 [inline]
 slab_free_hook mm/slub.c:1759 [inline]
 slab_free_freelist_hook+0x278/0x370 mm/slub.c:1785
 slab_free mm/slub.c:3539 [inline]
 kmem_cache_free+0x11a/0x310 mm/slub.c:3556
 net_free net/core/net_namespace.c:433 [inline]
 cleanup_net+0xd68/0xe20 net/core/net_namespace.c:616
 process_one_work+0x83f/0x11a0 kernel/workqueue.c:2289
 worker_thread+0xa6c/0x1290 kernel/workqueue.c:2436
 kthread+0x28a/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306

Last potentially related work creation:
 kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
 __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
 insert_work+0x54/0x400 kernel/workqueue.c:1358
 __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
 call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
 expire_timers kernel/time/timer.c:1514 [inline]
 __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
 __do_softirq+0x372/0x783 kernel/softirq.c:571

Second to last potentially related work creation:
 kasan_save_stack+0x2b/0x50 mm/kasan/common.c:38
 __kasan_record_aux_stack+0xac/0xc0 mm/kasan/generic.c:348
 insert_work+0x54/0x400 kernel/workqueue.c:1358
 __queue_work+0xa95/0xe00 kernel/workqueue.c:1517
 call_timer_fn+0xf6/0x220 kernel/time/timer.c:1474
 expire_timers kernel/time/timer.c:1514 [inline]
 __run_timers+0x7a2/0x980 kernel/time/timer.c:1790
 __do_softirq+0x372/0x783 kernel/softirq.c:571

The buggy address belongs to the object at ffff888046bc8000
 which belongs to the cache net_namespace of size 6784
The buggy address is located 1701 bytes inside of
 6784-byte region [ffff888046bc8000, ffff888046bc9a80)

The buggy address belongs to the physical page:
page:ffffea00011af200 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x46bc8
head:ffffea00011af200 order:3 compound_mapcount:0 compound_pincount:0
flags: 0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000010200 0000000000000000 dead000000000122 ffff888013618f00
raw: 0000000000000000 0000000080040004 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 3, migratetype Unmovable, gfp_mask 0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 6664, tgid 6664 (syz-executor.0), ts 88505587135, free_ts 0
 prep_new_page mm/page_alloc.c:2532 [inline]
 get_page_from_freelist+0x800/0xc10 mm/page_alloc.c:4283
 __alloc_pages+0x2f0/0x650 mm/page_alloc.c:5549
 alloc_slab_page mm/slub.c:1829 [inline]
 allocate_slab+0x1eb/0xc00 mm/slub.c:1974
 new_slab mm/slub.c:2034 [inline]
 ___slab_alloc+0x581/0xff0 mm/slub.c:3036
 __slab_alloc mm/slub.c:3123 [inline]
 slab_alloc_node mm/slub.c:3214 [inline]
 slab_alloc mm/slub.c:3256 [inline]
 __kmem_cache_alloc_lru mm/slub.c:3263 [inline]
 kmem_cache_alloc+0x386/0x450 mm/slub.c:3273
 kmem_cache_zalloc include/linux/slab.h:723 [inline]
 net_alloc net/core/net_namespace.c:404 [inline]
 copy_net_ns+0x193/0x6d0 net/core/net_namespace.c:459
 create_new_namespaces+0x4db/0xa40 kernel/nsproxy.c:110
 unshare_nsproxy_namespaces+0x11e/0x180 kernel/nsproxy.c:226
 ksys_unshare+0x5a9/0xbc0 kernel/fork.c:3183
 __do_sys_unshare kernel/fork.c:3254 [inline]
 __se_sys_unshare kernel/fork.c:3252 [inline]
 __x64_sys_unshare+0x34/0x40 kernel/fork.c:3252
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x4e/0xa0 arch/x86/entry/common.c:82
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
page_owner free stack trace missing

Memory state around the buggy address:
 ffff888046bc8580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff888046bc8600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888046bc8680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                               ^
 ffff888046bc8700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff888046bc8780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-04-05 22:17 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-22  1:43 KASAN: use-after-free Read in tcp_write_timer_handler Hao Sun
2021-09-22  1:56 ` Eric Dumazet
2023-04-05 10:41 Dae R. Jeong
2023-04-05 11:28 ` Eric Dumazet
2023-04-05 19:41   ` Kuniyuki Iwashima
2023-04-05 19:47     ` Eric Dumazet
2023-04-05 22:17       ` Dae R. Jeong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).