[syzbot] WARNING in sk_stream_kill_queues (8)

[syzbot] WARNING in sk_stream_kill_queues (8)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    aab35c3d5112 Add linux-next specific files for 20220627
git tree:       linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=126fef90080000
kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14ae0c98080000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145124f4080000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com

nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
------------[ cut here ]------------
WARNING: CPU: 1 PID: 3601 at net/core/stream.c:205 sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
Modules linked in:
CPU: 1 PID: 3601 Comm: syz-executor340 Not tainted 5.19.0-rc4-next-20220627-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
Code: 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e ec 00 00 00 8b ab 28 02 00 00 e9 60 ff ff ff e8 3b 9a 29 fa 0f 0b eb 97 e8 32 9a 29 fa <0f> 0b eb a0 e8 29 9a 29 fa 0f 0b e9 6a fe ff ff e8 0d a1 75 fa e9
RSP: 0018:ffffc90002e6fbf0 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff88801e90ba80 RSI: ffffffff87511cce RDI: 0000000000000005
RBP: 0000000000000b00 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000b00 R11: 0000000000000004 R12: ffff88801e0c8e28
R13: ffffffff913121c0 R14: ffff88801e0c8c28 R15: ffff88801e0c8db8
FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000045b630 CR3: 000000000ba8e000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 inet_csk_destroy_sock+0x1a5/0x440 net/ipv4/inet_connection_sock.c:1013
 __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
 tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
 inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
 __sock_release+0xcd/0x280 net/socket.c:650
 sock_close+0x18/0x20 net/socket.c:1365
 __fput+0x277/0x9d0 fs/file_table.c:317
 task_work_run+0xdd/0x1a0 kernel/task_work.c:177
 exit_task_work include/linux/task_work.h:38 [inline]
 do_exit+0xaf1/0x29f0 kernel/exit.c:795
 do_group_exit+0xd2/0x2f0 kernel/exit.c:925
 __do_sys_exit_group kernel/exit.c:936 [inline]
 __se_sys_exit_group kernel/exit.c:934 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x46/0xb0
RIP: 0033:0x7f080e760989
Code: Unable to access opcode bytes at RIP 0x7f080e76095f.
RSP: 002b:00007ffcee785818 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007f080e7d4270 RCX: 00007f080e760989
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
R10: 0000000000000010 R11: 0000000000000246 R12: 00007f080e7d4270
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[Eric Dumazet]

On Wed, Jun 29, 2022 at 7:41 PM syzbot
<syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:    aab35c3d5112 Add linux-next specific files for 20220627
> git tree:       linux-next
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=126fef90080000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
> dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
> compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14ae0c98080000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145124f4080000
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com
>
> nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 3601 at net/core/stream.c:205 sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> Modules linked in:
> CPU: 1 PID: 3601 Comm: syz-executor340 Not tainted 5.19.0-rc4-next-20220627-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> RIP: 0010:sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> Code: 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e ec 00 00 00 8b ab 28 02 00 00 e9 60 ff ff ff e8 3b 9a 29 fa 0f 0b eb 97 e8 32 9a 29 fa <0f> 0b eb a0 e8 29 9a 29 fa 0f 0b e9 6a fe ff ff e8 0d a1 75 fa e9
> RSP: 0018:ffffc90002e6fbf0 EFLAGS: 00010293
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff88801e90ba80 RSI: ffffffff87511cce RDI: 0000000000000005
> RBP: 0000000000000b00 R08: 0000000000000005 R09: 0000000000000000
> R10: 0000000000000b00 R11: 0000000000000004 R12: ffff88801e0c8e28
> R13: ffffffff913121c0 R14: ffff88801e0c8c28 R15: ffff88801e0c8db8
> FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000000000045b630 CR3: 000000000ba8e000 CR4: 00000000003506e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  inet_csk_destroy_sock+0x1a5/0x440 net/ipv4/inet_connection_sock.c:1013
>  __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
>  tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
>  inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
>  __sock_release+0xcd/0x280 net/socket.c:650
>  sock_close+0x18/0x20 net/socket.c:1365
>  __fput+0x277/0x9d0 fs/file_table.c:317
>  task_work_run+0xdd/0x1a0 kernel/task_work.c:177
>  exit_task_work include/linux/task_work.h:38 [inline]
>  do_exit+0xaf1/0x29f0 kernel/exit.c:795
>  do_group_exit+0xd2/0x2f0 kernel/exit.c:925
>  __do_sys_exit_group kernel/exit.c:936 [inline]
>  __se_sys_exit_group kernel/exit.c:934 [inline]
>  __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
>  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
>  do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
>  entry_SYSCALL_64_after_hwframe+0x46/0xb0
> RIP: 0033:0x7f080e760989
> Code: Unable to access opcode bytes at RIP 0x7f080e76095f.
> RSP: 002b:00007ffcee785818 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> RAX: ffffffffffffffda RBX: 00007f080e7d4270 RCX: 00007f080e760989
> RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
> R10: 0000000000000010 R11: 0000000000000246 R12: 00007f080e7d4270
> R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
>  </TASK>
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> syzbot can test patches for this issue, for details see:
> https://goo.gl/tpsmEJ#testing-patches

Stanislav has bisected the issue to:

commit 965b57b469a589d64d81b1688b38dcb537011bb0
Author: Cong Wang <cong.wang@bytedance.com>
Date:   Wed Jun 15 09:20:12 2022 -0700

    net: Introduce a new proto_ops ->read_skb()

    Currently both splice() and sockmap use ->read_sock() to
    read skb from receive queue, but for sockmap we only read
    one entire skb at a time, so ->read_sock() is too conservative
    to use. Introduce a new proto_ops ->read_skb() which supports
    this sematic, with this we can finally pass the ownership of
    skb to recv actors.

    For non-TCP protocols, all ->read_sock() can be simply
    converted to ->read_skb().

    Signed-off-by: Cong Wang <cong.wang@bytedance.com>
    Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
    Reviewed-by: John Fastabend <john.fastabend@gmail.com>
    Link: https://lore.kernel.org/bpf/20220615162014.89193-3-xiyou.wangcong@gmail.com

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[Eric Dumazet]

On Wed, Jun 29, 2022 at 7:45 PM Eric Dumazet <edumazet@google.com> wrote:
>
> On Wed, Jun 29, 2022 at 7:41 PM syzbot
> <syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:    aab35c3d5112 Add linux-next specific files for 20220627
> > git tree:       linux-next
> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=126fef90080000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
> > dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
> > compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14ae0c98080000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145124f4080000
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com
> >
> > nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
> > ------------[ cut here ]------------
> > WARNING: CPU: 1 PID: 3601 at net/core/stream.c:205 sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > Modules linked in:
> > CPU: 1 PID: 3601 Comm: syz-executor340 Not tainted 5.19.0-rc4-next-20220627-syzkaller #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > RIP: 0010:sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > Code: 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e ec 00 00 00 8b ab 28 02 00 00 e9 60 ff ff ff e8 3b 9a 29 fa 0f 0b eb 97 e8 32 9a 29 fa <0f> 0b eb a0 e8 29 9a 29 fa 0f 0b e9 6a fe ff ff e8 0d a1 75 fa e9
> > RSP: 0018:ffffc90002e6fbf0 EFLAGS: 00010293
> > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > RDX: ffff88801e90ba80 RSI: ffffffff87511cce RDI: 0000000000000005
> > RBP: 0000000000000b00 R08: 0000000000000005 R09: 0000000000000000
> > R10: 0000000000000b00 R11: 0000000000000004 R12: ffff88801e0c8e28
> > R13: ffffffff913121c0 R14: ffff88801e0c8c28 R15: ffff88801e0c8db8
> > FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 000000000045b630 CR3: 000000000ba8e000 CR4: 00000000003506e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > Call Trace:
> >  <TASK>
> >  inet_csk_destroy_sock+0x1a5/0x440 net/ipv4/inet_connection_sock.c:1013
> >  __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
> >  tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
> >  inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
> >  __sock_release+0xcd/0x280 net/socket.c:650
> >  sock_close+0x18/0x20 net/socket.c:1365
> >  __fput+0x277/0x9d0 fs/file_table.c:317
> >  task_work_run+0xdd/0x1a0 kernel/task_work.c:177
> >  exit_task_work include/linux/task_work.h:38 [inline]
> >  do_exit+0xaf1/0x29f0 kernel/exit.c:795
> >  do_group_exit+0xd2/0x2f0 kernel/exit.c:925
> >  __do_sys_exit_group kernel/exit.c:936 [inline]
> >  __se_sys_exit_group kernel/exit.c:934 [inline]
> >  __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
> >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> >  do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
> >  entry_SYSCALL_64_after_hwframe+0x46/0xb0
> > RIP: 0033:0x7f080e760989
> > Code: Unable to access opcode bytes at RIP 0x7f080e76095f.
> > RSP: 002b:00007ffcee785818 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> > RAX: ffffffffffffffda RBX: 00007f080e7d4270 RCX: 00007f080e760989
> > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> > RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
> > R10: 0000000000000010 R11: 0000000000000246 R12: 00007f080e7d4270
> > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
> >  </TASK>
> >
> >
> > ---
> > This report is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at syzkaller@googlegroups.com.
> >
> > syzbot will keep track of this issue. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > syzbot can test patches for this issue, for details see:
> > https://goo.gl/tpsmEJ#testing-patches
>
> Stanislav has bisected the issue to:
>
> commit 965b57b469a589d64d81b1688b38dcb537011bb0
> Author: Cong Wang <cong.wang@bytedance.com>
> Date:   Wed Jun 15 09:20:12 2022 -0700
>
>     net: Introduce a new proto_ops ->read_skb()
>
>     Currently both splice() and sockmap use ->read_sock() to
>     read skb from receive queue, but for sockmap we only read
>     one entire skb at a time, so ->read_sock() is too conservative
>     to use. Introduce a new proto_ops ->read_skb() which supports
>     this sematic, with this we can finally pass the ownership of
>     skb to recv actors.
>
>     For non-TCP protocols, all ->read_sock() can be simply
>     converted to ->read_skb().
>
>     Signed-off-by: Cong Wang <cong.wang@bytedance.com>
>     Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
>     Reviewed-by: John Fastabend <john.fastabend@gmail.com>
>     Link: https://lore.kernel.org/bpf/20220615162014.89193-3-xiyou.wangcong@gmail.com

Repro is doing something like:


mmap(0x1ffff000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
-1, 0) = 0x1ffff000
mmap(0x20000000, 16777216, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x20000000
mmap(0x21000000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
-1, 0) = 0x21000000
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SK_SKB, insn_cnt=4,
insns=0x20000040, license="GPL", log_level=4, log_size=64912,
log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
prog_name="", prog_ifindex=0,
expected_attach_type=BPF_CGROUP_INET_INGRESS}, 72) = 4
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_SOCKHASH, key_size=4,
value_size=4, max_entries=18, map_flags=0, inner_map_fd=-1,
map_name="", map_ifindex=0, btf_fd=-1, btf_key_type_id=0,
btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) =
5
bpf(BPF_PROG_ATTACH, {target_fd=5, attach_bpf_fd=4,
attach_type=BPF_SK_SKB_STREAM_VERDICT, attach_flags=0}, 16) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(20000),
sin_addr=inet_addr("224.0.0.2")}, 16) = 0
sendto(3, NULL, 0, MSG_OOB|MSG_SENDPAGE_NOTLAST|MSG_FASTOPEN,
{sa_family=AF_INET, sin_port=htons(20000),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x200002c0, value=0x20000340,
flags=BPF_ANY}, 32) = 0
shutdown(3, SHUT_WR)                    = 0
exit_group(0)                           = ?

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[John Fastabend]

Eric Dumazet wrote:
> On Wed, Jun 29, 2022 at 7:45 PM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Wed, Jun 29, 2022 at 7:41 PM syzbot
> > <syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    aab35c3d5112 Add linux-next specific files for 20220627
> > > git tree:       linux-next
> > > console+strace: https://syzkaller.appspot.com/x/log.txt?x=126fef90080000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
> > > compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14ae0c98080000
> > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145124f4080000
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com
> > >
> > > nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
> > > ------------[ cut here ]------------
> > > WARNING: CPU: 1 PID: 3601 at net/core/stream.c:205 sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > > Modules linked in:
> > > CPU: 1 PID: 3601 Comm: syz-executor340 Not tainted 5.19.0-rc4-next-20220627-syzkaller #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > > RIP: 0010:sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > > Code: 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e ec 00 00 00 8b ab 28 02 00 00 e9 60 ff ff ff e8 3b 9a 29 fa 0f 0b eb 97 e8 32 9a 29 fa <0f> 0b eb a0 e8 29 9a 29 fa 0f 0b e9 6a fe ff ff e8 0d a1 75 fa e9
> > > RSP: 0018:ffffc90002e6fbf0 EFLAGS: 00010293
> > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > > RDX: ffff88801e90ba80 RSI: ffffffff87511cce RDI: 0000000000000005
> > > RBP: 0000000000000b00 R08: 0000000000000005 R09: 0000000000000000
> > > R10: 0000000000000b00 R11: 0000000000000004 R12: ffff88801e0c8e28
> > > R13: ffffffff913121c0 R14: ffff88801e0c8c28 R15: ffff88801e0c8db8
> > > FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
> > > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > CR2: 000000000045b630 CR3: 000000000ba8e000 CR4: 00000000003506e0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > Call Trace:
> > >  <TASK>
> > >  inet_csk_destroy_sock+0x1a5/0x440 net/ipv4/inet_connection_sock.c:1013
> > >  __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
> > >  tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
> > >  inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
> > >  __sock_release+0xcd/0x280 net/socket.c:650
> > >  sock_close+0x18/0x20 net/socket.c:1365
> > >  __fput+0x277/0x9d0 fs/file_table.c:317
> > >  task_work_run+0xdd/0x1a0 kernel/task_work.c:177
> > >  exit_task_work include/linux/task_work.h:38 [inline]
> > >  do_exit+0xaf1/0x29f0 kernel/exit.c:795
> > >  do_group_exit+0xd2/0x2f0 kernel/exit.c:925
> > >  __do_sys_exit_group kernel/exit.c:936 [inline]
> > >  __se_sys_exit_group kernel/exit.c:934 [inline]
> > >  __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
> > >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > >  do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
> > >  entry_SYSCALL_64_after_hwframe+0x46/0xb0
> > > RIP: 0033:0x7f080e760989
> > > Code: Unable to access opcode bytes at RIP 0x7f080e76095f.
> > > RSP: 002b:00007ffcee785818 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> > > RAX: ffffffffffffffda RBX: 00007f080e7d4270 RCX: 00007f080e760989
> > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> > > RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
> > > R10: 0000000000000010 R11: 0000000000000246 R12: 00007f080e7d4270
> > > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
> > >  </TASK>
> > >
> > >
> > > ---
> > > This report is generated by a bot. It may contain errors.
> > > See https://goo.gl/tpsmEJ for more information about syzbot.
> > > syzbot engineers can be reached at syzkaller@googlegroups.com.
> > >
> > > syzbot will keep track of this issue. See:
> > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > > syzbot can test patches for this issue, for details see:
> > > https://goo.gl/tpsmEJ#testing-patches
> >
> > Stanislav has bisected the issue to:
> >
> > commit 965b57b469a589d64d81b1688b38dcb537011bb0
> > Author: Cong Wang <cong.wang@bytedance.com>
> > Date:   Wed Jun 15 09:20:12 2022 -0700
> >
> >     net: Introduce a new proto_ops ->read_skb()
> >
> >     Currently both splice() and sockmap use ->read_sock() to
> >     read skb from receive queue, but for sockmap we only read
> >     one entire skb at a time, so ->read_sock() is too conservative
> >     to use. Introduce a new proto_ops ->read_skb() which supports
> >     this sematic, with this we can finally pass the ownership of
> >     skb to recv actors.
> >
> >     For non-TCP protocols, all ->read_sock() can be simply
> >     converted to ->read_skb().
> >
> >     Signed-off-by: Cong Wang <cong.wang@bytedance.com>
> >     Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> >     Reviewed-by: John Fastabend <john.fastabend@gmail.com>
> >     Link: https://lore.kernel.org/bpf/20220615162014.89193-3-xiyou.wangcong@gmail.com
> 
> Repro is doing something like:
> 
> 
> mmap(0x1ffff000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
> -1, 0) = 0x1ffff000
> mmap(0x20000000, 16777216, PROT_READ|PROT_WRITE|PROT_EXEC,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x20000000
> mmap(0x21000000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
> -1, 0) = 0x21000000
> socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SK_SKB, insn_cnt=4,
> insns=0x20000040, license="GPL", log_level=4, log_size=64912,
> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
> prog_name="", prog_ifindex=0,
> expected_attach_type=BPF_CGROUP_INET_INGRESS}, 72) = 4
> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_SOCKHASH, key_size=4,
> value_size=4, max_entries=18, map_flags=0, inner_map_fd=-1,
> map_name="", map_ifindex=0, btf_fd=-1, btf_key_type_id=0,
> btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 72) =
> 5
> bpf(BPF_PROG_ATTACH, {target_fd=5, attach_bpf_fd=4,
> attach_type=BPF_SK_SKB_STREAM_VERDICT, attach_flags=0}, 16) = 0
> bind(3, {sa_family=AF_INET, sin_port=htons(20000),
> sin_addr=inet_addr("224.0.0.2")}, 16) = 0
> sendto(3, NULL, 0, MSG_OOB|MSG_SENDPAGE_NOTLAST|MSG_FASTOPEN,
> {sa_family=AF_INET, sin_port=htons(20000),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x200002c0, value=0x20000340,
> flags=BPF_ANY}, 32) = 0
> shutdown(3, SHUT_WR)                    = 0
> exit_group(0)                           = ?

Thanks Eric, Stanislav for the bisect. I'll take a look this afternoon. 

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[Cong Wang]

On Wed, Jun 29, 2022 at 07:46:35PM +0200, Eric Dumazet wrote:
> On Wed, Jun 29, 2022 at 7:45 PM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Wed, Jun 29, 2022 at 7:41 PM syzbot
> > <syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    aab35c3d5112 Add linux-next specific files for 20220627
> > > git tree:       linux-next
> > > console+strace: https://syzkaller.appspot.com/x/log.txt?x=126fef90080000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
> > > compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=14ae0c98080000
> > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=145124f4080000
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com
> > >
> > > nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
> > > ------------[ cut here ]------------
> > > WARNING: CPU: 1 PID: 3601 at net/core/stream.c:205 sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > > Modules linked in:
> > > CPU: 1 PID: 3601 Comm: syz-executor340 Not tainted 5.19.0-rc4-next-20220627-syzkaller #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> > > RIP: 0010:sk_stream_kill_queues+0x2ee/0x3d0 net/core/stream.c:205
> > > Code: 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e ec 00 00 00 8b ab 28 02 00 00 e9 60 ff ff ff e8 3b 9a 29 fa 0f 0b eb 97 e8 32 9a 29 fa <0f> 0b eb a0 e8 29 9a 29 fa 0f 0b e9 6a fe ff ff e8 0d a1 75 fa e9
> > > RSP: 0018:ffffc90002e6fbf0 EFLAGS: 00010293
> > > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > > RDX: ffff88801e90ba80 RSI: ffffffff87511cce RDI: 0000000000000005
> > > RBP: 0000000000000b00 R08: 0000000000000005 R09: 0000000000000000
> > > R10: 0000000000000b00 R11: 0000000000000004 R12: ffff88801e0c8e28
> > > R13: ffffffff913121c0 R14: ffff88801e0c8c28 R15: ffff88801e0c8db8
> > > FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
> > > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > CR2: 000000000045b630 CR3: 000000000ba8e000 CR4: 00000000003506e0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > Call Trace:
> > >  <TASK>
> > >  inet_csk_destroy_sock+0x1a5/0x440 net/ipv4/inet_connection_sock.c:1013
> > >  __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
> > >  tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
> > >  inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
> > >  __sock_release+0xcd/0x280 net/socket.c:650
> > >  sock_close+0x18/0x20 net/socket.c:1365
> > >  __fput+0x277/0x9d0 fs/file_table.c:317
> > >  task_work_run+0xdd/0x1a0 kernel/task_work.c:177
> > >  exit_task_work include/linux/task_work.h:38 [inline]
> > >  do_exit+0xaf1/0x29f0 kernel/exit.c:795
> > >  do_group_exit+0xd2/0x2f0 kernel/exit.c:925
> > >  __do_sys_exit_group kernel/exit.c:936 [inline]
> > >  __se_sys_exit_group kernel/exit.c:934 [inline]
> > >  __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
> > >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > >  do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
> > >  entry_SYSCALL_64_after_hwframe+0x46/0xb0
> > > RIP: 0033:0x7f080e760989
> > > Code: Unable to access opcode bytes at RIP 0x7f080e76095f.
> > > RSP: 002b:00007ffcee785818 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> > > RAX: ffffffffffffffda RBX: 00007f080e7d4270 RCX: 00007f080e760989
> > > RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> > > RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
> > > R10: 0000000000000010 R11: 0000000000000246 R12: 00007f080e7d4270
> > > R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
> > >  </TASK>
> > >
> > >
> > > ---
> > > This report is generated by a bot. It may contain errors.
> > > See https://goo.gl/tpsmEJ for more information about syzbot.
> > > syzbot engineers can be reached at syzkaller@googlegroups.com.
> > >
> > > syzbot will keep track of this issue. See:
> > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > > syzbot can test patches for this issue, for details see:
> > > https://goo.gl/tpsmEJ#testing-patches
> >
> > Stanislav has bisected the issue to:
> >
> > commit 965b57b469a589d64d81b1688b38dcb537011bb0
> > Author: Cong Wang <cong.wang@bytedance.com>
> > Date:   Wed Jun 15 09:20:12 2022 -0700
> >
> >     net: Introduce a new proto_ops ->read_skb()
> >
> >     Currently both splice() and sockmap use ->read_sock() to
> >     read skb from receive queue, but for sockmap we only read
> >     one entire skb at a time, so ->read_sock() is too conservative
> >     to use. Introduce a new proto_ops ->read_skb() which supports
> >     this sematic, with this we can finally pass the ownership of
> >     skb to recv actors.
> >
> >     For non-TCP protocols, all ->read_sock() can be simply
> >     converted to ->read_skb().
> >
> >     Signed-off-by: Cong Wang <cong.wang@bytedance.com>
> >     Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> >     Reviewed-by: John Fastabend <john.fastabend@gmail.com>
> >     Link: https://lore.kernel.org/bpf/20220615162014.89193-3-xiyou.wangcong@gmail.com
> 
> Repro is doing something like:
> 

I will look into this tonight.

Thanks for the report!

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[Cong Wang]

On Wed, Jul 6, 2022 at 9:40 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
>
> I will look into this tonight.

The following patch could work. It uncharges the sk mem before passing skb
to recv actor but still keeps skb->sk.

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 9d2fd3ced21b..c6b1effb2afd 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1749,6 +1749,7 @@ int tcp_read_skb(struct sock *sk,
skb_read_actor_t recv_actor)
                int used;

                __skb_unlink(skb, &sk->sk_receive_queue);
+               WARN_ON(!skb_set_owner_sk_safe(skb, sk));
                used = recv_actor(sk, skb);
                if (used <= 0) {
                        if (!copied)

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[Stanislav Fomichev]

On Thu, Jul 7, 2022 at 11:39 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
>
> On Wed, Jul 6, 2022 at 9:40 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
> >
> > I will look into this tonight.
>
> The following patch could work. It uncharges the sk mem before passing skb
> to recv actor but still keeps skb->sk.

This seems to make syzkaller happy.

Tested-by: Stanislav Fomichev <sdf@google.com>

> diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> index 9d2fd3ced21b..c6b1effb2afd 100644
> --- a/net/ipv4/tcp.c
> +++ b/net/ipv4/tcp.c
> @@ -1749,6 +1749,7 @@ int tcp_read_skb(struct sock *sk,
> skb_read_actor_t recv_actor)
>                 int used;
>
>                 __skb_unlink(skb, &sk->sk_receive_queue);
> +               WARN_ON(!skb_set_owner_sk_safe(skb, sk));
>                 used = recv_actor(sk, skb);
>                 if (used <= 0) {
>                         if (!copied)

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+a0e6f8738b58f7654417@syzkaller.appspotmail.com

Tested on:

commit:         aab35c3d Add linux-next specific files for 20220627
git tree:       http://kernel.source.codeaurora.cn/pub/scm/linux/kernel/git/next/linux-next.git
console output: https://syzkaller.appspot.com/x/log.txt?x=10deb7e0080000
kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
patch:          https://syzkaller.appspot.com/x/patch.diff?x=1450de24080000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] WARNING in sk_stream_kill_queues (8)

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
BUG: sleeping function called from invalid context in sk_psock_stop

nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
BUG: sleeping function called from invalid context at kernel/workqueue.c:3010
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 14037, name: syz-executor244
preempt_count: 201, expected: 0
RCU nest depth: 0, expected: 0
3 locks held by syz-executor244/14037:
 #0: ffff88806b374410 (&sb->s_type->i_mutex_key#10){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:761 [inline]
 #0: ffff88806b374410 (&sb->s_type->i_mutex_key#10){+.+.}-{3:3}, at: __sock_release+0x86/0x280 net/socket.c:649
 #1: ffff88807ecaed30 (sk_lock-AF_INET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1664 [inline]
 #1: ffff88807ecaed30 (sk_lock-AF_INET){+.+.}-{0:0}, at: tcp_close+0x1e/0xc0 net/ipv4/tcp.c:2974
 #2: ffff88807ecaecb0 (slock-AF_INET){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:360 [inline]
 #2: ffff88807ecaecb0 (slock-AF_INET){+.-.}-{2:2}, at: __tcp_close+0x4d3/0xf50 net/ipv4/tcp.c:2897
Preemption disabled at:
[<ffffffff87dbd2fb>] local_bh_disable include/linux/bottom_half.h:20 [inline]
[<ffffffff87dbd2fb>] __tcp_close+0x4cb/0xf50 net/ipv4/tcp.c:2896
CPU: 1 PID: 14037 Comm: syz-executor244 Not tainted 5.19.0-rc4-next-20220627-syzkaller-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/18/2022
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 __might_resched.cold+0x222/0x26b kernel/sched/core.c:9859
 start_flush_work kernel/workqueue.c:3010 [inline]
 __flush_work+0x109/0xb10 kernel/workqueue.c:3074
 __cancel_work_timer+0x3f9/0x570 kernel/workqueue.c:3162
 sk_psock_stop+0x4cb/0x630 net/core/skmsg.c:810
 sock_map_destroy+0x333/0x760 net/core/sock_map.c:1581
 inet_csk_destroy_sock+0x196/0x440 net/ipv4/inet_connection_sock.c:1011
 __tcp_close+0xb92/0xf50 net/ipv4/tcp.c:2963
 tcp_close+0x29/0xc0 net/ipv4/tcp.c:2975
 sock_map_close+0x3b9/0x780 net/core/sock_map.c:1607
 inet_release+0x12e/0x270 net/ipv4/af_inet.c:428
 __sock_release+0xcd/0x280 net/socket.c:650
 sock_close+0x18/0x20 net/socket.c:1365
 __fput+0x277/0x9d0 fs/file_table.c:317
 task_work_run+0xdd/0x1a0 kernel/task_work.c:177
 exit_task_work include/linux/task_work.h:38 [inline]
 do_exit+0xaf1/0x29f0 kernel/exit.c:795
 do_group_exit+0xd2/0x2f0 kernel/exit.c:925
 __do_sys_exit_group kernel/exit.c:936 [inline]
 __se_sys_exit_group kernel/exit.c:934 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:934
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x46/0xb0
RIP: 0033:0x7f1abfde9989
Code: Unable to access opcode bytes at RIP 0x7f1abfde995f.
RSP: 002b:00007ffd0bbbdb28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007f1abfe5d270 RCX: 00007f1abfde9989
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000010
R10: 0000000000000010 R11: 0000000000000246 R12: 00007f1abfe


Tested on:

commit:         aab35c3d Add linux-next specific files for 20220627
git tree:       http://kernel.source.codeaurora.cn/pub/scm/linux/kernel/git/next/linux-next.git
console output: https://syzkaller.appspot.com/x/log.txt?x=15f66b98080000
kernel config:  https://syzkaller.appspot.com/x/.config?x=6a874f114a1e4a6b
dashboard link: https://syzkaller.appspot.com/bug?extid=a0e6f8738b58f7654417
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
patch:          https://syzkaller.appspot.com/x/patch.diff?x=15b63ef4080000