* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231126030321.950-1-hdanton@sina.com>
@ 2023-11-26 3:24 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-26 3:24 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch and the reproducer did not trigger any issue:
Reported-and-tested-by: syzbot+e18ac85757292b7baf96@syzkaller.appspotmail.com
Tested on:
commit: 090472ed Merge tag 'usb-6.7-rc3' of git://git.kernel.o..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=13959724e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=1191a4d8e80000
Note: testing is done by a robot and is best-effort only.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <tencent_F694D4E91AEE12CC2C7B566C7C2F7D6ECC0A@qq.com>
@ 2023-11-27 12:25 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-27 12:25 UTC (permalink / raw)
To: eadavis, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc3-syzkaller-g2cc14f52aeb7-dirty #0 Not tainted
--------------------------------------------
syz-executor.0/5355 is trying to acquire lock:
ffff88807f4474d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807f4474d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff88807f4474d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
but task is already holding lock:
ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x208/0x650 net/sched/sch_generic.c:342
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(_xmit_ETHER#2);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5355:
#0: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
#2: ffff8880755cc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff8880755cc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff8880755cc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff8880755cc258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d0/0x38e0 net/core/dev.c:4312
#3: ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff8880272278d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x208/0x650 net/sched/sch_generic.c:342
#4: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25aa0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25b00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
stack backtrace:
CPU: 0 PID: 5355 Comm: syz-executor.0 Not tainted 6.7.0-rc3-syzkaller-g2cc14f52aeb7-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6aa3/0x7fb0 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5754
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x303/0x650 net/sched/sch_generic.c:344
__dev_queue_xmit+0x187c/0x38e0 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f7eb66798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f7eb77fa0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f7eb678bf60 RCX: 00007f7eb66798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f7eb66d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f7eb678bf60 R15: 00007ffeb4c7fc58
</TASK>
Tested on:
commit: 2cc14f52 Linux 6.7-rc3
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=16f08d9f680000
kernel config: https://syzkaller.appspot.com/x/.config?x=a57e0d60eeda7b44
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=12057162e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <tencent_955C09A52EC46EC24C9327E746D852CD4606@qq.com>
@ 2023-11-26 10:33 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-26 10:33 UTC (permalink / raw)
To: eadavis, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc2-syzkaller-00242-g090472ed9c92-dirty #0 Not tainted
--------------------------------------------
syz-executor.0/5354 is trying to acquire lock:
ffff8880216e40d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff8880216e40d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff8880216e40d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
but task is already holding lock:
ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x24d/0x5f0 net/sched/sch_generic.c:341
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(_xmit_ETHER#2);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5354:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
#2: ffff88814b0da258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff88814b0da258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff88814b0da258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff88814b0da258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d0/0x38e0 net/core/dev.c:4312
#3: ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff88807a6cfcd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x24d/0x5f0 net/sched/sch_generic.c:341
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
stack backtrace:
CPU: 0 PID: 5354 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00242-g090472ed9c92-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6a81/0x7f70 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2db/0x5f0 net/sched/sch_generic.c:343
__dev_queue_xmit+0x187c/0x38e0 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f6d49e798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f6d4b01e0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f6d49f8bf60 RCX: 00007f6d49e798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f6d49ed5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f6d49f8bf60 R15: 00007ffe004ace78
</TASK>
syz-executor.0 (5354) used greatest stack depth: 18544 bytes left
Tested on:
commit: 090472ed Merge tag 'usb-6.7-rc3' of git://git.kernel.o..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=150d960ce80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=142cb6e8e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <tencent_94065D4991EECA6EDE4C8AE7C446C512F906@qq.com>
@ 2023-11-26 7:10 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-26 7:10 UTC (permalink / raw)
To: eadavis, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc2-syzkaller-00242-g090472ed9c92-dirty #0 Not tainted
--------------------------------------------
syz-executor.0/5358 is trying to acquire lock:
ffff88801eeed4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88801eeed4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff88801eeed4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
but task is already holding lock:
ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x610 net/sched/sch_generic.c:340
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(_xmit_ETHER#2);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5358:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
#2: ffff8881454ab258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff8881454ab258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff8881454ab258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff8881454ab258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d0/0x38e0 net/core/dev.c:4312
#3: ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff888025da68d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x610 net/sched/sch_generic.c:340
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
stack backtrace:
CPU: 1 PID: 5358 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00242-g090472ed9c92-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6a81/0x7f70 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2cc/0x610 net/sched/sch_generic.c:343
__dev_queue_xmit+0x187c/0x38e0 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f7cce0798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f7ccf23b0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f7cce18bf60 RCX: 00007f7cce0798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f7cce0d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f7cce18bf60 R15: 00007ffee66b2b98
</TASK>
Tested on:
commit: 090472ed Merge tag 'usb-6.7-rc3' of git://git.kernel.o..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=175326ece80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=11497694e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231126011259.821-1-hdanton@sina.com>
@ 2023-11-26 1:33 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-26 1:33 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING: bad unlock balance in __dev_queue_xmit
=====================================
WARNING: bad unlock balance detected!
6.7.0-rc2-syzkaller-00206-gb46ae77f6787-dirty #0 Not tainted
-------------------------------------
swapper/1/0 is trying to release lock (_xmit_ETHER) at:
[<ffffffff8854284e>] spin_unlock include/linux/spinlock.h:391 [inline]
[<ffffffff8854284e>] __netif_tx_unlock include/linux/netdevice.h:4441 [inline]
[<ffffffff8854284e>] __dev_queue_xmit+0x1ece/0x3950 net/core/dev.c:4354
but there are no more locks to release!
other info that might help us debug this:
3 locks held by swapper/1/0:
#0: ffffc900001f0c00 ((&lapb->t1timer)){+.-.}-{0:0}, at: call_timer_fn+0xc0/0x5e0 kernel/time/timer.c:1697
#1: ffff8880274dbdc0 (&lapb->lock){+.-.}-{2:2}, at: spin_lock_bh include/linux/spinlock.h:356 [inline]
#1: ffff8880274dbdc0 (&lapb->lock){+.-.}-{2:2}, at: lapb_t1timer_expiry+0x33/0xb20 net/lapb/lapb_timer.c:99
#2: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#2: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#2: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x3950 net/core/dev.c:4272
stack backtrace:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.7.0-rc2-syzkaller-00206-gb46ae77f6787-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
print_unlock_imbalance_bug+0x252/0x2c0 kernel/locking/lockdep.c:5193
lock_release+0x59d/0x9d0 kernel/locking/lockdep.c:5430
_raw_spin_unlock+0x16/0x40 include/linux/spinlock_api_smp.h:141
spin_unlock include/linux/spinlock.h:391 [inline]
__netif_tx_unlock include/linux/netdevice.h:4441 [inline]
__dev_queue_xmit+0x1ece/0x3950 net/core/dev.c:4354
lapb_data_transmit+0x89/0xa0 net/lapb/lapb_iface.c:447
lapb_transmit_buffer+0x168/0x1f0 net/lapb/lapb_out.c:149
lapb_t1timer_expiry+0x6b8/0xb20
call_timer_fn+0x17a/0x5e0 kernel/time/timer.c:1700
__run_timers+0x64f/0x860 kernel/time/timer.c:1751
run_timer_softirq+0x67/0xf0 kernel/time/timer.c:2035
__do_softirq+0x2bf/0x93a kernel/softirq.c:553
__irq_exit_rcu+0xf1/0x1b0 kernel/softirq.c:427
irq_exit_rcu+0x9/0x20 kernel/softirq.c:644
sysvec_apic_timer_interrupt+0x95/0xb0 arch/x86/kernel/apic/apic.c:1076
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:72 [inline]
RIP: 0010:acpi_safe_halt+0x20/0x30 drivers/acpi/processor_idle.c:113
Code: 7f 04 eb 36 66 0f 1f 44 00 00 65 48 8b 05 68 a8 96 75 48 f7 00 08 00 00 00 75 10 66 90 0f 00 2d 36 d4 97 00 f3 0f 1e fa fb f4 <fa> c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 89 fa ec 48 8b 05
RSP: 0018:ffffc90000187d08 EFLAGS: 00000246
RAX: ffff88801641bb80 RBX: ffff888016b8a064 RCX: 00000000003c8e21
RDX: 0000000000000001 RSI: ffff888016b8a000 RDI: ffff888016b8a064
RBP: 0000000000038e38 R08: ffff8880b9b36c0b R09: 1ffff11017366d81
R10: dffffc0000000000 R11: ffffed1017366d82 R12: ffff888017e39000
R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff8d236d60
acpi_idle_enter+0xe4/0x140 drivers/acpi/processor_idle.c:707
cpuidle_enter_state+0x10e/0x470 drivers/cpuidle/cpuidle.c:267
cpuidle_enter+0x5d/0x90 drivers/cpuidle/cpuidle.c:388
do_idle+0x374/0x5c0 kernel/sched/idle.c:134
cpu_startup_entry+0x41/0x60 kernel/sched/idle.c:380
start_secondary+0xee/0xf0 arch/x86/kernel/smpboot.c:336
secondary_startup_64_no_verify+0x167/0x16b
</TASK>
------------[ cut here ]------------
pvqspinlock: lock 0xffff888025c150c0 has corrupted value 0x0!
WARNING: CPU: 1 PID: 0 at kernel/locking/qspinlock_paravirt.h:510 __pv_queued_spin_unlock_slowpath+0x23b/0x2f0 kernel/locking/qspinlock_paravirt.h:508
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.7.0-rc2-syzkaller-00206-gb46ae77f6787-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
RIP: 0010:__pv_queued_spin_unlock_slowpath+0x23b/0x2f0 kernel/locking/qspinlock_paravirt.h:508
Code: e8 0a 70 71 f7 4c 89 f0 48 c1 e8 03 0f b6 04 18 84 c0 0f 85 9a 00 00 00 41 8b 16 48 c7 c7 40 c6 aa 8a 4c 89 f6 e8 45 19 db f6 <0f> 0b eb 95 44 89 f1 80 e1 07 38 c1 0f 8c 2e ff ff ff 4c 89 f7 e8
RSP: 0018:ffffc900001f0778 EFLAGS: 00010246
RAX: 9cb7c89c3804c300 RBX: dffffc0000000000 RCX: ffff88801641bb80
RDX: 0000000000000502 RSI: 0000000000000000 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff81524a02 R09: 1ffff9200003e044
R10: dffffc0000000000 R11: fffff5200003e045 R12: 1ffff11004b82a1a
R13: ffff888025c150d0 R14: ffff888025c150c0 R15: ffff888025c150c0
FS: 0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020004540 CR3: 000000000c930000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
__raw_callee_save___pv_queued_spin_unlock_slowpath+0x15/0x30
.slowpath+0x9/0x1a
do_raw_spin_unlock+0x13b/0x8b0 arch/x86/include/asm/paravirt.h:591
_raw_spin_unlock+0x1e/0x40 include/linux/spinlock_api_smp.h:142
spin_unlock include/linux/spinlock.h:391 [inline]
__netif_tx_unlock include/linux/netdevice.h:4441 [inline]
__dev_queue_xmit+0x1ece/0x3950 net/core/dev.c:4354
lapb_data_transmit+0x89/0xa0 net/lapb/lapb_iface.c:447
lapb_transmit_buffer+0x168/0x1f0 net/lapb/lapb_out.c:149
lapb_t1timer_expiry+0x6b8/0xb20
call_timer_fn+0x17a/0x5e0 kernel/time/timer.c:1700
__run_timers+0x64f/0x860 kernel/time/timer.c:1751
run_timer_softirq+0x67/0xf0 kernel/time/timer.c:2035
__do_softirq+0x2bf/0x93a kernel/softirq.c:553
__irq_exit_rcu+0xf1/0x1b0 kernel/softirq.c:427
irq_exit_rcu+0x9/0x20 kernel/softirq.c:644
sysvec_apic_timer_interrupt+0x95/0xb0 arch/x86/kernel/apic/apic.c:1076
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:72 [inline]
RIP: 0010:acpi_safe_halt+0x20/0x30 drivers/acpi/processor_idle.c:113
Code: 7f 04 eb 36 66 0f 1f 44 00 00 65 48 8b 05 68 a8 96 75 48 f7 00 08 00 00 00 75 10 66 90 0f 00 2d 36 d4 97 00 f3 0f 1e fa fb f4 <fa> c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 89 fa ec 48 8b 05
RSP: 0018:ffffc90000187d08 EFLAGS: 00000246
RAX: ffff88801641bb80 RBX: ffff888016b8a064 RCX: 00000000003c8e21
RDX: 0000000000000001 RSI: ffff888016b8a000 RDI: ffff888016b8a064
RBP: 0000000000038e38 R08: ffff8880b9b36c0b R09: 1ffff11017366d81
R10: dffffc0000000000 R11: ffffed1017366d82 R12: ffff888017e39000
R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff8d236d60
acpi_idle_enter+0xe4/0x140 drivers/acpi/processor_idle.c:707
cpuidle_enter_state+0x10e/0x470 drivers/cpuidle/cpuidle.c:267
cpuidle_enter+0x5d/0x90 drivers/cpuidle/cpuidle.c:388
do_idle+0x374/0x5c0 kernel/sched/idle.c:134
cpu_startup_entry+0x41/0x60 kernel/sched/idle.c:380
start_secondary+0xee/0xf0 arch/x86/kernel/smpboot.c:336
secondary_startup_64_no_verify+0x167/0x16b
</TASK>
----------------
Code disassembly (best guess):
0: 7f 04 jg 0x6
2: eb 36 jmp 0x3a
4: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
a: 65 48 8b 05 68 a8 96 mov %gs:0x7596a868(%rip),%rax # 0x7596a87a
11: 75
12: 48 f7 00 08 00 00 00 testq $0x8,(%rax)
19: 75 10 jne 0x2b
1b: 66 90 xchg %ax,%ax
1d: 0f 00 2d 36 d4 97 00 verw 0x97d436(%rip) # 0x97d45a
24: f3 0f 1e fa endbr64
28: fb sti
29: f4 hlt
* 2a: fa cli <-- trapping instruction
2b: c3 ret
2c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
33: 00 00 00
36: 0f 1f 40 00 nopl 0x0(%rax)
3a: 89 fa mov %edi,%edx
3c: ec in (%dx),%al
3d: 48 rex.W
3e: 8b .byte 0x8b
3f: 05 .byte 0x5
Tested on:
commit: b46ae77f Merge tag 'xfs-6.7-fixes-3' of git://git.kern..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=162a21c8e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=14cd35a4e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231125130757.765-1-hdanton@sina.com>
@ 2023-11-25 13:22 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-25 13:22 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING: bad unlock balance in __dev_queue_xmit
=====================================
WARNING: bad unlock balance detected!
6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0 Not tainted
-------------------------------------
syz-executor.0/5357 is trying to release lock (_xmit_ETHER) at:
[<ffffffff8854264e>] spin_unlock include/linux/spinlock.h:391 [inline]
[<ffffffff8854264e>] __netif_tx_unlock include/linux/netdevice.h:4441 [inline]
[<ffffffff8854264e>] __dev_queue_xmit+0x1dce/0x3940 net/core/dev.c:4353
but there are no more locks to release!
other info that might help us debug this:
6 locks held by syz-executor.0/5357:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x3940 net/core/dev.c:4272
#2: ffff88814ab86258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff88814ab86258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff88814ab86258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff88814ab86258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x10f4/0x3940 net/core/dev.c:4314
#3: ffff8880639ff8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff8880639ff8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff8880639ff8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:340
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x3940 net/core/dev.c:4272
stack backtrace:
CPU: 1 PID: 5357 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
print_unlock_imbalance_bug+0x252/0x2c0 kernel/locking/lockdep.c:5193
lock_release+0x59d/0x9d0 kernel/locking/lockdep.c:5430
_raw_spin_unlock+0x16/0x40 include/linux/spinlock_api_smp.h:141
spin_unlock include/linux/spinlock.h:391 [inline]
__netif_tx_unlock include/linux/netdevice.h:4441 [inline]
__dev_queue_xmit+0x1dce/0x3940 net/core/dev.c:4353
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:342
__dev_queue_xmit+0x17f5/0x3940 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7fc4686798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fc4697f40c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007fc46878bf60 RCX: 00007fc4686798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007fc4686d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007fc46878bf60 R15: 00007fff01466ec8
</TASK>
------------[ cut here ]------------
pvqspinlock: lock 0xffff88807a476cc0 has corrupted value 0x0!
WARNING: CPU: 1 PID: 5357 at kernel/locking/qspinlock_paravirt.h:510 __pv_queued_spin_unlock_slowpath+0x23b/0x2f0 kernel/locking/qspinlock_paravirt.h:508
Modules linked in:
CPU: 1 PID: 5357 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
RIP: 0010:__pv_queued_spin_unlock_slowpath+0x23b/0x2f0 kernel/locking/qspinlock_paravirt.h:508
Code: e8 0a 70 71 f7 4c 89 f0 48 c1 e8 03 0f b6 04 18 84 c0 0f 85 9a 00 00 00 41 8b 16 48 c7 c7 40 c6 aa 8a 4c 89 f6 e8 45 19 db f6 <0f> 0b eb 95 44 89 f1 80 e1 07 38 c1 0f 8c 2e ff ff ff 4c 89 f7 e8
RSP: 0018:ffffc900050ce398 EFLAGS: 00010246
RAX: 3a091d8f59dedc00 RBX: dffffc0000000000 RCX: ffff88807bebbb80
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff81524a02 R09: 1ffff92000a19c14
R10: dffffc0000000000 R11: fffff52000a19c15 R12: 1ffff1100f48ed9a
R13: ffff88807a476cd0 R14: ffff88807a476cc0 R15: ffff88807a476cc0
FS: 00007fc4697f46c0(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020004540 CR3: 000000001e0b6000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
__raw_callee_save___pv_queued_spin_unlock_slowpath+0x15/0x30
.slowpath+0x9/0x1a
do_raw_spin_unlock+0x13b/0x8b0 arch/x86/include/asm/paravirt.h:591
_raw_spin_unlock+0x1e/0x40 include/linux/spinlock_api_smp.h:142
spin_unlock include/linux/spinlock.h:391 [inline]
__netif_tx_unlock include/linux/netdevice.h:4441 [inline]
__dev_queue_xmit+0x1dce/0x3940 net/core/dev.c:4353
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:342
__dev_queue_xmit+0x17f5/0x3940 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7fc4686798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fc4697f40c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007fc46878bf60 RCX: 00007fc4686798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007fc4686d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007fc46878bf60 R15: 00007fff01466ec8
</TASK>
Tested on:
commit: 0f5cc96c Merge tag 's390-6.7-3' of git://git.kernel.or..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=128b55af680000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=16d18da4e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231125110148.694-1-hdanton@sina.com>
@ 2023-11-25 11:39 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-25 11:39 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0 Not tainted
--------------------------------------------
syz-executor.0/5352 is trying to acquire lock:
ffff88806adeb8d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88806adeb8d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff88806adeb8d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: __dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4344
but task is already holding lock:
ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:340
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&queue->_xmit_lock);
lock(&queue->_xmit_lock);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5352:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4273
#2: ffff88807f8e0258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff88807f8e0258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff88807f8e0258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3761 [inline]
#2: ffff88807f8e0258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d0/0x38e0 net/core/dev.c:4314
#3: ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff8880762430d8 (&queue->_xmit_lock){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:340
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4273
stack backtrace:
CPU: 1 PID: 5352 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6a81/0x7f70 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4344
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3547 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3563
sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:342
__dev_queue_xmit+0x187c/0x38e0 net/core/dev.c:3774
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f25aa8798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f25a9ffe0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f25aa98bf60 RCX: 00007f25aa8798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f25aa8d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f25aa98bf60 R15: 00007fff36da4898
</TASK>
Tested on:
commit: 0f5cc96c Merge tag 's390-6.7-3' of git://git.kernel.or..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=117ea4d8e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=139c7daf680000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231125071138.1665-1-hdanton@sina.com>
@ 2023-11-25 8:04 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-25 8:04 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0 Not tainted
--------------------------------------------
syz-executor.0/5360 is trying to acquire lock:
ffff888077a4d0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff888077a4d0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff888077a4d0d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x161e/0x38d0 net/core/dev.c:4342
but task is already holding lock:
ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c1/0x5f0 net/sched/sch_generic.c:340
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(_xmit_ETHER#2);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5360:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38d0 net/core/dev.c:4271
#2: ffff88801a9d1258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff88801a9d1258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff88801a9d1258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff88801a9d1258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d2/0x38d0 net/core/dev.c:4312
#3: ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff88807adc2cd8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c1/0x5f0 net/sched/sch_generic.c:340
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38d0 net/core/dev.c:4271
stack backtrace:
CPU: 0 PID: 5360 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00195-g0f5cc96c367f-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6a81/0x7f70 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x161e/0x38d0 net/core/dev.c:4342
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2bb/0x5f0 net/sched/sch_generic.c:342
__dev_queue_xmit+0x187e/0x38d0 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f668a8798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f668ba420c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f668a98bf60 RCX: 00007f668a8798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f668a8d5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f668a98bf60 R15: 00007ffed808a478
</TASK>
Tested on:
commit: 0f5cc96c Merge tag 's390-6.7-3' of git://git.kernel.or..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=1183cbcce80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=10d71c94e80000
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
[not found] <20231125044045.1597-1-hdanton@sina.com>
@ 2023-11-25 4:55 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-25 4:55 UTC (permalink / raw)
To: hdanton, linux-kernel, syzkaller-bugs
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __dev_queue_xmit
============================================
WARNING: possible recursive locking detected
6.7.0-rc2-syzkaller-00195-g0f5cc96c367f #0 Not tainted
--------------------------------------------
syz-executor.0/5356 is trying to acquire lock:
ffff888074e0e8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff888074e0e8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff888074e0e8d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
but task is already holding lock:
ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:340
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(_xmit_ETHER#2);
lock(_xmit_ETHER#2);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by syz-executor.0/5356:
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#0: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#1: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
#2: ffff88801c335258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline]
#2: ffff88801c335258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: qdisc_run_begin include/net/sch_generic.h:194 [inline]
#2: ffff88801c335258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3759 [inline]
#2: ffff88801c335258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x11d0/0x38e0 net/core/dev.c:4312
#3: ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline]
#3: ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4403 [inline]
#3: ffff888075f1c4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:340
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:301 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:747 [inline]
#4: ffffffff8cb25ba0 (rcu_read_lock){....}-{1:2}, at: ip_finish_output2+0x467/0x1360 net/ipv4/ip_output.c:228
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:799 [inline]
#5: ffffffff8cb25c00 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x23e/0x38e0 net/core/dev.c:4271
stack backtrace:
CPU: 0 PID: 5356 Comm: syz-executor.0 Not tainted 6.7.0-rc2-syzkaller-00195-g0f5cc96c367f #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/10/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
__lock_acquire+0x6a81/0x7f70 kernel/locking/lockdep.c:3062
lock_acquire+0x1e3/0x520 kernel/locking/lockdep.c:5753
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:351 [inline]
__netif_tx_lock include/linux/netdevice.h:4403 [inline]
__dev_queue_xmit+0x1622/0x38e0 net/core/dev.c:4342
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
iptunnel_xmit+0x540/0x9b0 net/ipv4/ip_tunnel_core.c:82
ip_tunnel_xmit+0x20e4/0x2940 net/ipv4/ip_tunnel.c:831
erspan_xmit+0x9c6/0x13e0 net/ipv4/ip_gre.c:717
__netdev_start_xmit include/linux/netdevice.h:4940 [inline]
netdev_start_xmit include/linux/netdevice.h:4954 [inline]
xmit_one net/core/dev.c:3545 [inline]
dev_hard_start_xmit+0x241/0x750 net/core/dev.c:3561
sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:342
__dev_queue_xmit+0x187c/0x38e0 net/core/dev.c:3772
ip_finish_output2+0xe6d/0x1360 include/net/neighbour.h:542
ip_send_skb+0x117/0x1b0 include/net/dst.h:451
udp_send_skb+0x931/0x1200 net/ipv4/udp.c:963
udp_sendmsg+0x1c17/0x2a70 net/ipv4/udp.c:1250
udpv6_sendmsg+0x1342/0x3220 net/ipv6/udp.c:1390
____sys_sendmsg+0x592/0x890 net/socket.c:730
__sys_sendmmsg+0x3b2/0x730 net/socket.c:2638
__do_sys_sendmmsg net/socket.c:2753 [inline]
__se_sys_sendmmsg net/socket.c:2750 [inline]
__x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2750
do_syscall_x64 arch/x86/entry/common.c:51 [inline]
do_syscall_64+0x44/0x110 arch/x86/entry/common.c:82
entry_SYSCALL_64_after_hwframe+0x63/0x6b
RIP: 0033:0x7f7c7ca798a9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f7c7dc9f0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f7c7cb8bf60 RCX: 00007f7c7ca798a9
RDX: 0000000000000001 RSI: 0000000020004d80 RDI: 0000000000000004
RBP: 00007f7c7cad5074 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f7c7cb8bf60 R15: 00007ffc2753f048
</TASK>
Tested on:
commit: 0f5cc96c Merge tag 's390-6.7-3' of git://git.kernel.or..
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
console output: https://syzkaller.appspot.com/x/log.txt?x=15e071c8e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=3813bb4934ffb745
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
Note: no patches were applied.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [syzbot] [net?] possible deadlock in sch_direct_xmit (2)
2020-04-29 0:59 syzbot
@ 2023-11-24 0:38 ` syzbot
0 siblings, 0 replies; 10+ messages in thread
From: syzbot @ 2023-11-24 0:38 UTC (permalink / raw)
To: administracion, ap420073, davem, edumazet, hdanton, jhs, jiri,
kuba, linux-kernel, netdev, pabeni, syzkaller-bugs,
xiyou.wangcong
syzbot has bisected this issue to:
commit 1a33e10e4a95cb109ff1145098175df3113313ef
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Sun May 3 05:22:19 2020 +0000
net: partially revert dynamic lockdep key changes
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=17cd55af680000
start commit: feb9c5e19e91 Merge tag 'for_linus' of git://git.kernel.org..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=142d55af680000
console output: https://syzkaller.appspot.com/x/log.txt?x=102d55af680000
kernel config: https://syzkaller.appspot.com/x/.config?x=78013caa620443d6
dashboard link: https://syzkaller.appspot.com/bug?extid=e18ac85757292b7baf96
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14430eb9f00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13738f71f00000
Reported-by: syzbot+e18ac85757292b7baf96@syzkaller.appspotmail.com
Fixes: 1a33e10e4a95 ("net: partially revert dynamic lockdep key changes")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-11-27 12:25 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20231126030321.950-1-hdanton@sina.com>
2023-11-26 3:24 ` [syzbot] [net?] possible deadlock in sch_direct_xmit (2) syzbot
[not found] <tencent_F694D4E91AEE12CC2C7B566C7C2F7D6ECC0A@qq.com>
2023-11-27 12:25 ` syzbot
[not found] <tencent_955C09A52EC46EC24C9327E746D852CD4606@qq.com>
2023-11-26 10:33 ` syzbot
[not found] <tencent_94065D4991EECA6EDE4C8AE7C446C512F906@qq.com>
2023-11-26 7:10 ` syzbot
[not found] <20231126011259.821-1-hdanton@sina.com>
2023-11-26 1:33 ` syzbot
[not found] <20231125130757.765-1-hdanton@sina.com>
2023-11-25 13:22 ` syzbot
[not found] <20231125110148.694-1-hdanton@sina.com>
2023-11-25 11:39 ` syzbot
[not found] <20231125071138.1665-1-hdanton@sina.com>
2023-11-25 8:04 ` syzbot
[not found] <20231125044045.1597-1-hdanton@sina.com>
2023-11-25 4:55 ` syzbot
2020-04-29 0:59 syzbot
2023-11-24 0:38 ` [syzbot] [net?] " syzbot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).