From: Pedro Tammela <pctammela@mojatatu.com>
To: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Peilin Ye <yepeilin.cs@gmail.com>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Cong Wang <xiyou.wangcong@gmail.com>,
Jiri Pirko <jiri@resnulli.us>,
Peilin Ye <peilin.ye@bytedance.com>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Hillf Danton <hdanton@sina.com>,
netdev@vger.kernel.org, Cong Wang <cong.wang@bytedance.com>,
Vlad Buslov <vladbu@nvidia.com>
Subject: Re: [PATCH v5 net 6/6] net/sched: qdisc_destroy() old ingress and clsact Qdiscs before grafting
Date: Fri, 26 May 2023 17:21:34 -0300 [thread overview]
Message-ID: <7879f218-c712-e9cc-57ba-665990f5f4c9@mojatatu.com> (raw)
In-Reply-To: <CAM0EoM=FS2arxv0__aQXF1a7ViJnM0hST=TL9dcnJpkf-ipjvA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]
On 26/05/2023 16:47, Jamal Hadi Salim wrote:
> [...] Peilin, Pedro will post the new repro.
Hi!
We tweaked the reproducer to:
---
r0 = socket$netlink(0x10, 0x3, 0x0)
r1 = socket(0x10, 0x803, 0x0)
sendmsg$nl_route_sched(r1, &(0x7f0000000300)={0x0, 0x0,
&(0x7f0000000240)={&(0x7f0000000380)=ANY=[], 0x24}}, 0x0)
getsockname$packet(r1, &(0x7f0000000200)={0x11, 0x0, <r2=>0x0, 0x1, 0x0,
0x6, @broadcast}, &(0x7f0000000440)=0x14)
sendmsg$nl_route(r0, &(0x7f0000000040)={0x0, 0x0,
&(0x7f0000000000)={&(0x7f0000000080)=ANY=[@ANYBLOB="480000001000050700"/20,
@ANYRES32=r2, @ANYBLOB="0000000000000000280012000900010076657468"],
0x48}}, 0x0)
sendmsg$nl_route_sched(0xffffffffffffffff, &(0x7f00000002c0)={0x0, 0x0,
&(0x7f0000000280)={&(0x7f0000000540)=@newqdisc={0x30, 0x24, 0xf0b, 0x0,
0x0, {0x0, 0x0, 0x0, r2, {}, {0xfff1, 0xffff}},
[@qdisc_kind_options=@q_ingress={0xc}]}, 0x30}}, 0x0)
sendmsg$nl_route_sched(0xffffffffffffffff, &(0x7f0000000340)={0x0, 0x0,
&(0x7f00000000c0)={&(0x7f0000000580)=@newtfilter={0x3c, 0x2c, 0xd27,
0x0, 0x0, {0x0, 0x0, 0x0, r2, {}, {0xfff1, 0xffff}, {0xc}},
[@filter_kind_options=@f_flower={{0xb}, {0xc, 0x2,
[@TCA_FLOWER_CLASSID={0x8}]}}]}, 0x3c}}, 0x0)
r4 = socket$netlink(0x10, 0x3, 0x0)
sendmmsg(r4, &(0x7f00000002c0), 0x40000000000009f, 0x0)
r5 = socket$netlink(0x10, 0x3, 0x0)
sendmmsg(r5, &(0x7f00000002c0), 0x40000000000009f, 0x0)
---
We then generate the C program with:
syz-prog2c -sandbox none -enable net_dev -threaded -repeat 0 -prog
peilin.syz > repro.c
Now here comes a very important detail. The above will create a new net
namespace to shoot the netlink messages. We are only able to reproduce
the deadlock with your patches if we comment the creation of the new
namespace out:
---
diff --git a/repro.c b/repro.c
index ee8eb0726..5cdbfb289 100644
--- a/repro.c
+++ b/repro.c
@@ -1121,9 +1121,8 @@ static int do_sandbox_none(void)
sandbox_common();
drop_caps();
initialize_netdevices_init();
- if (unshare(CLONE_NEWNET)) {
- }
+ // Doesn't seem to deadlock in a new netns
+ // if (unshare(CLONE_NEWNET)) {
+ // }
write_file("/proc/sys/net/ipv4/ping_group_range", "0 65535");
initialize_netdevices();
setup_binderfs();
---
The reason we did this was to check on the event with 'tc mon'.
The splat is quite big, see attached. It has all the indications of a
deadlock in the rtnl_lock.
Thanks,
Pedro
[-- Attachment #2: deadlock-splat --]
[-- Type: text/plain, Size: 27730 bytes --]
[ 291.882988][ T64] INFO: task kworker/1:2:4513 blocked for more than 143 seconds.
[ 291.885716][ T64] Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 291.887658][ T64] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 291.890484][ T64] task:kworker/1:2 state:D stack:28272 pid:4513 ppid:2 flags:0x00004000
[ 291.893628][ T64] Workqueue: events_power_efficient reg_check_chans_work
[ 291.896167][ T64] Call Trace:
[ 291.897406][ T64] <TASK>
[ 291.898476][ T64] __schedule+0xecb/0x59d0
[ 291.900101][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.902218][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.904563][ T64] ? io_schedule_timeout+0x150/0x150
[ 291.906437][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 291.908241][ T64] ? _raw_spin_unlock_irq+0x23/0x50
[ 291.909564][ T64] ? lockdep_hardirqs_on+0x7d/0x100
[ 291.910912][ T64] schedule+0xe7/0x1b0
[ 291.912060][ T64] schedule_preempt_disabled+0x13/0x20
[ 291.913608][ T64] __mutex_lock+0x907/0x12d0
[ 291.914850][ T64] ? reg_check_chans_work+0x7d/0xfe0
[ 291.916249][ T64] ? mutex_lock_io_nested+0x1120/0x1120
[ 291.917689][ T64] ? reg_check_chans_work+0x7d/0xfe0
[ 291.919051][ T64] ? rtnl_lock+0x9/0x20
[ 291.920164][ T64] reg_check_chans_work+0x7d/0xfe0
[ 291.921451][ T64] ? lock_sync+0x190/0x190
[ 291.922567][ T64] ? freq_reg_info+0x1d0/0x1d0
[ 291.923802][ T64] ? spin_bug+0x1d0/0x1d0
[ 291.924910][ T64] process_one_work+0x9f9/0x15f0
[ 291.926155][ T64] ? pwq_dec_nr_in_flight+0x2a0/0x2a0
[ 291.927544][ T64] ? spin_bug+0x1d0/0x1d0
[ 291.928706][ T64] worker_thread+0x687/0x1110
[ 291.929867][ T64] ? process_one_work+0x15f0/0x15f0
[ 291.930968][ T64] kthread+0x334/0x430
[ 291.931869][ T64] ? kthread_complete_and_exit+0x40/0x40
[ 291.933092][ T64] ret_from_fork+0x1f/0x30
[ 291.934037][ T64] </TASK>
[ 291.934675][ T64] INFO: task kworker/3:2:6525 blocked for more than 143 seconds.
[ 291.936271][ T64] Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 291.937672][ T64] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 291.939514][ T64] task:kworker/3:2 state:D stack:28272 pid:6525 ppid:2 flags:0x00004000
[ 291.941718][ T64] Workqueue: events linkwatch_event
[ 291.943042][ T64] Call Trace:
[ 291.943834][ T64] <TASK>
[ 291.944526][ T64] __schedule+0xecb/0x59d0
[ 291.945582][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.946952][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.948389][ T64] ? io_schedule_timeout+0x150/0x150
[ 291.949615][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 291.950847][ T64] ? _raw_spin_unlock_irq+0x23/0x50
[ 291.952077][ T64] ? lockdep_hardirqs_on+0x7d/0x100
[ 291.953344][ T64] schedule+0xe7/0x1b0
[ 291.954286][ T64] schedule_preempt_disabled+0x13/0x20
[ 291.955512][ T64] __mutex_lock+0x907/0x12d0
[ 291.956603][ T64] ? linkwatch_event+0xf/0x70
[ 291.957680][ T64] ? mutex_lock_io_nested+0x1120/0x1120
[ 291.958953][ T64] ? spin_bug+0x1d0/0x1d0
[ 291.959996][ T64] ? linkwatch_event+0xf/0x70
[ 291.960738][ T64] ? rtnl_lock+0x9/0x20
[ 291.961249][ T64] linkwatch_event+0xf/0x70
[ 291.961770][ T64] process_one_work+0x9f9/0x15f0
[ 291.962306][ T64] ? pwq_dec_nr_in_flight+0x2a0/0x2a0
[ 291.962928][ T64] ? spin_bug+0x1d0/0x1d0
[ 291.963409][ T64] worker_thread+0x687/0x1110
[ 291.963982][ T64] ? __kthread_parkme+0x152/0x220
[ 291.964567][ T64] ? process_one_work+0x15f0/0x15f0
[ 291.965163][ T64] kthread+0x334/0x430
[ 291.965641][ T64] ? kthread_complete_and_exit+0x40/0x40
[ 291.966303][ T64] ret_from_fork+0x1f/0x30
[ 291.966836][ T64] </TASK>
[ 291.967189][ T64] INFO: task systemd-udevd:6764 blocked for more than 143 seconds.
[ 291.968062][ T64] Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 291.968813][ T64] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 291.969691][ T64] task:systemd-udevd state:D stack:25968 pid:6764 ppid:4544 flags:0x00000006
[ 291.970609][ T64] Call Trace:
[ 291.970939][ T64] <TASK>
[ 291.971239][ T64] __schedule+0xecb/0x59d0
[ 291.971718][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.972341][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 291.972965][ T64] ? io_schedule_timeout+0x150/0x150
[ 291.973572][ T64] ? __mutex_lock+0x902/0x12d0
[ 291.974120][ T64] schedule+0xe7/0x1b0
[ 291.974579][ T64] schedule_preempt_disabled+0x13/0x20
[ 291.975174][ T64] __mutex_lock+0x907/0x12d0
[ 291.975702][ T64] ? rtnetlink_rcv_msg+0x3e2/0xd30
[ 291.976299][ T64] ? mutex_lock_io_nested+0x1120/0x1120
[ 291.976938][ T64] ? rtnetlink_rcv_msg+0x3e2/0xd30
[ 291.977506][ T64] rtnetlink_rcv_msg+0x3e2/0xd30
[ 291.978042][ T64] ? rtnl_getlink+0xb10/0xb10
[ 291.978573][ T64] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 291.979295][ T64] ? __module_address+0x55/0x3b0
[ 291.979937][ T64] ? netdev_core_pick_tx+0x390/0x390
[ 291.980580][ T64] netlink_rcv_skb+0x166/0x440
[ 291.981164][ T64] ? rtnl_getlink+0xb10/0xb10
[ 291.981726][ T64] ? netlink_ack+0x1370/0x1370
[ 291.982310][ T64] ? kasan_set_track+0x25/0x30
[ 291.982973][ T64] ? netlink_deliver_tap+0x1b1/0xd00
[ 291.983579][ T64] netlink_unicast+0x530/0x800
[ 291.984132][ T64] ? netlink_attachskb+0x880/0x880
[ 291.984682][ T64] ? __sanitizer_cov_trace_switch+0x54/0x90
[ 291.985330][ T64] ? __phys_addr_symbol+0x30/0x70
[ 291.985904][ T64] ? __check_object_size+0x323/0x740
[ 291.986488][ T64] netlink_sendmsg+0x90b/0xe10
[ 291.987023][ T64] ? netlink_unicast+0x800/0x800
[ 291.987584][ T64] ? bpf_lsm_socket_sendmsg+0x9/0x10
[ 291.988189][ T64] ? netlink_unicast+0x800/0x800
[ 291.988737][ T64] sock_sendmsg+0xd9/0x180
[ 291.989228][ T64] __sys_sendto+0x201/0x2f0
[ 291.989714][ T64] ? __ia32_sys_getpeername+0xb0/0xb0
[ 291.990296][ T64] ? get_random_bytes+0x20/0x20
[ 291.990833][ T64] ? rcu_is_watching+0x12/0xb0
[ 291.991368][ T64] ? __rseq_handle_notify_resume+0x5c7/0xfe0
[ 291.992062][ T64] __x64_sys_sendto+0xe0/0x1b0
[ 291.992600][ T64] ? syscall_enter_from_user_mode+0x26/0x80
[ 291.993324][ T64] do_syscall_64+0x38/0xb0
[ 291.993816][ T64] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 291.994436][ T64] RIP: 0033:0x7f1de823fcf7
[ 291.994902][ T64] RSP: 002b:00007ffd8c5d3208 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
[ 291.995834][ T64] RAX: ffffffffffffffda RBX: 000055d9974d14e0 RCX: 00007f1de823fcf7
[ 291.996711][ T64] RDX: 0000000000000020 RSI: 000055d997332930 RDI: 0000000000000005
[ 291.997593][ T64] RBP: 00007ffd8c5d33b8 R08: 00007ffd8c5d3280 R09: 0000000000000080
[ 291.998477][ T64] R10: 0000000000000000 R11: 0000000000000202 R12: 00000000043abe07
[ 291.999360][ T64] R13: 0000000000000001 R14: 000055d9974845a0 R15: 0000000000000000
[ 292.000254][ T64] </TASK>
[ 292.000609][ T64] INFO: task systemd-udevd:6773 blocked for more than 143 seconds.
[ 292.001482][ T64] Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 292.002233][ T64] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 292.003237][ T64] task:systemd-udevd state:D stack:25968 pid:6773 ppid:4544 flags:0x00004006
[ 292.004292][ T64] Call Trace:
[ 292.004743][ T64] <TASK>
[ 292.005128][ T64] __schedule+0xecb/0x59d0
[ 292.005637][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.006304][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.007005][ T64] ? io_schedule_timeout+0x150/0x150
[ 292.007601][ T64] ? __mutex_lock+0x902/0x12d0
[ 292.008174][ T64] schedule+0xe7/0x1b0
[ 292.008648][ T64] schedule_preempt_disabled+0x13/0x20
[ 292.009257][ T64] __mutex_lock+0x907/0x12d0
[ 292.009793][ T64] ? dev_ethtool+0x21d/0x53e0
[ 292.010374][ T64] ? mutex_lock_io_nested+0x1120/0x1120
[ 292.011105][ T64] ? kasan_unpoison+0x44/0x70
[ 292.011987][ T64] ? __kmem_cache_alloc_node+0x19e/0x330
[ 292.012994][ T64] ? dev_ethtool+0x170/0x53e0
[ 292.013757][ T64] ? dev_ethtool+0x21d/0x53e0
[ 292.014470][ T64] ? rtnl_lock+0x9/0x20
[ 292.015032][ T64] dev_ethtool+0x21d/0x53e0
[ 292.015633][ T64] ? filter_irq_stacks+0x90/0x90
[ 292.016269][ T64] ? print_usage_bug.part.0+0x670/0x670
[ 292.016941][ T64] ? ethtool_get_module_info_call+0x1c0/0x1c0
[ 292.017674][ T64] ? kasan_set_track+0x25/0x30
[ 292.018228][ T64] ? ____kasan_slab_free+0x15e/0x1b0
[ 292.018794][ T64] ? slab_free_freelist_hook+0x10b/0x1e0
[ 292.019287][ T64] ? __kmem_cache_free+0xaf/0x2e0
[ 292.019729][ T64] ? tomoyo_path_number_perm+0x43a/0x530
[ 292.020250][ T64] ? security_file_ioctl+0x72/0xb0
[ 292.020700][ T64] ? __x64_sys_ioctl+0xbb/0x210
[ 292.021128][ T64] ? do_syscall_64+0x38/0xb0
[ 292.021538][ T64] ? mark_lock+0x105/0x1960
[ 292.021950][ T64] ? __lock_acquire+0xc42/0x5cf0
[ 292.022399][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.022990][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.023514][ T64] ? lockdep_hardirqs_on+0x7d/0x100
[ 292.023995][ T64] ? __sanitizer_cov_trace_switch+0x54/0x90
[ 292.024513][ T64] ? find_held_lock+0x2d/0x110
[ 292.024940][ T64] ? dev_load+0x7d/0x240
[ 292.025322][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.025801][ T64] ? full_name_hash+0xbc/0x110
[ 292.026220][ T64] dev_ioctl+0x29e/0x1090
[ 292.026601][ T64] sock_do_ioctl+0x1be/0x250
[ 292.027006][ T64] ? get_user_ifreq+0x250/0x250
[ 292.027433][ T64] ? do_vfs_ioctl+0x34a/0x17f0
[ 292.027870][ T64] ? vfs_fileattr_set+0xbf0/0xbf0
[ 292.028318][ T64] sock_ioctl+0x205/0x6a0
[ 292.028697][ T64] ? br_ioctl_call+0xb0/0xb0
[ 292.029145][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.029827][ T64] ? handle_mm_fault+0x32c/0x9e0
[ 292.030501][ T64] ? bpf_lsm_file_ioctl+0x9/0x10
[ 292.031054][ T64] ? br_ioctl_call+0xb0/0xb0
[ 292.031527][ T64] __x64_sys_ioctl+0x189/0x210
[ 292.032091][ T64] do_syscall_64+0x38/0xb0
[ 292.032609][ T64] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 292.033390][ T64] RIP: 0033:0x7f1de82319ef
[ 292.033908][ T64] RSP: 002b:00007ffd8c5d32f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[ 292.034751][ T64] RAX: ffffffffffffffda RBX: 000055d997476d20 RCX: 00007f1de82319ef
[ 292.035448][ T64] RDX: 00007ffd8c5d3430 RSI: 0000000000008946 RDI: 000000000000000b
[ 292.036179][ T64] RBP: 000055d997488a50 R08: 0000000000000000 R09: 0000000000000006
[ 292.036884][ T64] R10: 00007ffd8c5d3400 R11: 0000000000000246 R12: 000055d99732a230
[ 292.037581][ T64] R13: 00007ffd8c5d3430 R14: 0000000000000000 R15: 000055d99732a238
[ 292.038268][ T64] </TASK>
[ 292.038557][ T64] INFO: task repro:7652 blocked for more than 143 seconds.
[ 292.039202][ T64] Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 292.039776][ T64] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 292.040509][ T64] task:repro state:D stack:25984 pid:7652 ppid:6753 flags:0x00004006
[ 292.041289][ T64] Call Trace:
[ 292.041575][ T64] <TASK>
[ 292.041848][ T64] __schedule+0xecb/0x59d0
[ 292.042297][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.043017][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.043711][ T64] ? io_schedule_timeout+0x150/0x150
[ 292.044360][ T64] ? __mutex_lock+0x902/0x12d0
[ 292.044845][ T64] schedule+0xe7/0x1b0
[ 292.045300][ T64] schedule_preempt_disabled+0x13/0x20
[ 292.045872][ T64] __mutex_lock+0x907/0x12d0
[ 292.046396][ T64] ? fl_change+0x2102/0x51e0
[ 292.046833][ T64] ? tc_setup_cb_reoffload+0x100/0x100
[ 292.047353][ T64] ? mutex_lock_io_nested+0x1120/0x1120
[ 292.047915][ T64] ? _raw_spin_unlock+0x28/0x40
[ 292.048368][ T64] ? tcf_exts_init_ex+0x1bc/0x610
[ 292.048828][ T64] ? fl_change+0x2102/0x51e0
[ 292.049328][ T64] ? rtnl_lock+0x9/0x20
[ 292.049768][ T64] fl_change+0x2102/0x51e0
[ 292.050232][ T64] ? find_held_lock+0x2d/0x110
[ 292.050681][ T64] ? __rhashtable_insert_fast.constprop.0.isra.0+0x14a0/0x14a0
[ 292.051416][ T64] ? fl_get+0x21c/0x3c0
[ 292.051791][ T64] ? fl_put+0x20/0x20
[ 292.052170][ T64] ? mini_qdisc_pair_swap+0x128/0x1f0
[ 292.052655][ T64] ? __rhashtable_insert_fast.constprop.0.isra.0+0x14a0/0x14a0
[ 292.053395][ T64] tc_new_tfilter+0x992/0x22b0
[ 292.053858][ T64] ? tc_del_tfilter+0x1570/0x1570
[ 292.054419][ T64] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.055066][ T64] ? kasan_quarantine_put+0x102/0x230
[ 292.055660][ T64] ? rtnetlink_rcv_msg+0x94a/0xd30
[ 292.056224][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.056799][ T64] ? bpf_lsm_capable+0x9/0x10
[ 292.057311][ T64] ? tc_del_tfilter+0x1570/0x1570
[ 292.057843][ T64] rtnetlink_rcv_msg+0x98a/0xd30
[ 292.058381][ T64] ? rtnl_getlink+0xb10/0xb10
[ 292.058833][ T64] ? __x64_sys_sendmmsg+0x9c/0x100
[ 292.059305][ T64] ? do_syscall_64+0x38/0xb0
[ 292.059752][ T64] ? netdev_core_pick_tx+0x390/0x390
[ 292.060297][ T64] netlink_rcv_skb+0x166/0x440
[ 292.060733][ T64] ? rtnl_getlink+0xb10/0xb10
[ 292.061146][ T64] ? netlink_ack+0x1370/0x1370
[ 292.061566][ T64] ? kasan_set_track+0x25/0x30
[ 292.062003][ T64] ? netlink_deliver_tap+0x1b1/0xd00
[ 292.062468][ T64] netlink_unicast+0x530/0x800
[ 292.062955][ T64] ? netlink_attachskb+0x880/0x880
[ 292.063416][ T64] ? __sanitizer_cov_trace_switch+0x54/0x90
[ 292.063958][ T64] ? __phys_addr_symbol+0x30/0x70
[ 292.064468][ T64] ? __check_object_size+0x323/0x740
[ 292.065009][ T64] netlink_sendmsg+0x90b/0xe10
[ 292.065517][ T64] ? netlink_unicast+0x800/0x800
[ 292.066010][ T64] ? bpf_lsm_socket_sendmsg+0x9/0x10
[ 292.066467][ T64] ? netlink_unicast+0x800/0x800
[ 292.066933][ T64] sock_sendmsg+0xd9/0x180
[ 292.067358][ T64] ____sys_sendmsg+0x264/0x910
[ 292.067794][ T64] ? kernel_sendmsg+0x50/0x50
[ 292.068252][ T64] ? __copy_msghdr+0x460/0x460
[ 292.068690][ T64] ___sys_sendmsg+0x11d/0x1b0
[ 292.069111][ T64] ? do_recvmmsg+0x700/0x700
[ 292.069551][ T64] ? find_held_lock+0x2d/0x110
[ 292.069972][ T64] ? __might_fault+0xe5/0x190
[ 292.070397][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.070862][ T64] __sys_sendmmsg+0x18e/0x430
[ 292.071272][ T64] ? __ia32_sys_sendmsg+0xb0/0xb0
[ 292.071705][ T64] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.072204][ T64] ? rcu_is_watching+0x12/0xb0
[ 292.072620][ T64] ? xfd_validate_state+0x5d/0x180
[ 292.073099][ T64] ? restore_fpregs_from_fpstate+0xc1/0x1d0
[ 292.073611][ T64] ? unlock_page_memcg+0x2d0/0x2d0
[ 292.074062][ T64] ? do_futex+0x350/0x350
[ 292.074461][ T64] __x64_sys_sendmmsg+0x9c/0x100
[ 292.074893][ T64] ? syscall_enter_from_user_mode+0x26/0x80
[ 292.075494][ T64] do_syscall_64+0x38/0xb0
[ 292.075960][ T64] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 292.076473][ T64] RIP: 0033:0x7f3f4319289d
[ 292.076850][ T64] RSP: 002b:00007f3f43056e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
[ 292.077557][ T64] RAX: ffffffffffffffda RBX: 0000558de6727248 RCX: 00007f3f4319289d
[ 292.078234][ T64] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 0000000000000006
[ 292.078924][ T64] RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
[ 292.079586][ T64] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558de672724c
[ 292.080259][ T64] R13: 0000558de6723030 R14: 040000000000009f R15: 0000558de6727240
[ 292.080943][ T64] </TASK>
[ 292.081221][ T64]
[ 292.081221][ T64] Showing all locks held in the system:
[ 292.081868][ T64] 1 lock held by rcu_tasks_kthre/14:
[ 292.082316][ T64] #0: ffffffff8c798430 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2c/0xe20
[ 292.083266][ T64] 1 lock held by rcu_tasks_trace/15:
[ 292.083721][ T64] #0: ffffffff8c798130 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2c/0xe20
[ 292.084702][ T64] 5 locks held by ksoftirqd/4/38:
[ 292.085152][ T64] 3 locks held by kworker/4:0/39:
[ 292.085598][ T64] #0: ffff88811b78b938 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0x8e2/0x15f0
[ 292.086704][ T64] #1: ffffc9000139fda0 ((work_completion)(&(&net->ipv6.addr_chk_work)->work)){+.+.}-{0:0}, at: process_one_work+0x916/0x15f0
[ 292.088126][ T64] #2: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: addrconf_verify_work+0x12/0x30
[ 292.089159][ T64] 1 lock held by khungtaskd/64:
[ 292.089656][ T64] #0: ffffffff8c799040 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x340
[ 292.090713][ T64] 3 locks held by kworker/1:2/4513:
[ 292.091175][ T64] #0: ffff888100079d38 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: process_one_work+0x8e2/0x15f0
[ 292.092274][ T64] #1: ffffc90016fcfda0 ((reg_check_chans).work){+.+.}-{0:0}, at: process_one_work+0x916/0x15f0
[ 292.093319][ T64] #2: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: reg_check_chans_work+0x7d/0xfe0
[ 292.094269][ T64] 3 locks held by kworker/3:2/6525:
[ 292.094801][ T64] #0: ffff888100078d38 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x8e2/0x15f0
[ 292.095834][ T64] #1: ffffc9000648fda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0x916/0x15f0
[ 292.096754][ T64] #2: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: linkwatch_event+0xf/0x70
[ 292.097662][ T64] 1 lock held by systemd-udevd/6764:
[ 292.098148][ T64] #0: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x3e2/0xd30
[ 292.098984][ T64] 1 lock held by systemd-udevd/6773:
[ 292.099462][ T64] #0: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: dev_ethtool+0x21d/0x53e0
[ 292.100300][ T64] 1 lock held by repro/7652:
[ 292.100700][ T64] #0: ffffffff8e115668 (rtnl_mutex){+.+.}-{3:3}, at: fl_change+0x2102/0x51e0
[ 292.101494][ T64] 2 locks held by repro/7653:
[ 292.101905][ T64]
[ 292.102135][ T64] =============================================
[ 292.102135][ T64]
[ 292.102905][ T64] NMI backtrace for cpu 5
[ 292.103277][ T64] CPU: 5 PID: 64 Comm: khungtaskd Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 292.104054][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[ 292.104931][ T64] Call Trace:
[ 292.105222][ T64] <TASK>
[ 292.105481][ T64] dump_stack_lvl+0xd9/0x1b0
[ 292.105889][ T64] nmi_cpu_backtrace+0x277/0x380
[ 292.106318][ T64] ? lapic_can_unplug_cpu+0xa0/0xa0
[ 292.106770][ T64] nmi_trigger_cpumask_backtrace+0x2ac/0x310
[ 292.107290][ T64] watchdog+0xed1/0x1150
[ 292.107715][ T64] ? proc_dohung_task_timeout_secs+0x90/0x90
[ 292.108330][ T64] kthread+0x334/0x430
[ 292.108736][ T64] ? kthread_complete_and_exit+0x40/0x40
[ 292.109285][ T64] ret_from_fork+0x1f/0x30
[ 292.109720][ T64] </TASK>
[ 292.109991][ T64] Sending NMI from CPU 5 to CPUs 0-4,6-7:
[ 292.110569][ C3] NMI backtrace for cpu 3
[ 292.110577][ C3] CPU: 3 PID: 7653 Comm: repro Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 292.110586][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[ 292.110591][ C3] RIP: 0010:unwind_get_return_address+0x6e/0xa0
[ 292.110604][ C3] Code: ea 03 80 3c 02 00 75 32 48 8b 7b 48 e8 3b 46 1c 00 85 c0 74 d3 48 b8 00 00 00 00 00 fc ff df 48 89 ea 48 c1 ea 03 80 3c 02 00 <75> 18 48 8b 43 48 5b 5d c3 e8 14 c5 9e 00 eb a8 48 89 ef e8 2a c5
[ 292.110611][ C3] RSP: 0018:ffffc900095d6ed0 EFLAGS: 00000246
[ 292.110618][ C3] RAX: dffffc0000000000 RBX: ffffc900095d6ee8 RCX: 0000000000000000
[ 292.110623][ C3] RDX: 1ffff920012bade6 RSI: ffffc900095d72a0 RDI: ffffffff88370cd5
[ 292.110628][ C3] RBP: ffffc900095d6f30 R08: ffffc900095d6f1c R09: ffffffff8f831a34
[ 292.110633][ C3] R10: ffffc900095d6ee8 R11: 0000000000072d0d R12: ffffc900095d6fa0
[ 292.110637][ C3] R13: 0000000000000000 R14: ffff888173271dc0 R15: ffff888171b140a0
[ 292.110645][ C3] FS: 00007f3f430376c0(0000) GS:ffff888437c80000(0000) knlGS:0000000000000000
[ 292.110653][ C3] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 292.110658][ C3] CR2: 0000558de83422d8 CR3: 0000000116ff7000 CR4: 0000000000350ee0
[ 292.110664][ C3] Call Trace:
[ 292.110667][ C3] <TASK>
[ 292.110670][ C3] ? write_profile+0x440/0x440
[ 292.110684][ C3] arch_stack_walk+0x97/0xf0
[ 292.110697][ C3] ? ingress_init+0x165/0x220
[ 292.110706][ C3] stack_trace_save+0x96/0xd0
[ 292.110716][ C3] ? filter_irq_stacks+0x90/0x90
[ 292.110727][ C3] kasan_save_stack+0x20/0x40
[ 292.110741][ C3] ? kasan_save_stack+0x20/0x40
[ 292.110753][ C3] ? kasan_set_track+0x25/0x30
[ 292.110766][ C3] ? __kasan_kmalloc+0xa2/0xb0
[ 292.110776][ C3] ? tcf_block_get_ext+0x14f/0x1900
[ 292.110785][ C3] ? ingress_init+0x165/0x220
[ 292.110792][ C3] ? qdisc_create+0x4f7/0x1090
[ 292.110799][ C3] ? tc_modify_qdisc+0xd38/0x1b70
[ 292.110806][ C3] ? rtnetlink_rcv_msg+0x439/0xd30
[ 292.110813][ C3] ? netlink_rcv_skb+0x166/0x440
[ 292.110824][ C3] ? netlink_unicast+0x530/0x800
[ 292.110835][ C3] ? netlink_sendmsg+0x90b/0xe10
[ 292.110847][ C3] ? find_held_lock+0x2d/0x110
[ 292.110857][ C3] ? __kmem_cache_alloc_node+0x49/0x330
[ 292.110867][ C3] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.110878][ C3] ? kasan_unpoison+0x44/0x70
[ 292.110885][ C3] ? __kasan_slab_alloc+0x4e/0x90
[ 292.110896][ C3] ? __kmem_cache_alloc_node+0x19e/0x330
[ 292.110906][ C3] ? tcf_block_get_ext+0x14f/0x1900
[ 292.110915][ C3] kasan_set_track+0x25/0x30
[ 292.110925][ C3] __kasan_kmalloc+0xa2/0xb0
[ 292.110935][ C3] tcf_block_get_ext+0x14f/0x1900
[ 292.110945][ C3] ingress_init+0x165/0x220
[ 292.110952][ C3] ? clsact_init+0x330/0x330
[ 292.110959][ C3] qdisc_create+0x4f7/0x1090
[ 292.110967][ C3] ? tc_get_qdisc+0xb80/0xb80
[ 292.110975][ C3] tc_modify_qdisc+0xd38/0x1b70
[ 292.110984][ C3] ? qdisc_create+0x1090/0x1090
[ 292.110994][ C3] ? bpf_lsm_capable+0x9/0x10
[ 292.111004][ C3] ? qdisc_create+0x1090/0x1090
[ 292.111012][ C3] rtnetlink_rcv_msg+0x439/0xd30
[ 292.111019][ C3] ? rtnl_getlink+0xb10/0xb10
[ 292.111025][ C3] ? __x64_sys_sendmmsg+0x9c/0x100
[ 292.111034][ C3] ? do_syscall_64+0x38/0xb0
[ 292.111043][ C3] ? netdev_core_pick_tx+0x390/0x390
[ 292.111056][ C3] netlink_rcv_skb+0x166/0x440
[ 292.111067][ C3] ? rtnl_getlink+0xb10/0xb10
[ 292.111074][ C3] ? netlink_ack+0x1370/0x1370
[ 292.111085][ C3] ? kasan_set_track+0x25/0x30
[ 292.111097][ C3] ? netlink_deliver_tap+0x1b1/0xd00
[ 292.111109][ C3] netlink_unicast+0x530/0x800
[ 292.111121][ C3] ? netlink_attachskb+0x880/0x880
[ 292.111132][ C3] ? __sanitizer_cov_trace_switch+0x54/0x90
[ 292.111142][ C3] ? __phys_addr_symbol+0x30/0x70
[ 292.111153][ C3] ? __check_object_size+0x323/0x740
[ 292.111161][ C3] netlink_sendmsg+0x90b/0xe10
[ 292.111173][ C3] ? netlink_unicast+0x800/0x800
[ 292.111185][ C3] ? bpf_lsm_socket_sendmsg+0x9/0x10
[ 292.111193][ C3] ? netlink_unicast+0x800/0x800
[ 292.111204][ C3] sock_sendmsg+0xd9/0x180
[ 292.111216][ C3] ____sys_sendmsg+0x264/0x910
[ 292.111227][ C3] ? kernel_sendmsg+0x50/0x50
[ 292.111238][ C3] ? __copy_msghdr+0x460/0x460
[ 292.111246][ C3] ? lockdep_hardirqs_on_prepare+0x410/0x410
[ 292.111258][ C3] ___sys_sendmsg+0x11d/0x1b0
[ 292.111266][ C3] ? do_recvmmsg+0x700/0x700
[ 292.111274][ C3] ? __fget_files+0x260/0x420
[ 292.111284][ C3] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.111296][ C3] ? __fget_files+0x282/0x420
[ 292.111307][ C3] ? __fget_light+0xe6/0x270
[ 292.111318][ C3] __sys_sendmmsg+0x18e/0x430
[ 292.111327][ C3] ? __ia32_sys_sendmsg+0xb0/0xb0
[ 292.111334][ C3] ? reacquire_held_locks+0x4b0/0x4b0
[ 292.111346][ C3] ? rcu_is_watching+0x12/0xb0
[ 292.111358][ C3] ? __rseq_handle_notify_resume+0x5c7/0xfe0
[ 292.111368][ C3] ? __do_sys_rseq+0x750/0x750
[ 292.111376][ C3] ? unlock_page_memcg+0x2d0/0x2d0
[ 292.111387][ C3] ? do_futex+0x350/0x350
[ 292.111395][ C3] __x64_sys_sendmmsg+0x9c/0x100
[ 292.111403][ C3] ? syscall_enter_from_user_mode+0x26/0x80
[ 292.111414][ C3] do_syscall_64+0x38/0xb0
[ 292.111421][ C3] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 292.111433][ C3] RIP: 0033:0x7f3f4319289d
[ 292.111440][ C3] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4b 05 0e 00 f7 d8 64 89 01 48
[ 292.111447][ C3] RSP: 002b:00007f3f43035e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
[ 292.111454][ C3] RAX: ffffffffffffffda RBX: 0000558de6727258 RCX: 00007f3f4319289d
[ 292.111459][ C3] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 0000000000000007
[ 292.111463][ C3] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 292.111467][ C3] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558de672725c
[ 292.111472][ C3] R13: 0000558de6723030 R14: 040000000000009f R15: 0000558de6727250
[ 292.111479][ C3] </TASK>
[ 292.111481][ C4] NMI backtrace for cpu 4 skipped: idling at default_idle+0xf/0x20
[ 292.092844][ C7] NMI backtrace for cpu 7 skipped: idling at default_idle+0xf/0x20
[ 292.092832][ C2] NMI backtrace for cpu 2 skipped: idling at default_idle+0xf/0x20
[ 292.092832][ C2] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.014 msecs
[ 292.092820][ C1] NMI backtrace for cpu 1 skipped: idling at default_idle+0xf/0x20
[ 292.092820][ C1] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.033 msecs
[ 292.102824][ C6] NMI backtrace for cpu 6 skipped: idling at default_idle+0xf/0x20
[ 292.102824][ C6] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.054 msecs
[ 292.102812][ C0] NMI backtrace for cpu 0 skipped: idling at default_idle+0xf/0x20
[ 292.102812][ C0] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.092 msecs
[ 292.112559][ T64] Kernel panic - not syncing: hung_task: blocked tasks
[ 292.112565][ T64] CPU: 5 PID: 64 Comm: khungtaskd Not tainted 6.4.0-rc2-00215-g6078d01d2926 #13
[ 292.112575][ T64] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[ 292.112582][ T64] Call Trace:
[ 292.112586][ T64] <TASK>
[ 292.112590][ T64] dump_stack_lvl+0xd9/0x1b0
[ 292.112603][ T64] panic+0x689/0x730
[ 292.112615][ T64] ? panic_smp_self_stop+0xa0/0xa0
[ 292.112631][ T64] ? __irq_work_queue_local+0x132/0x3f0
[ 292.112641][ T64] ? irq_work_queue+0x2a/0x70
[ 292.112649][ T64] ? __wake_up_klogd.part.0+0x99/0xf0
[ 292.112663][ T64] ? watchdog+0xc89/0x1150
[ 292.112678][ T64] watchdog+0xc9a/0x1150
[ 292.112698][ T64] ? proc_dohung_task_timeout_secs+0x90/0x90
[ 292.112713][ T64] kthread+0x334/0x430
[ 292.112723][ T64] ? kthread_complete_and_exit+0x40/0x40
[ 292.112734][ T64] ret_from_fork+0x1f/0x30
[ 292.112750][ T64] </TASK>
[ 292.112866][ T64] Kernel Offset: disabled
[ 292.112866][ T64] Rebooting in 86400 seconds..
next prev parent reply other threads:[~2023-05-26 20:21 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-24 1:16 [PATCH v5 net 0/6] net/sched: Fixes for sch_ingress and sch_clsact Peilin Ye
2023-05-24 1:17 ` [PATCH v5 net 1/6] net/sched: sch_ingress: Only create under TC_H_INGRESS Peilin Ye
2023-05-24 15:37 ` Pedro Tammela
2023-05-24 15:57 ` Jamal Hadi Salim
2023-05-24 1:18 ` [PATCH v5 net 2/6] net/sched: sch_clsact: Only create under TC_H_CLSACT Peilin Ye
2023-05-24 15:38 ` Pedro Tammela
2023-05-24 15:58 ` Jamal Hadi Salim
2023-05-24 1:19 ` [PATCH v5 net 3/6] net/sched: Reserve TC_H_INGRESS (TC_H_CLSACT) for ingress (clsact) Qdiscs Peilin Ye
2023-05-24 15:38 ` Pedro Tammela
2023-05-24 1:19 ` [PATCH v5 net 4/6] net/sched: Prohibit regrafting ingress or clsact Qdiscs Peilin Ye
2023-05-24 15:38 ` Pedro Tammela
2023-05-24 1:20 ` [PATCH v5 net 5/6] net/sched: Refactor qdisc_graft() for ingress and " Peilin Ye
2023-05-24 1:20 ` [PATCH v5 net 6/6] net/sched: qdisc_destroy() old ingress and clsact Qdiscs before grafting Peilin Ye
2023-05-24 15:39 ` Pedro Tammela
2023-05-24 16:09 ` Jamal Hadi Salim
2023-05-25 9:25 ` Paolo Abeni
2023-05-26 12:19 ` Jamal Hadi Salim
2023-05-26 12:20 ` Jamal Hadi Salim
2023-05-26 19:47 ` Jamal Hadi Salim
2023-05-26 20:21 ` Pedro Tammela [this message]
2023-05-26 23:09 ` Peilin Ye
2023-05-27 2:33 ` Jakub Kicinski
2023-05-27 8:23 ` Peilin Ye
2023-05-28 18:54 ` Jamal Hadi Salim
2023-05-29 11:50 ` Vlad Buslov
2023-05-29 12:58 ` Vlad Buslov
2023-05-30 1:03 ` Jakub Kicinski
2023-05-30 9:11 ` Peilin Ye
2023-05-30 12:18 ` Vlad Buslov
2023-05-31 0:29 ` Peilin Ye
2023-06-01 3:57 ` Peilin Ye
2023-06-01 6:20 ` Vlad Buslov
2023-06-07 0:57 ` Peilin Ye
2023-06-07 8:18 ` Vlad Buslov
2023-06-08 1:08 ` Peilin Ye
2023-06-08 7:48 ` Vlad Buslov
2023-06-11 3:25 ` Peilin Ye
2023-06-08 0:39 ` Peilin Ye
2023-06-08 9:17 ` Vlad Buslov
2023-06-10 0:20 ` Peilin Ye
2023-06-01 13:03 ` Pedro Tammela
2023-06-07 4:25 ` Peilin Ye
2023-05-29 13:55 ` Jamal Hadi Salim
2023-05-29 19:14 ` Peilin Ye
2023-05-25 17:16 ` [PATCH v5 net 0/6] net/sched: Fixes for sch_ingress and sch_clsact Vlad Buslov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7879f218-c712-e9cc-57ba-665990f5f4c9@mojatatu.com \
--to=pctammela@mojatatu.com \
--cc=cong.wang@bytedance.com \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hdanton@sina.com \
--cc=jhs@mojatatu.com \
--cc=jiri@resnulli.us \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=peilin.ye@bytedance.com \
--cc=vladbu@nvidia.com \
--cc=xiyou.wangcong@gmail.com \
--cc=yepeilin.cs@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.