From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Hemminger Subject: [RFC] lockdep warning in gen_kill_estimator Date: Wed, 8 Sep 2010 11:27:19 -0700 Message-ID: <20100908112719.3f4b4644@nehalam> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: netdev@vger.kernel.org To: David Miller , Eric Dumazet Return-path: Received: from mail.vyatta.com ([76.74.103.46]:50541 "EHLO mail.vyatta.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752673Ab0IHS11 (ORCPT ); Wed, 8 Sep 2010 14:27:27 -0400 Sender: netdev-owner@vger.kernel.org List-ID: Saw big lockdep SOFTIRQ-unsafe lock warning in gen_kill_estimator when doing. tc qdisc del dev dummy0 root Looks like this is new problem in 2.6.35 with the introduction of est_tree_lock. Here is a patch that moves the bh disable to outside the loop. Signed-off-by: Stephen Hemminger --- [12075.878153] ====================================================== [12075.878157] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ] [12075.878161] 2.6.36-rc3-next+ #30 [12075.878163] ------------------------------------------------------ [12075.878166] tc/17952 [HC0[0]:SC0[1]:HE1:SE0] is trying to acquire: [12075.878169] (est_tree_lock){+.+...}, at: [] gen_estimator_active+0x22/0xb0 [12075.878180] [12075.878181] and this task is already holding: [12075.878184] (&(&list->lock)->rlock#5){+.-.-.}, at: [] gnet_stats_start_copy_compat+0x44/0xb0 [12075.878192] which would create a new lock dependency: [12075.878194] (&(&list->lock)->rlock#5){+.-.-.} -> (est_tree_lock){+.+...} [12075.878202] [12075.878203] but this new dependency connects a SOFTIRQ-irq-safe lock: [12075.878205] (&(&list->lock)->rlock#5){+.-.-.} [12075.878209] ... which became SOFTIRQ-irq-safe at: [12075.878211] [] __lock_acquire+0x58f/0x1490 [12075.878218] [] lock_acquire+0x93/0x130 [12075.878222] [] _raw_spin_lock+0x31/0x70 [12075.878227] [] dev_queue_xmit+0xcc/0x6b0 [12075.878232] [] neigh_resolve_output+0x10e/0x3f0 [12075.878238] [] ip6_finish_output2+0x107/0x3d0 [ipv6] [12075.878252] [] ip6_finish_output+0x564/0xe50 [ipv6] [12075.878261] [] ip6_output+0x4b/0x190 [ipv6] [12075.878270] [] dst_output+0x1e/0x70 [ipv6] [12075.878283] [] mld_sendpack+0x358/0x390 [ipv6] [12075.878297] [] mld_ifc_timer_expire+0x283/0x310 [ipv6] [12075.878310] [] run_timer_softirq+0x1d7/0x400 [12075.878315] [] __do_softirq+0xc2/0x250 [12075.878320] [] call_softirq+0x1c/0x50 [12075.878325] [] do_softirq+0x75/0xb0 [12075.878329] [] irq_exit+0x8d/0xa0 [12075.878333] [] do_IRQ+0x70/0xf0 [12075.878337] [] ret_from_intr+0x0/0xf [12075.878342] [] avc_has_perm_noaudit+0x14f/0x4e0 [12075.878348] [] avc_has_perm+0x46/0x90 [12075.878353] [] inode_has_perm+0x4f/0x90 [12075.878357] [] file_has_perm+0xd1/0xe0 [12075.878361] [] selinux_file_permission+0xba/0x150 [12075.878366] [] security_file_permission+0x1a/0x70 [12075.878371] [] rw_verify_area+0x58/0xc0 [12075.878376] [] vfs_read+0x9e/0x190 [12075.878381] [] sys_read+0x4f/0x90 [12075.878385] [] system_call_fastpath+0x16/0x1b [12075.878390] [12075.878391] to a SOFTIRQ-irq-unsafe lock: [12075.878393] (est_tree_lock){+.+...} [12075.878396] ... which became SOFTIRQ-irq-unsafe at: [12075.878398] ... [] __lock_acquire+0x5f2/0x1490 [12075.878404] [] lock_acquire+0x93/0x130 [12075.878408] [] _raw_spin_lock+0x31/0x70 [12075.878412] [] gen_kill_estimator+0x1f/0xe0 [12075.878417] [] qdisc_destroy+0x56/0xb0 [12075.878422] [] dev_shutdown+0x71/0xa0 [12075.878426] [] rollback_registered_many+0x1a8/0x310 [12075.878432] [] unregister_netdevice_queue+0x96/0xf0 [12075.878437] [] unregister_netdev+0x1b/0x30 [12075.878442] [] ppp_shutdown_interface+0x15e/0x170 [12075.878447] [] ppp_release+0x61/0xa0 [12075.878452] [] fput+0xf6/0x260 [12075.878456] [] filp_close+0x58/0x90 [12075.878461] [] sys_close+0xb4/0x120 [12075.878466] [] system_call_fastpath+0x16/0x1b [12075.878471] [12075.878472] other info that might help us debug this: [12075.878473] [12075.878476] 3 locks held by tc/17952: [12075.878478] #0: (rtnl_mutex){+.+.+.}, at: [] netlink_dump+0x4b/0x200 [12075.878486] #1: (rcu_read_lock){.+.+..}, at: [] tc_dump_qdisc+0x0/0x120 [12075.878495] #2: (&(&list->lock)->rlock#5){+.-.-.}, at: [] gnet_stats_start_copy_compat+0x44/0xb0 [12075.878504] [12075.878505] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [12075.878519] -> (&(&list->lock)->rlock#5){+.-.-.} ops: 678027 { [12075.878526] HARDIRQ-ON-W at: [12075.878529] [] __lock_acquire+0x5be/0x1490 [12075.878534] [] lock_acquire+0x93/0x130 [12075.878539] [] _raw_spin_lock+0x31/0x70 [12075.878544] [] dev_queue_xmit+0xcc/0x6b0 [12075.878549] [] neigh_resolve_output+0x10e/0x3f0 [12075.878554] [] ip6_finish_output2+0x107/0x3d0 [ipv6] [12075.878565] [] ip6_finish_output+0x564/0xe50 [ipv6] [12075.878575] [] ip6_output+0x4b/0x190 [ipv6] [12075.878584] [] dst_output+0x1e/0x70 [ipv6] [12075.878597] [] mld_sendpack+0x358/0x390 [ipv6] [12075.878611] [] mld_ifc_timer_expire+0x283/0x310 [ipv6] [12075.878625] [] run_timer_softirq+0x1d7/0x400 [12075.878630] [] __do_softirq+0xc2/0x250 [12075.878635] [] call_softirq+0x1c/0x50 [12075.878640] [] do_softirq+0x75/0xb0 [12075.878645] [] irq_exit+0x8d/0xa0 [12075.878649] [] do_IRQ+0x70/0xf0 [12075.878654] [] ret_from_intr+0x0/0xf [12075.878659] [] avc_has_perm_noaudit+0x14f/0x4e0 [12075.878665] [] avc_has_perm+0x46/0x90 [12075.878670] [] inode_has_perm+0x4f/0x90 [12075.878675] [] file_has_perm+0xd1/0xe0 [12075.878679] [] selinux_file_permission+0xba/0x150 [12075.878684] [] security_file_permission+0x1a/0x70 [12075.878690] [] rw_verify_area+0x58/0xc0 [12075.878695] [] vfs_read+0x9e/0x190 [12075.878700] [] sys_read+0x4f/0x90 [12075.878705] [] system_call_fastpath+0x16/0x1b [12075.878711] IN-SOFTIRQ-W at: [12075.878714] [] __lock_acquire+0x58f/0x1490 [12075.878719] [] lock_acquire+0x93/0x130 [12075.878724] [] _raw_spin_lock+0x31/0x70 [12075.878728] [] dev_queue_xmit+0xcc/0x6b0 [12075.878733] [] neigh_resolve_output+0x10e/0x3f0 [12075.878738] [] ip6_finish_output2+0x107/0x3d0 [ipv6] [12075.878749] [] ip6_finish_output+0x564/0xe50 [ipv6] [12075.878759] [] ip6_output+0x4b/0x190 [ipv6] [12075.878768] [] dst_output+0x1e/0x70 [ipv6] [12075.878782] [] mld_sendpack+0x358/0x390 [ipv6] [12075.878795] [] mld_ifc_timer_expire+0x283/0x310 [ipv6] [12075.878809] [] run_timer_softirq+0x1d7/0x400 [12075.878814] [] __do_softirq+0xc2/0x250 [12075.878819] [] call_softirq+0x1c/0x50 [12075.878824] [] do_softirq+0x75/0xb0 [12075.878828] [] irq_exit+0x8d/0xa0 [12075.878833] [] do_IRQ+0x70/0xf0 [12075.878838] [] ret_from_intr+0x0/0xf [12075.878842] [] avc_has_perm_noaudit+0x14f/0x4e0 [12075.878848] [] avc_has_perm+0x46/0x90 [12075.878853] [] inode_has_perm+0x4f/0x90 [12075.878858] [] file_has_perm+0xd1/0xe0 [12075.878863] [] selinux_file_permission+0xba/0x150 [12075.878868] [] security_file_permission+0x1a/0x70 [12075.878873] [] rw_verify_area+0x58/0xc0 [12075.878878] [] vfs_read+0x9e/0x190 [12075.878883] [] sys_read+0x4f/0x90 [12075.878888] [] system_call_fastpath+0x16/0x1b [12075.878894] IN-RECLAIM_FS-W at: [12075.878897] [] __lock_acquire+0x443/0x1490 [12075.878902] [] lock_acquire+0x93/0x130 [12075.878907] [] _raw_spin_lock+0x31/0x70 [12075.878912] [] dev_queue_xmit+0xcc/0x6b0 [12075.878917] [] ip_finish_output+0x1ac/0x4b0 [12075.878922] [] ip_output+0xa1/0x100 [12075.878926] [] ip_local_out+0x28/0x80 [12075.878931] [] ip_queue_xmit+0x1bd/0x4f0 [12075.878936] [] tcp_transmit_skb+0x3f0/0x890 [12075.878942] [] tcp_send_ack+0xd8/0x120 [12075.878947] [] tcp_delack_timer+0x12a/0x220 [12075.878953] [] run_timer_softirq+0x1d7/0x400 [12075.878958] [] __do_softirq+0xc2/0x250 [12075.878963] [] call_softirq+0x1c/0x50 [12075.878967] [] do_softirq+0x75/0xb0 [12075.878972] [] irq_exit+0x8d/0xa0 [12075.878977] [] smp_apic_timer_interrupt+0x6b/0xa0 [12075.878983] [] apic_timer_interrupt+0x13/0x20 [12075.878989] [] _raw_spin_lock+0x31/0x70 [12075.878993] [] __shrink_dcache_sb+0x13e/0x410 [12075.878999] [] shrink_dcache_memory+0x176/0x200 [12075.879004] [] shrink_slab+0x127/0x190 [12075.879009] [] balance_pgdat+0x2cf/0x590 [12075.879015] [] kswapd+0x135/0x3e0 [12075.879020] [] kthread+0xb6/0xc0 [12075.879025] [] kernel_thread_helper+0x4/0x10 [12075.879030] INITIAL USE at: [12075.879033] [] __lock_acquire+0x22b/0x1490 [12075.879038] [] lock_acquire+0x93/0x130 [12075.879043] [] _raw_spin_lock+0x31/0x70 [12075.879047] [] dev_queue_xmit+0xcc/0x6b0 [12075.879052] [] neigh_resolve_output+0x10e/0x3f0 [12075.879058] [] ip6_finish_output2+0x107/0x3d0 [ipv6] [12075.879068] [] ip6_finish_output+0x564/0xe50 [ipv6] [12075.879078] [] ip6_output+0x4b/0x190 [ipv6] [12075.879087] [] dst_output+0x1e/0x70 [ipv6] [12075.879101] [] mld_sendpack+0x358/0x390 [ipv6] [12075.879114] [] mld_ifc_timer_expire+0x283/0x310 [ipv6] [12075.879128] [] run_timer_softirq+0x1d7/0x400 [12075.879133] [] __do_softirq+0xc2/0x250 [12075.879138] [] call_softirq+0x1c/0x50 [12075.879143] [] do_softirq+0x75/0xb0 [12075.879147] [] irq_exit+0x8d/0xa0 [12075.879152] [] do_IRQ+0x70/0xf0 [12075.879156] [] ret_from_intr+0x0/0xf [12075.879161] [] avc_has_perm_noaudit+0x14f/0x4e0 [12075.879167] [] avc_has_perm+0x46/0x90 [12075.879172] [] inode_has_perm+0x4f/0x90 [12075.879177] [] file_has_perm+0xd1/0xe0 [12075.879182] [] selinux_file_permission+0xba/0x150 [12075.879187] [] security_file_permission+0x1a/0x70 [12075.879192] [] rw_verify_area+0x58/0xc0 [12075.879198] [] vfs_read+0x9e/0x190 [12075.879203] [] sys_read+0x4f/0x90 [12075.879208] [] system_call_fastpath+0x16/0x1b [12075.879213] } [12075.879215] ... key at: [] __key.26712+0x0/0x8 [12075.879221] ... acquired at: [12075.879223] [] check_irq_usage+0x60/0xf0 [12075.879228] [] __lock_acquire+0xc91/0x1490 [12075.879232] [] lock_acquire+0x93/0x130 [12075.879237] [] _raw_spin_lock+0x31/0x70 [12075.879241] [] gen_estimator_active+0x22/0xb0 [12075.879246] [] gnet_stats_copy_rate_est+0x1d/0x80 [12075.879251] [] tc_fill_qdisc+0x2c1/0x360 [12075.879256] [] tc_dump_qdisc_root+0x108/0x120 [12075.879261] [] tc_dump_qdisc+0x98/0x120 [12075.879265] [] netlink_dump+0x65/0x200 [12075.879270] [] netlink_dump_start+0x195/0x220 [12075.879274] [] rtnetlink_rcv_msg+0x232/0x270 [12075.879280] [] netlink_rcv_skb+0xa1/0xd0 [12075.879284] [] rtnetlink_rcv+0x20/0x30 [12075.879288] [] netlink_unicast+0x296/0x2e0 [12075.879293] [] netlink_sendmsg+0x250/0x360 [12075.879297] [] sock_sendmsg+0xee/0x120 [12075.879302] [] sys_sendmsg+0x230/0x390 [12075.879307] [] system_call_fastpath+0x16/0x1b [12075.879311] [12075.879313] [12075.879314] the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: [12075.879326] -> (est_tree_lock){+.+...} ops: 7 { [12075.879332] HARDIRQ-ON-W at: [12075.879335] [] __lock_acquire+0x5be/0x1490 [12075.879340] [] lock_acquire+0x93/0x130 [12075.879345] [] _raw_spin_lock+0x31/0x70 [12075.879350] [] gen_kill_estimator+0x1f/0xe0 [12075.879355] [] qdisc_destroy+0x56/0xb0 [12075.879360] [] dev_shutdown+0x71/0xa0 [12075.879365] [] rollback_registered_many+0x1a8/0x310 [12075.879371] [] unregister_netdevice_queue+0x96/0xf0 [12075.879376] [] unregister_netdev+0x1b/0x30 [12075.879382] [] ppp_shutdown_interface+0x15e/0x170 [12075.879387] [] ppp_release+0x61/0xa0 [12075.879392] [] fput+0xf6/0x260 [12075.879397] [] filp_close+0x58/0x90 [12075.879402] [] sys_close+0xb4/0x120 [12075.879407] [] system_call_fastpath+0x16/0x1b [12075.879412] SOFTIRQ-ON-W at: [12075.879415] [] __lock_acquire+0x5f2/0x1490 [12075.879420] [] lock_acquire+0x93/0x130 [12075.879425] [] _raw_spin_lock+0x31/0x70 [12075.879430] [] gen_kill_estimator+0x1f/0xe0 [12075.879435] [] qdisc_destroy+0x56/0xb0 [12075.879440] [] dev_shutdown+0x71/0xa0 [12075.879445] [] rollback_registered_many+0x1a8/0x310 [12075.879451] [] unregister_netdevice_queue+0x96/0xf0 [12075.879457] [] unregister_netdev+0x1b/0x30 [12075.879462] [] ppp_shutdown_interface+0x15e/0x170 [12075.879467] [] ppp_release+0x61/0xa0 [12075.879472] [] fput+0xf6/0x260 [12075.879477] [] filp_close+0x58/0x90 [12075.879482] [] sys_close+0xb4/0x120 [12075.879487] [] system_call_fastpath+0x16/0x1b [12075.879493] INITIAL USE at: [12075.879496] [] __lock_acquire+0x22b/0x1490 [12075.879501] [] lock_acquire+0x93/0x130 [12075.879506] [] _raw_spin_lock+0x31/0x70 [12075.879511] [] gen_kill_estimator+0x1f/0xe0 [12075.879517] [] qdisc_destroy+0x56/0xb0 [12075.879522] [] dev_shutdown+0x71/0xa0 [12075.879527] [] rollback_registered_many+0x1a8/0x310 [12075.879533] [] unregister_netdevice_queue+0x96/0xf0 [12075.879539] [] unregister_netdev+0x1b/0x30 [12075.879545] [] ppp_shutdown_interface+0x15e/0x170 [12075.879551] [] ppp_release+0x61/0xa0 [12075.879556] [] fput+0xf6/0x260 [12075.879561] [] filp_close+0x58/0x90 [12075.879567] [] sys_close+0xb4/0x120 [12075.879572] [] system_call_fastpath+0x16/0x1b [12075.879578] } [12075.879579] ... key at: [] est_tree_lock+0x18/0x40 [12075.879585] ... acquired at: [12075.879586] [] check_irq_usage+0x60/0xf0 [12075.879591] [] __lock_acquire+0xc91/0x1490 [12075.879596] [] lock_acquire+0x93/0x130 [12075.879600] [] _raw_spin_lock+0x31/0x70 [12075.879605] [] gen_estimator_active+0x22/0xb0 [12075.879610] [] gnet_stats_copy_rate_est+0x1d/0x80 [12075.879615] [] tc_fill_qdisc+0x2c1/0x360 [12075.879620] [] tc_dump_qdisc_root+0x108/0x120 [12075.879625] [] tc_dump_qdisc+0x98/0x120 [12075.879629] [] netlink_dump+0x65/0x200 [12075.879634] [] netlink_dump_start+0x195/0x220 [12075.879638] [] rtnetlink_rcv_msg+0x232/0x270 [12075.879643] [] netlink_rcv_skb+0xa1/0xd0 [12075.879647] [] rtnetlink_rcv+0x20/0x30 [12075.879652] [] netlink_unicast+0x296/0x2e0 [12075.879656] [] netlink_sendmsg+0x250/0x360 [12075.879660] [] sock_sendmsg+0xee/0x120 [12075.879665] [] sys_sendmsg+0x230/0x390 [12075.879670] [] system_call_fastpath+0x16/0x1b [12075.879674] [12075.879676] [12075.879677] stack backtrace: [12075.879680] Pid: 17952, comm: tc Not tainted 2.6.36-rc3-next+ #30 [12075.879683] Call Trace: [12075.879687] [] check_usage+0x480/0x490 [12075.879693] [] check_irq_usage+0x60/0xf0 [12075.879697] [] __lock_acquire+0xc91/0x1490 [12075.879702] [] lock_acquire+0x93/0x130 [12075.879706] [] ? gen_estimator_active+0x22/0xb0 [12075.879711] [] ? __nla_reserve+0x53/0x70 [12075.879715] [] _raw_spin_lock+0x31/0x70 [12075.879720] [] ? gen_estimator_active+0x22/0xb0 [12075.879724] [] ? __nla_put+0x2a/0x40 [12075.879728] [] gen_estimator_active+0x22/0xb0 [12075.879733] [] gnet_stats_copy_rate_est+0x1d/0x80 [12075.879738] [] tc_fill_qdisc+0x2c1/0x360 [12075.879743] [] tc_dump_qdisc_root+0x108/0x120 [12075.879748] [] ? __mutex_lock_common+0x27d/0x3a0 [12075.879752] [] ? netlink_dump+0x4b/0x200 [12075.879757] [] tc_dump_qdisc+0x98/0x120 [12075.879761] [] ? tc_dump_qdisc+0x0/0x120 [12075.879765] [] netlink_dump+0x65/0x200 [12075.879770] [] ? trace_hardirqs_on+0xd/0x10 [12075.879774] [] netlink_dump_start+0x195/0x220 [12075.879779] [] ? tc_dump_qdisc+0x0/0x120 [12075.879783] [] rtnetlink_rcv_msg+0x232/0x270 [12075.879788] [] ? rtnetlink_rcv_msg+0x0/0x270 [12075.879792] [] netlink_rcv_skb+0xa1/0xd0 [12075.879797] [] rtnetlink_rcv+0x20/0x30 [12075.879801] [] netlink_unicast+0x296/0x2e0 [12075.879805] [] netlink_sendmsg+0x250/0x360 [12075.879810] [] sock_sendmsg+0xee/0x120 [12075.879815] [] ? might_fault+0x57/0xb0 [12075.879819] [] ? might_fault+0x57/0xb0 [12075.879823] [] ? might_fault+0xa0/0xb0 [12075.879827] [] ? might_fault+0x57/0xb0 [12075.879831] [] ? move_addr_to_kernel+0x5d/0x60 [12075.879836] [] ? verify_iovec+0x80/0xe0 [12075.879840] [] sys_sendmsg+0x230/0x390 [12075.879846] [] ? sched_clock_cpu+0xcd/0x110 [12075.879850] [] ? trace_hardirqs_off+0xd/0x10 [12075.879854] [] ? up_read+0x1e/0x40 [12075.879860] [] ? audit_syscall_entry+0x24c/0x270 [12075.879865] [] ? trace_hardirqs_on_caller+0x145/0x190 [12075.879869] [] ? trace_hardirqs_on_thunk+0x3a/0x3f [12075.879874] [] system_call_fastpath+0x16/0x1b --- a/net/core/gen_estimator.c 2010-09-08 11:16:10.184368926 -0700 +++ b/net/core/gen_estimator.c 2010-09-08 11:16:24.694194050 -0700 @@ -270,18 +270,18 @@ void gen_kill_estimator(struct gnet_stat { struct gen_estimator *e; - spin_lock(&est_tree_lock); + spin_lock_bh(&est_tree_lock); while ((e = gen_find_node(bstats, rate_est))) { rb_erase(&e->node, &est_root); - write_lock_bh(&est_lock); + write_lock(&est_lock); e->bstats = NULL; - write_unlock_bh(&est_lock); + write_unlock(&est_lock); list_del_rcu(&e->list); call_rcu(&e->e_rcu, __gen_kill_estimator); } - spin_unlock(&est_tree_lock); + spin_unlock_bh(&est_tree_lock); } EXPORT_SYMBOL(gen_kill_estimator);