* tbl->lock not taken in neigh_lookup() ?
@ 2016-01-21 15:35 Ani Sinha
2016-01-25 4:41 ` Ani Sinha
0 siblings, 1 reply; 4+ messages in thread
From: Ani Sinha @ 2016-01-21 15:35 UTC (permalink / raw)
To: netdev, Eric W. Biederman
hi guys
As per the comment at the top of net/core/neighbor.c we should be
taking this lock even for scanning the hash buckets. I do see that
this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i
missing something?
For the context I am investigating the following crash which happened
on one of our boxes. This is in 3.18.19 kernel where the following
code in neigh_lookup() returns a corrupt value of neighbour* :
for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]);
[ 4918.117044] BUG: unable to handle kernel paging request at 0000000014000270
[ 4918.200203] IP: [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
[ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0
[ 4918.266795] Oops: 0000 [#1] PREEMPT SMP
[ 4918.266798] Modules linked in: macvlan l2mod_dma(PO)
arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6
nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4
nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl
xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp
iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr
kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O)
plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables
xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables
x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm
[last unloaded: plxnt_ll]
[ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 #1
[ 4918.266840] task: ffff8803c6dd3720 ti: ffff88007b98c000 task.ti:
ffff88007b98c000
[ 4918.266841] RIP: 0010:[<ffffffff813b84d5>] [<ffffffff813b84d5>]
neigh_lookup+0x64/0xdf
[ 4918.266847] RSP: 0018:ffff88007b98faa8 EFLAGS: 00210206
[ 4918.266849] RAX: ffff8803203bf278 RBX: ffffffff81866990 RCX: 0000000000000014
[ 4918.266851] RDX: ffff88032a46624c RSI: ffff88005ff82000 RDI: ffff880319ee6020
[ 4918.266852] RBP: ffff88007b98fad8 R08: 000000000a000000 R09: ffff880319ee6000
[ 4918.266854] R10: ffff88007b98faf8 R11: ffff88043d001900 R12: 0000000014000100
[ 4918.266855] R13: ffff880319ee6020 R14: ffff88005ff82000 R15: 0000000000000010
[ 4918.266857] FS: 0000000000000000(0000) GS:ffff88044f540000(0063)
knlGS:00000000f73cb8e0
[ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[ 4918.266860] CR2: 0000000014000270 CR3: 000000007e371000 CR4: 00000000000007e0
[ 4918.266862] Stack:
[ 4918.266863] 0000000000000180 ffffffff8185cd00 ffffffff81866990
ffff880319ee6010
[ 4918.266866] ffff880319ee601c ffff88005ff82000 ffff88007b98fb28
ffffffff813b97c2
[ 4918.266869] ffff88007b98faf8 ffffffff818282c0 ffff88007b98fb18
ffff880377411600
[ 4918.266872] Call Trace:
[ 4918.266875] [<ffffffff813b97c2>] neigh_delete+0x113/0x17f
[ 4918.266878] [<ffffffff813bdf1e>] rtnetlink_rcv_msg+0x18a/0x1a0
[ 4918.266882] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
[ 4918.266885] [<ffffffff812005df>] ? rhashtable_lookup_compare+0x4b/0x71
[ 4918.266887] [<ffffffff813bdd94>] ? rtnetlink_rcv_msg+0x0/0x1a0
[ 4918.266890] [<ffffffff813cfd58>] netlink_rcv_skb+0x3e/0x94
[ 4918.266891] [<ffffffff813be5ea>] rtnetlink_rcv+0x21/0x28
[ 4918.266893] [<ffffffff813cfaed>] netlink_unicast+0x10b/0x1ad
[ 4918.266895] [<ffffffff813d0304>] netlink_sendmsg+0x2e6/0x327
[ 4918.266898] [<ffffffff81397efa>] sock_sendmsg+0x6d/0x86
[ 4918.266901] [<ffffffff8139a2c3>] ? sock_poll+0x10e/0x11c
[ 4918.266903] [<ffffffff813974d0>] ? sockfd_lookup_light+0x12/0x5d
[ 4918.266906] [<ffffffff81398006>] SyS_sendto+0xf3/0x11b
[ 4918.266909] [<ffffffff81486667>] ? __mutex_lock_slowpath+0x2de/0x2fe
[ 4918.266911] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
[ 4918.266914] [<ffffffff81396dd0>] SyS_send+0xf/0x11
[ 4918.266916] [<ffffffff813c61c7>] compat_SyS_socketcall+0x122/0x1e3
[ 4918.266919] [<ffffffff8148b6df>] sysenter_dispatch+0x7/0x1e
[ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18
b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c
8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c
89 fa
[ 4918.266947] RIP [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
[ 4918.334502] RSP <ffff88007b98faa8>
[ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4
12:34:37 PST 2016
[ 4918.455186] CR2: 0000000014000270
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tbl->lock not taken in neigh_lookup() ?
2016-01-21 15:35 tbl->lock not taken in neigh_lookup() ? Ani Sinha
@ 2016-01-25 4:41 ` Ani Sinha
2016-01-25 5:10 ` Eric Dumazet
2016-01-25 5:20 ` David Miller
0 siblings, 2 replies; 4+ messages in thread
From: Ani Sinha @ 2016-01-25 4:41 UTC (permalink / raw)
To: netdev, Eric W. Biederman, Eric Dumazet
Hi All:
Can I get some insights into this? I am sure I am missing something.
thanks
ani
On Thu, Jan 21, 2016 at 9:05 PM, Ani Sinha <ani@arista.com> wrote:
> hi guys
>
> As per the comment at the top of net/core/neighbor.c we should be
> taking this lock even for scanning the hash buckets. I do see that
> this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i
> missing something?
>
> For the context I am investigating the following crash which happened
> on one of our boxes. This is in 3.18.19 kernel where the following
> code in neigh_lookup() returns a corrupt value of neighbour* :
>
> for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]);
>
>
>
> [ 4918.117044] BUG: unable to handle kernel paging request at 0000000014000270
> [ 4918.200203] IP: [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
> [ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0
> [ 4918.266795] Oops: 0000 [#1] PREEMPT SMP
> [ 4918.266798] Modules linked in: macvlan l2mod_dma(PO)
> arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6
> nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4
> nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl
> xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp
> iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr
> kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O)
> plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables
> xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables
> x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm
> [last unloaded: plxnt_ll]
> [ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 #1
> [ 4918.266840] task: ffff8803c6dd3720 ti: ffff88007b98c000 task.ti:
> ffff88007b98c000
> [ 4918.266841] RIP: 0010:[<ffffffff813b84d5>] [<ffffffff813b84d5>]
> neigh_lookup+0x64/0xdf
> [ 4918.266847] RSP: 0018:ffff88007b98faa8 EFLAGS: 00210206
> [ 4918.266849] RAX: ffff8803203bf278 RBX: ffffffff81866990 RCX: 0000000000000014
> [ 4918.266851] RDX: ffff88032a46624c RSI: ffff88005ff82000 RDI: ffff880319ee6020
> [ 4918.266852] RBP: ffff88007b98fad8 R08: 000000000a000000 R09: ffff880319ee6000
> [ 4918.266854] R10: ffff88007b98faf8 R11: ffff88043d001900 R12: 0000000014000100
> [ 4918.266855] R13: ffff880319ee6020 R14: ffff88005ff82000 R15: 0000000000000010
> [ 4918.266857] FS: 0000000000000000(0000) GS:ffff88044f540000(0063)
> knlGS:00000000f73cb8e0
> [ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
> [ 4918.266860] CR2: 0000000014000270 CR3: 000000007e371000 CR4: 00000000000007e0
> [ 4918.266862] Stack:
> [ 4918.266863] 0000000000000180 ffffffff8185cd00 ffffffff81866990
> ffff880319ee6010
> [ 4918.266866] ffff880319ee601c ffff88005ff82000 ffff88007b98fb28
> ffffffff813b97c2
> [ 4918.266869] ffff88007b98faf8 ffffffff818282c0 ffff88007b98fb18
> ffff880377411600
> [ 4918.266872] Call Trace:
> [ 4918.266875] [<ffffffff813b97c2>] neigh_delete+0x113/0x17f
> [ 4918.266878] [<ffffffff813bdf1e>] rtnetlink_rcv_msg+0x18a/0x1a0
> [ 4918.266882] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
> [ 4918.266885] [<ffffffff812005df>] ? rhashtable_lookup_compare+0x4b/0x71
> [ 4918.266887] [<ffffffff813bdd94>] ? rtnetlink_rcv_msg+0x0/0x1a0
> [ 4918.266890] [<ffffffff813cfd58>] netlink_rcv_skb+0x3e/0x94
> [ 4918.266891] [<ffffffff813be5ea>] rtnetlink_rcv+0x21/0x28
> [ 4918.266893] [<ffffffff813cfaed>] netlink_unicast+0x10b/0x1ad
> [ 4918.266895] [<ffffffff813d0304>] netlink_sendmsg+0x2e6/0x327
> [ 4918.266898] [<ffffffff81397efa>] sock_sendmsg+0x6d/0x86
> [ 4918.266901] [<ffffffff8139a2c3>] ? sock_poll+0x10e/0x11c
> [ 4918.266903] [<ffffffff813974d0>] ? sockfd_lookup_light+0x12/0x5d
> [ 4918.266906] [<ffffffff81398006>] SyS_sendto+0xf3/0x11b
> [ 4918.266909] [<ffffffff81486667>] ? __mutex_lock_slowpath+0x2de/0x2fe
> [ 4918.266911] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
> [ 4918.266914] [<ffffffff81396dd0>] SyS_send+0xf/0x11
> [ 4918.266916] [<ffffffff813c61c7>] compat_SyS_socketcall+0x122/0x1e3
> [ 4918.266919] [<ffffffff8148b6df>] sysenter_dispatch+0x7/0x1e
> [ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18
> b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c
> 8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c
> 89 fa
> [ 4918.266947] RIP [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
> [ 4918.334502] RSP <ffff88007b98faa8>
> [ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4
> 12:34:37 PST 2016
>
> [ 4918.455186] CR2: 0000000014000270
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tbl->lock not taken in neigh_lookup() ?
2016-01-25 4:41 ` Ani Sinha
@ 2016-01-25 5:10 ` Eric Dumazet
2016-01-25 5:20 ` David Miller
1 sibling, 0 replies; 4+ messages in thread
From: Eric Dumazet @ 2016-01-25 5:10 UTC (permalink / raw)
To: Ani Sinha; +Cc: netdev, Eric W. Biederman
On Mon, 2016-01-25 at 10:11 +0530, Ani Sinha wrote:
> Hi All:
>
> Can I get some insights into this? I am sure I am missing something.
>
> thanks
> ani
>
>
> On Thu, Jan 21, 2016 at 9:05 PM, Ani Sinha <ani@arista.com> wrote:
> > hi guys
> >
> > As per the comment at the top of net/core/neighbor.c we should be
> > taking this lock even for scanning the hash buckets. I do see that
> > this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i
> > missing something?
> >
> > For the context I am investigating the following crash which happened
> > on one of our boxes. This is in 3.18.19 kernel where the following
> > code in neigh_lookup() returns a corrupt value of neighbour* :
> >
> > for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]);
> >
neigh_lookup() encloses its loop in
rcu_read_lock_bh()
..
rcu_read_unlock_bh();
So it uses RCU BH locking.
> >
> >
> > [ 4918.117044] BUG: unable to handle kernel paging request at 0000000014000270
> > [ 4918.200203] IP: [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
> > [ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0
> > [ 4918.266795] Oops: 0000 [#1] PREEMPT SMP
> > [ 4918.266798] Modules linked in: macvlan l2mod_dma(PO)
> > arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6
> > nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4
> > nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl
> > xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp
> > iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr
> > kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O)
> > plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables
> > xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables
> > x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm
> > [last unloaded: plxnt_ll]
> > [ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 #1
> > [ 4918.266840] task: ffff8803c6dd3720 ti: ffff88007b98c000 task.ti:
> > ffff88007b98c000
> > [ 4918.266841] RIP: 0010:[<ffffffff813b84d5>] [<ffffffff813b84d5>]
> > neigh_lookup+0x64/0xdf
> > [ 4918.266847] RSP: 0018:ffff88007b98faa8 EFLAGS: 00210206
> > [ 4918.266849] RAX: ffff8803203bf278 RBX: ffffffff81866990 RCX: 0000000000000014
> > [ 4918.266851] RDX: ffff88032a46624c RSI: ffff88005ff82000 RDI: ffff880319ee6020
> > [ 4918.266852] RBP: ffff88007b98fad8 R08: 000000000a000000 R09: ffff880319ee6000
> > [ 4918.266854] R10: ffff88007b98faf8 R11: ffff88043d001900 R12: 0000000014000100
> > [ 4918.266855] R13: ffff880319ee6020 R14: ffff88005ff82000 R15: 0000000000000010
> > [ 4918.266857] FS: 0000000000000000(0000) GS:ffff88044f540000(0063)
> > knlGS:00000000f73cb8e0
> > [ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
> > [ 4918.266860] CR2: 0000000014000270 CR3: 000000007e371000 CR4: 00000000000007e0
> > [ 4918.266862] Stack:
> > [ 4918.266863] 0000000000000180 ffffffff8185cd00 ffffffff81866990
> > ffff880319ee6010
> > [ 4918.266866] ffff880319ee601c ffff88005ff82000 ffff88007b98fb28
> > ffffffff813b97c2
> > [ 4918.266869] ffff88007b98faf8 ffffffff818282c0 ffff88007b98fb18
> > ffff880377411600
> > [ 4918.266872] Call Trace:
> > [ 4918.266875] [<ffffffff813b97c2>] neigh_delete+0x113/0x17f
> > [ 4918.266878] [<ffffffff813bdf1e>] rtnetlink_rcv_msg+0x18a/0x1a0
> > [ 4918.266882] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
> > [ 4918.266885] [<ffffffff812005df>] ? rhashtable_lookup_compare+0x4b/0x71
> > [ 4918.266887] [<ffffffff813bdd94>] ? rtnetlink_rcv_msg+0x0/0x1a0
> > [ 4918.266890] [<ffffffff813cfd58>] netlink_rcv_skb+0x3e/0x94
> > [ 4918.266891] [<ffffffff813be5ea>] rtnetlink_rcv+0x21/0x28
> > [ 4918.266893] [<ffffffff813cfaed>] netlink_unicast+0x10b/0x1ad
> > [ 4918.266895] [<ffffffff813d0304>] netlink_sendmsg+0x2e6/0x327
> > [ 4918.266898] [<ffffffff81397efa>] sock_sendmsg+0x6d/0x86
> > [ 4918.266901] [<ffffffff8139a2c3>] ? sock_poll+0x10e/0x11c
> > [ 4918.266903] [<ffffffff813974d0>] ? sockfd_lookup_light+0x12/0x5d
> > [ 4918.266906] [<ffffffff81398006>] SyS_sendto+0xf3/0x11b
> > [ 4918.266909] [<ffffffff81486667>] ? __mutex_lock_slowpath+0x2de/0x2fe
> > [ 4918.266911] [<ffffffff8105580c>] ? get_parent_ip+0x11/0x42
> > [ 4918.266914] [<ffffffff81396dd0>] SyS_send+0xf/0x11
> > [ 4918.266916] [<ffffffff813c61c7>] compat_SyS_socketcall+0x122/0x1e3
> > [ 4918.266919] [<ffffffff8148b6df>] sysenter_dispatch+0x7/0x1e
> > [ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18
> > b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c
> > 8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c
> > 89 fa
> > [ 4918.266947] RIP [<ffffffff813b84d5>] neigh_lookup+0x64/0xdf
> > [ 4918.334502] RSP <ffff88007b98faa8>
> > [ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4
> > 12:34:37 PST 2016
> >
> > [ 4918.455186] CR2: 0000000014000270
You might try to reproduce the bug on linux-4.4.
3.18.19 is rather old.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: tbl->lock not taken in neigh_lookup() ?
2016-01-25 4:41 ` Ani Sinha
2016-01-25 5:10 ` Eric Dumazet
@ 2016-01-25 5:20 ` David Miller
1 sibling, 0 replies; 4+ messages in thread
From: David Miller @ 2016-01-25 5:20 UTC (permalink / raw)
To: ani; +Cc: netdev, ebiederm, eric.dumazet
From: Ani Sinha <ani@arista.com>
Date: Mon, 25 Jan 2016 10:11:15 +0530
> Can I get some insights into this? I am sure I am missing something.
The whole point of RCU locking is that read accesses in the fast paths
(lookups) do not need to take the spinlock. Proper RCU barriers, RCU
deferred freeing of objects, and other things make it safe.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-01-25 5:20 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-21 15:35 tbl->lock not taken in neigh_lookup() ? Ani Sinha
2016-01-25 4:41 ` Ani Sinha
2016-01-25 5:10 ` Eric Dumazet
2016-01-25 5:20 ` David Miller
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.