* Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
@ 2011-05-17 22:16 Denys Fedoryshchenko
2011-05-18 9:27 ` Denys Fedoryshchenko
0 siblings, 1 reply; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-17 22:16 UTC (permalink / raw)
To: netdev
Just got recently. 32Bit, PPPoE NAS, shapers, firewall, NAT
Kernel i mention in subject, 2.6.39-rc7-git11
If required i can give more information
sharanal (sorry for ugly name) is libpcap based traffic analyser, sure
userspace
[44528.153869] BUG: unable to handle kernel NULL pointer dereference
at 0000001a
[44528.153869] IP: [<c02e8614>] cleanup_once+0x49/0x1cf
[44528.153869] *pdpt = 0000000034a73001 *pde = 0000000000000000
[44528.153869] Oops: 0002 [#1] SMP
[44528.153869] last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_map
[44528.153869] Modules linked in: nf_conntrack_netlink nfnetlink
ipt_LOG rtc_cmos rtc_core rtc_lib act_skbedit sch_ingress sch_prio
bridge cls_flow cls_u32 em_meta cls_basic xt_dscp ipt_REJECT xt_hl ifb
cls_fw sch_tbf sch_htb a
[44528.153869]
[44528.153869] Pid: 1744, comm: sharanal Not tainted
2.6.39-rc7-git11-build-0058 #6 Intel
/SE7520BD2S
[44528.153869] EIP: 0060:[<c02e8614>] EFLAGS: 00010286 CPU: 1
[44528.153869] EIP is at cleanup_once+0x49/0x1cf
[44528.153869] EAX: dd378920 EBX: dd378900 ECX: 00000016 EDX: 06000001
[44528.153869] ESI: 00000000 EDI: e4b6a59c EBP: f5485b8c ESP: f5485b68
[44528.153869] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[44528.153869] Process sharanal (pid: 1744, ti=f5484000 task=f4c9b6c0
task.ti=f4fa4000)
[44528.153869] Stack:
[44528.153869] f5485c6c f5485ba4 f5485ba4 00000013 f5485ba4 f5485b84
e4b6a500 f5485c6c
[44528.153869] e4b6a59c f5485c50 c02e8b30 f5485bec f5485c58 00000000
c04602d4 c04602d4
[44528.153869] e4252504 f0aecc04 e3ee3200 e8d1b200 efd0ab00 e489dc00
e422e900 e9b18100
[44528.153869] Call Trace:
[44528.153869] [<c02e8b30>] inet_getpeer+0x2ab/0x2cf
[44528.153869] [<c03080bc>] ?
icmp_route_lookup.clone.19.clone.20+0x197/0x1cb
[44528.153869] [<c01320a5>] ? _local_bh_enable_ip.clone.6+0x18/0x71
[44528.153869] [<c0132106>] ? local_bh_enable_ip+0x8/0xa
[44528.153869] [<f88c0929>] ? nf_nat_setup_info+0x3b0/0x3db [nf_nat]
[44528.153869] [<c0318591>] ? ipt_do_table+0x41c/0x437
[44528.153869] [<c02e56a8>] inet_getpeer_v4+0x17/0x19
[44528.153869] [<c02e7a72>] rt_bind_peer+0xe/0x39
[44528.153869] [<c030818d>] icmpv4_xrlim_allow.clone.22+0x4b/0x5f
[44528.153869] [<c03083d4>] icmp_send+0x203/0x282
[44528.153869] [<f88b23aa>] ? ipv4_confirm+0x131/0x13e
[nf_conntrack_ipv4]
[44528.153869] [<c030633b>] __udp4_lib_rcv+0x2d7/0x3f9
[44528.153869] [<c02e2d25>] ? nf_iterate+0x52/0x65
[44528.153869] [<c02e8efb>] ? xfrm4_policy_check.clone.10+0x47/0x47
[44528.153869] [<c030646f>] udp_rcv+0x12/0x14
[44528.153869] [<c02e8fc5>] ip_local_deliver_finish+0xca/0x171
[44528.153869] [<c02e8efb>] ? xfrm4_policy_check.clone.10+0x47/0x47
[44528.153869] [<c02e90b2>] NF_HOOK.clone.11+0x46/0x4d
[44528.153869] [<c02e8efb>] ? xfrm4_policy_check.clone.10+0x47/0x47
[44528.153869] [<c02e91b6>] ip_local_deliver+0x41/0x45
[44528.153869] [<c02e8efb>] ? xfrm4_policy_check.clone.10+0x47/0x47
[44528.153869] [<c02e8e92>] ip_rcv_finish+0x2b4/0x2d6
[44528.153869] [<c02e8bde>] ? pskb_may_pull+0x30/0x30
[44528.153869] [<c02e90b2>] NF_HOOK.clone.11+0x46/0x4d
[44528.153869] [<c02e8bde>] ? pskb_may_pull+0x30/0x30
[44528.153869] [<c02e93a0>] ip_rcv+0x1e6/0x21a
[44528.153869] [<c02e8bde>] ? pskb_may_pull+0x30/0x30
[44528.153869] [<c02c929d>] __netif_receive_skb+0x351/0x379
[44528.153869] [<c02c93f5>] netif_receive_skb+0x46/0x4c
[44528.153869] [<c02c9826>] ? __napi_gro_receive+0x9e/0xa6
[44528.153869] [<c02c94b6>] napi_skb_finish+0x1e/0x34
[44528.153869] [<c02c987d>] napi_gro_receive+0x20/0x24
[44528.153869] [<f84e6516>] e1000_receive_skb+0x5a/0x62 [e1000]
[44528.153869] [<f84e8ac9>] e1000_clean_rx_irq+0x28d/0x323 [e1000]
[44528.153869] [<f84e845d>] e1000_clean+0x2cc/0x43e [e1000]
[44528.153869] [<c02db37e>] ? qdisc_watchdog_schedule+0x39/0x3e
[44528.153869] [<f8cd8221>] ? tbf_dequeue+0x1d/0x1b6 [sch_tbf]
[44528.153869] [<c02b0d50>] ? dma_issue_pending_all+0x60/0x6e
[44528.153869] [<c02c9961>] net_rx_action+0x86/0x139
[44528.153869] [<c0132179>] __do_softirq+0x67/0xf3
[44528.153869] [<c0132112>] ? local_bh_enable+0xa/0xa
[44528.153869] <IRQ>
[44528.153869] [<c0132362>] ? irq_exit+0x35/0x70
[44528.153869] [<c0103d13>] ? do_IRQ+0x79/0x8d
[44528.153869] [<c0132379>] ? irq_exit+0x4c/0x70
[44528.153869] [<c011598a>] ? smp_apic_timer_interrupt+0x66/0x73
[44528.153869] [<c0337c29>] ? common_interrupt+0x29/0x30
[44528.153869] [<c033007b>] ? cpu_init+0x65/0x1c5
[44528.153869] Code: c8 02 46 c0 74 3a 8d 58 e0 8b 15 40 9a 44 c0 2b
53 28 39 f2 73 0f b8 d0 02 46 c0 e8 6b e7 04 00 e9 81 01 00 00 8b 4b 20
8b 53 24
9>[44528.153869] 51 04 89 0a 89 43 20 89 43 24 83 c0 0c e8 cd fd ff
ff eb 02
[44528.153869] EIP: [<c02e8614>] cleanup_once+0x49/0x1cf SS:ESP
0068:f5485b68
[44528.153869] CR2: 000000000000001a
[44528.167278] ---[ end trace dd3639ec5ab2f01f ]---
[44528.167468] Kernel panic - not syncing: Fatal exception in
interrupt
[44528.167660] Pid: 1744, comm: sharanal Tainted: G D
2.6.39-rc7-git11-build-0058 #6
[44528.167992] Call Trace:
[44528.168190] [<c0335548>] ? printk+0x18/0x20
[44528.168374] [<c0335435>] panic+0x57/0x152
[44528.168567] [<c0104e7a>] oops_end+0x92/0x9f
[44528.168759] [<c011b95e>] no_context+0x151/0x159
[44528.168948] [<c011ba72>] __bad_area_nosemaphore+0x10c/0x114
[44528.169154] [<c02ca7df>] ? dev_hard_start_xmit+0x338/0x3f8
[44528.169353] [<c011ba8c>] bad_area_nosemaphore+0x12/0x14
[44528.169544] [<c011bd0c>] do_page_fault+0x12e/0x2ee
[44528.169737] [<c021ba8f>] ? memcmp+0xe/0x25
[44528.169922] [<c01320a5>] ? _local_bh_enable_ip.clone.6+0x18/0x71
[44528.170125] [<c0132110>] ? local_bh_enable+0x8/0xa
[44528.170326] [<c02d1a97>] ? neigh_lookup+0x8b/0x95
[44528.170517] [<c011bbde>] ? vmalloc_sync_all+0x5/0x5
[44528.170713] [<c03374da>] error_code+0x5a/0x60
[44528.170907] [<c01300d8>] ? wait_consider_task+0x3f2/0x76f
[44528.171111] [<c011bbde>] ? vmalloc_sync_all+0x5/0x5
[44528.171304] [<c02e8614>] ? cleanup_once+0x49/0x1cf
[44528.171484] [<c02e8b30>] inet_getpeer+0x2ab/0x2cf
[44528.171672] [<c03080bc>] ?
icmp_route_lookup.clone.19.clone.20+0x197/0x1cb
[44528.171861] [<c01320a5>] ? _local_bh_enable_ip.clone.6+0x18/0x71
[44528.172063] [<c0132106>] ? local_bh_enable_ip+0x8/0xa
[44528.172262] [<f88c0929>] ? nf_nat_setup_info+0x3b0/0x3db [nf_nat]
[44528.172445] [<c0318591>] ? ipt_do_table+0x41c/0x437
[44528.172626] [<c02e56a8>] inet_getpeer_v4+0x17/0x19
[44528.172809] [<c02e7a72>] rt_bind_peer+0xe/0x39
[44528.172990] [<c030818d>] icmpv4_xrlim_allow.clone.22+0x4b/0x5f
[44528.173196] [<c03083d4>] icmp_send+0x203/0x282+++
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-17 22:16 Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11 Denys Fedoryshchenko
@ 2011-05-18 9:27 ` Denys Fedoryshchenko
2011-05-18 9:37 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-18 9:27 UTC (permalink / raw)
To: netdev
On Wed, 18 May 2011 01:16:29 +0300, Denys Fedoryshchenko wrote:
> Just got recently. 32Bit, PPPoE NAS, shapers, firewall, NAT
> Kernel i mention in subject, 2.6.39-rc7-git11
> If required i can give more information
>
> sharanal (sorry for ugly name) is libpcap based traffic analyser,
> sure userspace
>
Here is some info, i hope it will be a little useful
(gdb) l *(cleanup_once + 0x49)
0xc02e85cc is in cleanup_once (include/linux/list.h:88).
83 * This is only for internal list manipulation where we know
84 * the prev/next entries already!
85 */
86 static inline void __list_del(struct list_head * prev, struct
list_head * next)
87 {
88 next->prev = prev;
89 prev->next = next;
90 }
91
92 /**
(gdb) l *(inet_getpeer + 0x2ab)
0xc02e8ae8 is in inet_getpeer (net/ipv4/inetpeer.c:530).
525 if (base->total >= inet_peer_threshold)
526 /* Remove one less-recently-used entry. */
527 cleanup_once(0, stack);
528
529 return p;
530 }
531
532 static int compute_total(void)
533 {
534 return v4_peers.total + v6_peers.total;
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 9:27 ` Denys Fedoryshchenko
@ 2011-05-18 9:37 ` Eric Dumazet
2011-05-18 9:53 ` Denys Fedoryshchenko
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-18 9:37 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev
Le mercredi 18 mai 2011 à 12:27 +0300, Denys Fedoryshchenko a écrit :
> On Wed, 18 May 2011 01:16:29 +0300, Denys Fedoryshchenko wrote:
> > Just got recently. 32Bit, PPPoE NAS, shapers, firewall, NAT
> > Kernel i mention in subject, 2.6.39-rc7-git11
> > If required i can give more information
> >
> > sharanal (sorry for ugly name) is libpcap based traffic analyser,
> > sure userspace
> >
> Here is some info, i hope it will be a little useful
>
> (gdb) l *(cleanup_once + 0x49)
> 0xc02e85cc is in cleanup_once (include/linux/list.h:88).
> 83 * This is only for internal list manipulation where we know
> 84 * the prev/next entries already!
> 85 */
> 86 static inline void __list_del(struct list_head * prev, struct
> list_head * next)
> 87 {
> 88 next->prev = prev;
> 89 prev->next = next;
> 90 }
> 91
> 92 /**
>
> (gdb) l *(inet_getpeer + 0x2ab)
> 0xc02e8ae8 is in inet_getpeer (net/ipv4/inetpeer.c:530).
> 525 if (base->total >= inet_peer_threshold)
> 526 /* Remove one less-recently-used entry. */
> 527 cleanup_once(0, stack);
> 528
> 529 return p;
> 530 }
> 531
> 532 static int compute_total(void)
> 533 {
> 534 return v4_peers.total + v6_peers.total;
>
I really begin to think we have a bug here...
In previous reports, I suggested to use slub_nomerge because I thought
one corruption from another kernel layer was going on.
(inetpeer was using 64 bytes objects). But now that inetpeer objects are
bigger and sit in another kmemcache, its bad news.
Could you try this, and eventually add some SLUB debugging stuff as
well ?
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 9:37 ` Eric Dumazet
@ 2011-05-18 9:53 ` Denys Fedoryshchenko
2011-05-18 10:05 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-18 9:53 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On Wed, 18 May 2011 11:37:51 +0200, Eric Dumazet wrote:
> Le mercredi 18 mai 2011 à 12:27 +0300, Denys Fedoryshchenko a écrit :
>> On Wed, 18 May 2011 01:16:29 +0300, Denys Fedoryshchenko wrote:
>> > Just got recently. 32Bit, PPPoE NAS, shapers, firewall, NAT
>> > Kernel i mention in subject, 2.6.39-rc7-git11
>> > If required i can give more information
>> >
>> > sharanal (sorry for ugly name) is libpcap based traffic analyser,
>> > sure userspace
>> >
>> Here is some info, i hope it will be a little useful
>>
>> (gdb) l *(cleanup_once + 0x49)
>> 0xc02e85cc is in cleanup_once (include/linux/list.h:88).
>> 83 * This is only for internal list manipulation where we
>> know
>> 84 * the prev/next entries already!
>> 85 */
>> 86 static inline void __list_del(struct list_head * prev,
>> struct
>> list_head * next)
>> 87 {
>> 88 next->prev = prev;
>> 89 prev->next = next;
>> 90 }
>> 91
>> 92 /**
>>
>> (gdb) l *(inet_getpeer + 0x2ab)
>> 0xc02e8ae8 is in inet_getpeer (net/ipv4/inetpeer.c:530).
>> 525 if (base->total >= inet_peer_threshold)
>> 526 /* Remove one less-recently-used entry. */
>> 527 cleanup_once(0, stack);
>> 528
>> 529 return p;
>> 530 }
>> 531
>> 532 static int compute_total(void)
>> 533 {
>> 534 return v4_peers.total + v6_peers.total;
>>
>
> I really begin to think we have a bug here...
>
> In previous reports, I suggested to use slub_nomerge because I
> thought
> one corruption from another kernel layer was going on.
>
> (inetpeer was using 64 bytes objects). But now that inetpeer objects
> are
> bigger and sit in another kmemcache, its bad news.
>
> Could you try this, and eventually add some SLUB debugging stuff as
> well ?
Yes, i will try. I should enable SLUB debugging only, or also anything
else?
But possible it will take time to reproduce bug, seems it is occuring
rare. With 2.6.39 release i will rollout update to few hundreds PPPoE's,
maybe it will increase chances to get information.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 9:53 ` Denys Fedoryshchenko
@ 2011-05-18 10:05 ` Eric Dumazet
2011-05-18 11:44 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-18 10:05 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev
Le mercredi 18 mai 2011 à 12:53 +0300, Denys Fedoryshchenko a écrit :
> Yes, i will try. I should enable SLUB debugging only, or also anything
> else?
>
> But possible it will take time to reproduce bug, seems it is occuring
> rare. With 2.6.39 release i will rollout update to few hundreds PPPoE's,
> maybe it will increase chances to get information.
>
I would try both : slub_debug=ZFP slub_nomerge
or maybe only slub_debug=ZFPU
Thanks !
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 10:05 ` Eric Dumazet
@ 2011-05-18 11:44 ` Eric Dumazet
2011-05-18 12:46 ` Denys Fedoryshchenko
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-18 11:44 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev
Le mercredi 18 mai 2011 à 12:05 +0200, Eric Dumazet a écrit :
> Le mercredi 18 mai 2011 à 12:53 +0300, Denys Fedoryshchenko a écrit :
>
> > Yes, i will try. I should enable SLUB debugging only, or also anything
> > else?
> >
> > But possible it will take time to reproduce bug, seems it is occuring
> > rare. With 2.6.39 release i will rollout update to few hundreds PPPoE's,
> > maybe it will increase chances to get information.
> >
>
> I would try both : slub_debug=ZFP slub_nomerge
>
> or maybe only slub_debug=ZFPU
>
> Thanks !
>
Hmm, do you have both ipv6/ipv4 trafic on your machine by any chance ?
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 11:44 ` Eric Dumazet
@ 2011-05-18 12:46 ` Denys Fedoryshchenko
2011-05-18 15:52 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-18 12:46 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
On Wed, 18 May 2011 13:44:07 +0200, Eric Dumazet wrote:
> Le mercredi 18 mai 2011 à 12:05 +0200, Eric Dumazet a écrit :
>> Le mercredi 18 mai 2011 à 12:53 +0300, Denys Fedoryshchenko a écrit
>> :
>>
>> > Yes, i will try. I should enable SLUB debugging only, or also
>> anything
>> > else?
>> >
>> > But possible it will take time to reproduce bug, seems it is
>> occuring
>> > rare. With 2.6.39 release i will rollout update to few hundreds
>> PPPoE's,
>> > maybe it will increase chances to get information.
>> >
>>
>> I would try both : slub_debug=ZFP slub_nomerge
>>
>> or maybe only slub_debug=ZFPU
>>
>> Thanks !
>>
>
> Hmm, do you have both ipv6/ipv4 trafic on your machine by any chance
> ?
It is NAS, has ipv6 enabled recently (i am preparing for ipv6), but
ipv6 is not routed anywhere, maybe just automatically addresses
appearing on interfaces, including the one looking to customer subnet.
ppp is ipv4 only.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 12:46 ` Denys Fedoryshchenko
@ 2011-05-18 15:52 ` Eric Dumazet
2011-05-18 19:29 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-18 15:52 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev
Le mercredi 18 mai 2011 à 15:46 +0300, Denys Fedoryshchenko a écrit :
> It is NAS, has ipv6 enabled recently (i am preparing for ipv6), but
> ipv6 is not routed anywhere, maybe just automatically addresses
> appearing on interfaces, including the one looking to customer subnet.
> ppp is ipv4 only.
>
Hmm, it seems we have some inetpeer refcount leak somewhere.
Maybe one (struct rtable)->peer is not released on dst/rtable removal,
or we also leak dst/rtable (and their ->peer inetpeer)
Watch :
grep peer /proc/slabinfo
grep dst /proc/slabinfo
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 15:52 ` Eric Dumazet
@ 2011-05-18 19:29 ` Eric Dumazet
2011-05-19 5:19 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-18 19:29 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev
Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
> Hmm, it seems we have some inetpeer refcount leak somewhere.
>
> Maybe one (struct rtable)->peer is not released on dst/rtable removal,
> or we also leak dst/rtable (and their ->peer inetpeer)
>
> Watch :
>
> grep peer /proc/slabinfo
> grep dst /proc/slabinfo
>
FYI, I started a bisection to find the faulty commit.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-18 19:29 ` Eric Dumazet
@ 2011-05-19 5:19 ` Eric Dumazet
2011-05-19 6:11 ` Denys Fedoryshchenko
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-19 5:19 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev, David Miller
Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
>
> > Hmm, it seems we have some inetpeer refcount leak somewhere.
> >
> > Maybe one (struct rtable)->peer is not released on dst/rtable removal,
> > or we also leak dst/rtable (and their ->peer inetpeer)
> >
> > Watch :
> >
> > grep peer /proc/slabinfo
> > grep dst /proc/slabinfo
> >
>
> FYI, I started a bisection to find the faulty commit.
>
Oh well, of course this came to 2c8cec5c10bced240
(ipv4: Cache learned PMTU information in inetpeer.)
So my method to check if we have a leak might be wrong, since the above
commit let cache full of garbage, and hope that following lookups will
find and evict obsolete dst.
Thats getting difficult :(
Could you please send us
grep . /proc/sys/net/ipv4/route/*
Thanks !
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-19 5:19 ` Eric Dumazet
@ 2011-05-19 6:11 ` Denys Fedoryshchenko
2011-05-19 6:30 ` Eric Dumazet
0 siblings, 1 reply; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-19 6:11 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, David Miller
On Thu, 19 May 2011 07:19:57 +0200, Eric Dumazet wrote:
> Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
>> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
>>
>> > Hmm, it seems we have some inetpeer refcount leak somewhere.
>> >
>> > Maybe one (struct rtable)->peer is not released on dst/rtable
>> removal,
>> > or we also leak dst/rtable (and their ->peer inetpeer)
>> >
>> > Watch :
>> >
>> > grep peer /proc/slabinfo
>> > grep dst /proc/slabinfo
>> >
>>
>> FYI, I started a bisection to find the faulty commit.
>>
>
> Oh well, of course this came to 2c8cec5c10bced240
> (ipv4: Cache learned PMTU information in inetpeer.)
>
> So my method to check if we have a leak might be wrong, since the
> above
> commit let cache full of garbage, and hope that following lookups
> will
> find and evict obsolete dst.
>
> Thats getting difficult :(
>
> Could you please send us
>
> grep . /proc/sys/net/ipv4/route/*
>
> Thanks !
NewNet-PPPoE ~ # grep . /proc/sys/net/ipv4/route/*
/proc/sys/net/ipv4/route/error_burst:5000
/proc/sys/net/ipv4/route/error_cost:1000
grep: /proc/sys/net/ipv4/route/flush: Permission denied
/proc/sys/net/ipv4/route/gc_elasticity:8
/proc/sys/net/ipv4/route/gc_interval:60
/proc/sys/net/ipv4/route/gc_min_interval:0
/proc/sys/net/ipv4/route/gc_min_interval_ms:500
/proc/sys/net/ipv4/route/gc_thresh:32768
/proc/sys/net/ipv4/route/gc_timeout:300
/proc/sys/net/ipv4/route/max_size:524288
/proc/sys/net/ipv4/route/min_adv_mss:256
/proc/sys/net/ipv4/route/min_pmtu:552
/proc/sys/net/ipv4/route/mtu_expires:600
/proc/sys/net/ipv4/route/redirect_load:20
/proc/sys/net/ipv4/route/redirect_number:9
/proc/sys/net/ipv4/route/redirect_silence:20480
I think it is default one.
PMTU is very actual for that, as it is pppoe, and up to 2k interfaces
terminated there.
I don't know, if it matters, but
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
--clamp-mss-to-pmtu
also there.
I can generate and put "ip route ls cache" and any other info.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-19 6:11 ` Denys Fedoryshchenko
@ 2011-05-19 6:30 ` Eric Dumazet
2011-05-19 6:39 ` Denys Fedoryshchenko
0 siblings, 1 reply; 13+ messages in thread
From: Eric Dumazet @ 2011-05-19 6:30 UTC (permalink / raw)
To: Denys Fedoryshchenko; +Cc: netdev, David Miller
Le jeudi 19 mai 2011 à 09:11 +0300, Denys Fedoryshchenko a écrit :
> On Thu, 19 May 2011 07:19:57 +0200, Eric Dumazet wrote:
> > Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
> >> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
> >>
> >> > Hmm, it seems we have some inetpeer refcount leak somewhere.
> >> >
> >> > Maybe one (struct rtable)->peer is not released on dst/rtable
> >> removal,
> >> > or we also leak dst/rtable (and their ->peer inetpeer)
> >> >
> >> > Watch :
> >> >
> >> > grep peer /proc/slabinfo
> >> > grep dst /proc/slabinfo
> >> >
> >>
> >> FYI, I started a bisection to find the faulty commit.
> >>
> >
> > Oh well, of course this came to 2c8cec5c10bced240
> > (ipv4: Cache learned PMTU information in inetpeer.)
> >
> > So my method to check if we have a leak might be wrong, since the
> > above
> > commit let cache full of garbage, and hope that following lookups
> > will
> > find and evict obsolete dst.
> >
> > Thats getting difficult :(
> >
> > Could you please send us
> >
> > grep . /proc/sys/net/ipv4/route/*
> >
> > Thanks !
> NewNet-PPPoE ~ # grep . /proc/sys/net/ipv4/route/*
> /proc/sys/net/ipv4/route/error_burst:5000
> /proc/sys/net/ipv4/route/error_cost:1000
> grep: /proc/sys/net/ipv4/route/flush: Permission denied
> /proc/sys/net/ipv4/route/gc_elasticity:8
> /proc/sys/net/ipv4/route/gc_interval:60
> /proc/sys/net/ipv4/route/gc_min_interval:0
> /proc/sys/net/ipv4/route/gc_min_interval_ms:500
> /proc/sys/net/ipv4/route/gc_thresh:32768
> /proc/sys/net/ipv4/route/gc_timeout:300
> /proc/sys/net/ipv4/route/max_size:524288
> /proc/sys/net/ipv4/route/min_adv_mss:256
> /proc/sys/net/ipv4/route/min_pmtu:552
> /proc/sys/net/ipv4/route/mtu_expires:600
> /proc/sys/net/ipv4/route/redirect_load:20
> /proc/sys/net/ipv4/route/redirect_number:9
> /proc/sys/net/ipv4/route/redirect_silence:20480
>
> I think it is default one.
>
> PMTU is very actual for that, as it is pppoe, and up to 2k interfaces
> terminated there.
>
Yes, and every time an interface is added -> new route added, route
cache is invalidated (we change rt_genid)
> I don't know, if it matters, but
> iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
> --clamp-mss-to-pmtu
> also there.
>
> I can generate and put "ip route ls cache" and any other info.
>
Hmm would you please send :
rtstat -c10 -i1
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11
2011-05-19 6:30 ` Eric Dumazet
@ 2011-05-19 6:39 ` Denys Fedoryshchenko
0 siblings, 0 replies; 13+ messages in thread
From: Denys Fedoryshchenko @ 2011-05-19 6:39 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, David Miller
On Thu, 19 May 2011 08:30:23 +0200, Eric Dumazet wrote:
> Le jeudi 19 mai 2011 à 09:11 +0300, Denys Fedoryshchenko a écrit :
>> On Thu, 19 May 2011 07:19:57 +0200, Eric Dumazet wrote:
>> > Le mercredi 18 mai 2011 à 21:29 +0200, Eric Dumazet a écrit :
>> >> Le mercredi 18 mai 2011 à 17:52 +0200, Eric Dumazet a écrit :
>> >>
>> >> > Hmm, it seems we have some inetpeer refcount leak somewhere.
>> >> >
>> >> > Maybe one (struct rtable)->peer is not released on dst/rtable
>> >> removal,
>> >> > or we also leak dst/rtable (and their ->peer inetpeer)
>> >> >
>> >> > Watch :
>> >> >
>> >> > grep peer /proc/slabinfo
>> >> > grep dst /proc/slabinfo
>> >> >
>> >>
>> >> FYI, I started a bisection to find the faulty commit.
>> >>
>> >
>> > Oh well, of course this came to 2c8cec5c10bced240
>> > (ipv4: Cache learned PMTU information in inetpeer.)
>> >
>> > So my method to check if we have a leak might be wrong, since the
>> > above
>> > commit let cache full of garbage, and hope that following lookups
>> > will
>> > find and evict obsolete dst.
>> >
>> > Thats getting difficult :(
>> >
>> > Could you please send us
>> >
>> > grep . /proc/sys/net/ipv4/route/*
>> >
>> > Thanks !
>> NewNet-PPPoE ~ # grep . /proc/sys/net/ipv4/route/*
>> /proc/sys/net/ipv4/route/error_burst:5000
>> /proc/sys/net/ipv4/route/error_cost:1000
>> grep: /proc/sys/net/ipv4/route/flush: Permission denied
>> /proc/sys/net/ipv4/route/gc_elasticity:8
>> /proc/sys/net/ipv4/route/gc_interval:60
>> /proc/sys/net/ipv4/route/gc_min_interval:0
>> /proc/sys/net/ipv4/route/gc_min_interval_ms:500
>> /proc/sys/net/ipv4/route/gc_thresh:32768
>> /proc/sys/net/ipv4/route/gc_timeout:300
>> /proc/sys/net/ipv4/route/max_size:524288
>> /proc/sys/net/ipv4/route/min_adv_mss:256
>> /proc/sys/net/ipv4/route/min_pmtu:552
>> /proc/sys/net/ipv4/route/mtu_expires:600
>> /proc/sys/net/ipv4/route/redirect_load:20
>> /proc/sys/net/ipv4/route/redirect_number:9
>> /proc/sys/net/ipv4/route/redirect_silence:20480
>>
>> I think it is default one.
>>
>> PMTU is very actual for that, as it is pppoe, and up to 2k
>> interfaces
>> terminated there.
>>
>
> Yes, and every time an interface is added -> new route added, route
> cache is invalidated (we change rt_genid)
If it matters, there is ifb with shaper on it (for shaping from ppp to
world).
>
>> I don't know, if it matters, but
>> iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
>> --clamp-mss-to-pmtu
>> also there.
>>
>> I can generate and put "ip route ls cache" and any other info.
>>
>
> Hmm would you please send :
>
> rtstat -c10 -i1
Note, it is offpeak time now, just 1447 interfaces, peak is after 12
hours
NewNet-PPPoE ~ # ./rtstat -c10 -i1
rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|rt_cache|
entries| in_hit|in_slow_|in_slow_|in_no_ro|
in_brd|in_marti|in_marti|
out_hit|out_slow|out_slow|gc_total|gc_ignor|gc_goal_|gc_dst_o|in_hlist|out_hlis|
| | tot| mc| ute| | an_dst|
an_src| | _tot| _mc| | ed| miss| verflow|
_search|t_search|
2256|355568844|85929285| 1649| 9| 59954| 293|
1460|14423031| 6865540| 0| 0| 0| 0|
0|22719682| 1262044|
3408| 14887| 2117| 0| 0| 1| 1|
0| 761| 159| 0| 0| 0| 0| 0|
1209| 46|
3189| 17185| 5613| 0| 0| 1| 0|
0| 987| 334| 0| 0| 0| 0| 0|
684| 22|
2698| 18312| 3417| 0| 0| 5| 0|
0| 923| 242| 0| 0| 0| 0| 0|
498| 10|
4996| 17268| 3604| 0| 0| 1| 0|
0| 847| 240| 0| 0| 0| 0| 0|
830| 23|
2457| 16439| 4227| 0| 0| 4| 0|
0| 663| 268| 0| 0| 0| 0| 0|
655| 22|
4763| 16895| 3634| 0| 0| 1| 0|
0| 880| 266| 0| 0| 0| 0| 0|
896| 32|
6299| 19169| 2220| 0| 0| 2| 0|
0| 898| 206| 0| 0| 0| 0| 0|
1213| 60|
7511| 20059| 1597| 0| 0| 2| 1|
0| 855| 197| 0| 0| 0| 0| 0|
1917| 54|
9271| 17731| 2919| 0| 0| 0| 0|
0| 855| 223| 0| 0| 0| 0| 0|
1664| 101|
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-05-19 6:39 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-17 22:16 Bug, kernel panic, NULL dereference , cleanup_once / icmp_route_lookup.clone.19.clone / nat , 2.6.39-rc7-git11 Denys Fedoryshchenko
2011-05-18 9:27 ` Denys Fedoryshchenko
2011-05-18 9:37 ` Eric Dumazet
2011-05-18 9:53 ` Denys Fedoryshchenko
2011-05-18 10:05 ` Eric Dumazet
2011-05-18 11:44 ` Eric Dumazet
2011-05-18 12:46 ` Denys Fedoryshchenko
2011-05-18 15:52 ` Eric Dumazet
2011-05-18 19:29 ` Eric Dumazet
2011-05-19 5:19 ` Eric Dumazet
2011-05-19 6:11 ` Denys Fedoryshchenko
2011-05-19 6:30 ` Eric Dumazet
2011-05-19 6:39 ` Denys Fedoryshchenko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.