* kmemleak reports in kernel 3.9.5+
@ 2013-06-10 18:22 Ben Greear
2013-06-10 22:32 ` Catalin Marinas
0 siblings, 1 reply; 8+ messages in thread
From: Ben Greear @ 2013-06-10 18:22 UTC (permalink / raw)
To: Linux Kernel Mailing List, netdev
We had a system go OOM while doing lots of wireless
stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and
I see the entries below. Any idea if these are real or not?
unreferenced object 0xffff880212281c80 (size 128):
comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s)
hex dump (first 32 bytes):
60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d605>] __kmalloc+0xf9/0x122
[<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
[<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
[<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
[<ffffffff8152c992>] ops_init+0xd6/0xf7
[<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
[<ffffffffa016db69>] 0xffffffffa016db69
[<ffffffffa016d109>] 0xffffffffa016d109
[<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
[<ffffffff810f3985>] do_init_module+0x44/0x18f
[<ffffffff810f5da7>] load_module+0x14d1/0x168e
[<ffffffff810f6114>] sys_init_module+0xfd/0x101
[<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
unreferenced object 0xffff880218ed92b0 (size 40):
comm "gnome-session", pid 934, jiffies 4294693971 (age 1148.267s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 97 41 8b 01 88 ff ff ..........A.....
02 00 00 00 00 00 00 00 20 44 46 08 00 ea ff ff ........ DF.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81175d98>] unlink_anon_vmas+0x9f/0x154
unreferenced object 0xffff88021716f870 (size 40):
comm "ip", pid 3146, jiffies 4294741842 (age 1100.453s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 c0 dc 37 ee 01 88 ff ff ..........7.....
01 00 00 00 00 00 00 00 88 35 0c e4 01 88 ff ff .........5......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810a70df>] init_timer_key+0x2e/0xb3
[<ffffffff81521dcb>] sock_init_data+0x69/0x1ed
[<ffffffff815551a5>] __netlink_create+0x4f/0xb5
[<ffffffff815552fd>] netlink_create+0xf2/0x148
[<ffffffff8151ec54>] __sock_create+0x135/0x1be
[<ffffffff8151ed33>] sock_create+0x30/0x32
[<ffffffff81520517>] sys_socket+0x2f/0x98
[<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff8801ebeb4450 (size 40):
comm "bash", pid 3200, jiffies 4294742742 (age 1099.584s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 c0 e5 91 f1 01 88 ff ff ................
01 00 00 00 00 00 00 00 40 4d 3d e4 01 88 ff ff ........@M=.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81175d98>] unlink_anon_vmas+0x9f/0x154
unreferenced object 0xffff8801f191e5c0 (size 40):
comm "clock-applet", pid 3100, jiffies 4294742818 (age 1099.508s)
hex dump (first 32 bytes):
50 44 eb eb 01 88 ff ff 70 f8 bb 9a 01 88 ff ff PD......p.......
01 00 00 00 00 00 00 00 58 45 3d e4 01 88 ff ff ........XE=.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff8801ed2468a0 (size 40):
comm "wnck-applet", pid 3093, jiffies 4294742922 (age 1099.404s)
hex dump (first 32 bytes):
c0 3c b3 f1 01 88 ff ff 80 19 6a 82 ff ff ff ff .<........j.....
01 00 00 00 00 00 00 00 c8 74 31 e4 01 88 ff ff .........t1.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118daf5>] kfree+0xdd/0x158
[<ffffffff815262b9>] skb_free_head+0x67/0x69
unreferenced object 0xffff8801f193cb80 (size 40):
comm "dbus-daemon", pid 1377, jiffies 4294743052 (age 1099.304s)
hex dump (first 32 bytes):
60 0e 39 ee 01 88 ff ff 70 e1 75 c6 01 88 ff ff `.9.....p.u.....
01 00 00 00 00 00 00 00 a0 2d 01 e5 01 88 ff ff .........-......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff8801f1b33cc0 (size 40):
comm "wpa_cli", pid 3221, jiffies 4294743135 (age 1099.221s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 a0 68 24 ed 01 88 ff ff .........h$.....
01 00 00 00 00 00 00 00 b0 7c 31 e4 01 88 ff ff .........|1.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119be61>] put_filp+0x32/0x36
[<ffffffff811a6bf1>] path_openat+0x340/0x379
[<ffffffff811a6d39>] do_filp_open+0x3d/0x89
[<ffffffff81199346>] do_sys_open+0x72/0x104
[<ffffffff8119940f>] sys_open+0x21/0x23
unreferenced object 0xffff8801ee390e60 (size 40):
comm "evolution-alarm", pid 3011, jiffies 4294743335 (age 1099.023s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 80 cb 93 f1 01 88 ff ff ................
01 00 00 00 00 00 00 00 b8 25 01 e5 01 88 ff ff .........%......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff880216e7ca10 (size 40):
comm "metacity", pid 2810, jiffies 4294744465 (age 1097.923s)
hex dump (first 32 bytes):
30 fe 97 ee 01 88 ff ff 80 6b 5c 82 ff ff ff ff 0........k\.....
01 00 00 00 00 00 00 00 70 3d 15 e5 01 88 ff ff ........p=......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81527493>] __kfree_skb+0x7d/0x82
unreferenced object 0xffff8801ee97fe30 (size 40):
comm "pool", pid 3081, jiffies 4294744513 (age 1097.875s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 10 ca e7 16 02 88 ff ff ................
01 00 00 00 00 00 00 00 88 35 15 e5 01 88 ff ff .........5......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-10 18:22 kmemleak reports in kernel 3.9.5+ Ben Greear
@ 2013-06-10 22:32 ` Catalin Marinas
2013-06-11 18:36 ` Ben Greear
2013-06-11 19:52 ` Ben Greear
0 siblings, 2 replies; 8+ messages in thread
From: Catalin Marinas @ 2013-06-10 22:32 UTC (permalink / raw)
To: Ben Greear; +Cc: Linux Kernel Mailing List, netdev
On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
> We had a system go OOM while doing lots of wireless
> stations. (System had 8GB of RAM, so I suspect a leak).
>
> I enabled kmemleak in a 3.9.5 (plus some local patches) and
> I see the entries below. Any idea if these are real or not?
>
> unreferenced object 0xffff880212281c80 (size 128):
> comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s)
> hex dump (first 32 bytes):
> 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
> 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> backtrace:
> [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
> [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
> [<ffffffff8118d605>] __kmalloc+0xf9/0x122
> [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
> [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
> [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
> [<ffffffff8152c992>] ops_init+0xd6/0xf7
> [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
> [<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
> [<ffffffffa016db69>] 0xffffffffa016db69
> [<ffffffffa016d109>] 0xffffffffa016d109
> [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
> [<ffffffff810f3985>] do_init_module+0x44/0x18f
> [<ffffffff810f5da7>] load_module+0x14d1/0x168e
> [<ffffffff810f6114>] sys_init_module+0xfd/0x101
> [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
No idea yet. You can try:
echo clear > /sys/kernel/debug/kmemleak
and see if there are more appearing after. All seem to have a common
allocation path via debug_object_activate -> ... ->
rcuhead_fixup_activate -> ... -> __debug_object_init.
--
Catalin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-10 22:32 ` Catalin Marinas
@ 2013-06-11 18:36 ` Ben Greear
2013-06-11 19:52 ` Ben Greear
1 sibling, 0 replies; 8+ messages in thread
From: Ben Greear @ 2013-06-11 18:36 UTC (permalink / raw)
To: Catalin Marinas; +Cc: Linux Kernel Mailing List, netdev
On 06/10/2013 03:32 PM, Catalin Marinas wrote:
> On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
>> We had a system go OOM while doing lots of wireless
>> stations. (System had 8GB of RAM, so I suspect a leak).
>>
>> I enabled kmemleak in a 3.9.5 (plus some local patches) and
>> I see the entries below. Any idea if these are real or not?
>>
>> unreferenced object 0xffff880212281c80 (size 128):
>> comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s)
>> hex dump (first 32 bytes):
>> 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
>> 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> backtrace:
>> [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
>> [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
>> [<ffffffff8118d605>] __kmalloc+0xf9/0x122
>> [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
>> [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
>> [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
>> [<ffffffff8152c992>] ops_init+0xd6/0xf7
>> [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
>> [<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
>> [<ffffffffa016db69>] 0xffffffffa016db69
>> [<ffffffffa016d109>] 0xffffffffa016d109
>> [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
>> [<ffffffff810f3985>] do_init_module+0x44/0x18f
>> [<ffffffff810f5da7>] load_module+0x14d1/0x168e
>> [<ffffffff810f6114>] sys_init_module+0xfd/0x101
>> [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
>
> No idea yet. You can try:
>
> echo clear > /sys/kernel/debug/kmemleak
>
> and see if there are more appearing after. All seem to have a common
> allocation path via debug_object_activate -> ... ->
> rcuhead_fixup_activate -> ... -> __debug_object_init.
I let the system run overnight, and there are a good many more
kmemleak reports this morning. I have not yet tried the clear
option.
I have SLUB debugging enabled, so that is probably explains
the debug stuff in the stacks.
Here are some more that may be of interest.
unreferenced object 0xffff88010b719700 (size 40):
comm "gua", pid 11301, jiffies 4352877266 (age 29217.682s)
hex dump (first 32 bytes):
00 e0 10 08 01 88 ff ff a0 68 1e 13 01 88 ff ff .........h......
01 00 00 00 00 00 00 00 70 3d 56 13 01 88 ff ff ........p=V.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c4c>] debug_object_init_on_stack+0x17/0x19
[<ffffffff810bb5fa>] hrtimer_init_on_stack+0x27/0x75
[<ffffffff815ed663>] schedule_hrtimeout_range_clock+0x67/0x113
[<ffffffff815ed722>] schedule_hrtimeout_range+0x13/0x15
[<ffffffff811aa1fa>] poll_schedule_timeout+0x48/0x64
[<ffffffff811aaff6>] do_select+0x519/0x550
[<ffffffff811df227>] compat_core_sys_select+0x1c7/0x25a
[<ffffffff811df526>] compat_sys_select+0x99/0xc1
[<ffffffff815f7c4c>] sysenter_dispatch+0x7/0x26
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff880111d33420 (size 40):
comm "ps", pid 25114, jiffies 4352875696 (age 29219.150s)
hex dump (first 32 bytes):
c0 3c 68 0b 01 88 ff ff 70 41 ec 11 01 88 ff ff .<h.....pA......
01 00 00 00 00 00 00 00 e0 6c 5a 12 01 88 ff ff .........lZ.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119c05a>] __fput+0x1bb/0x1db
[<ffffffff8119c0ca>] ____fput+0xe/0x10
[<ffffffff810b48d5>] task_work_run+0x85/0xb0
[<ffffffff810106c6>] do_notify_resume+0x6d/0x80
[<ffffffff815f68d2>] int_signal+0x12/0x17
unreferenced object 0xffff88010cc92e60 (size 40):
comm "updatedb", pid 24952, jiffies 4352871287 (age 29223.220s)
hex dump (first 32 bytes):
00 60 16 0e 01 88 ff ff d0 af 19 10 01 88 ff ff .`..............
01 00 00 00 00 00 00 00 a0 2d 39 0e 01 88 ff ff .........-9.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b296c>] __init_work+0x27/0x29
[<ffffffff81241371>] ext4_alloc_inode+0x173/0x1cb
[<ffffffff811b0db4>] alloc_inode+0x1d/0x78
[<ffffffff811b0e7c>] iget_locked+0x6d/0x12f
[<ffffffff81229817>] ext4_iget+0x2c/0x932
[<ffffffff8122f485>] ext4_lookup+0xd2/0x12a
[<ffffffff811a3556>] lookup_real+0x2c/0x47
[<ffffffff811a3e0c>] __lookup_hash+0x33/0x3c
[<ffffffff811a5897>] walk_component+0x73/0x171
[<ffffffff811a6e2c>] path_lookupat+0xa7/0x304
[<ffffffff811a70b4>] filename_lookup+0x2b/0x82
(I sent a similar one to the ext4 mailing list already.)
unreferenced object 0xffff8801c6d2eb80 (size 40):
comm "softirq", pid 0, jiffies 4296641468 (age 85451.733s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 70 01 6e 7b 01 88 ff ff ........p.n{....
02 00 00 00 00 00 00 00 58 54 24 ca 01 88 ff ff ........XT$.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810a70df>] init_timer_key+0x2e/0xb3
[<ffffffffa0128e4d>] ipv6_add_addr+0x19f/0x3bc [ipv6]
[<ffffffffa012c885>] addrconf_add_linklocal+0x4c/0x7f [ipv6]
[<ffffffffa012d06a>] addrconf_notify+0x773/0x95e [ipv6]
[<ffffffff815f30d6>] notifier_call_chain+0x37/0x63
[<ffffffff810bc994>] raw_notifier_call_chain+0x14/0x16
[<ffffffff815333ae>] call_netdevice_notifiers+0x4a/0x4f
[<ffffffff8153404d>] netdev_state_change+0x27/0x3a
[<ffffffff815418ef>] linkwatch_do_dev+0x46/0x54
[<ffffffff81541b0e>] __linkwatch_run_queue+0x180/0x1ca
[<ffffffff81541b7d>] linkwatch_event+0x25/0x2c
unreferenced object 0xffff8801c8e41e78 (size 192):
comm "kworker/u:2", pid 157, jiffies 4295509873 (age 86582.869s)
hex dump (first 32 bytes):
41 0d 00 30 02 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b A..0....kkkkkkkk
6b 6b 6b 6b 6b 6b 6b 6b 69 00 00 00 00 0c 2e 32 kkkkkkkki......2
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d605>] __kmalloc+0xf9/0x122
[<ffffffffa027cb27>] cfg80211_inform_bss_frame+0x114/0x1f8 [cfg80211]
[<ffffffffa03d6865>] ieee80211_bss_info_update+0x66/0x21f [mac80211]
[<ffffffffa040aec6>] ieee80211_rx_bss_info+0x12f/0x1ca [mac80211]
[<ffffffffa040b017>] ieee80211_rx_mgmt_probe_resp+0xb6/0x197 [mac80211]
[<ffffffffa040e8a3>] ieee80211_sta_rx_queued_mgmt+0xdd/0x60e [mac80211]
[<ffffffffa03df0ee>] ieee80211_iface_work+0x238/0x2cc [mac80211]
[<ffffffff810b0cd3>] process_one_work+0x292/0x42e
[<ffffffff810b36af>] worker_thread+0x14f/0x264
[<ffffffff810b7bea>] kthread+0xc7/0xcf
[<ffffffff815f64ec>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
(I plan to investigate the one above...I have some understanding of
the wifi code, and have some patches of my own applied to that code.)
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-10 22:32 ` Catalin Marinas
2013-06-11 18:36 ` Ben Greear
@ 2013-06-11 19:52 ` Ben Greear
2013-06-12 0:28 ` Ben Greear
2013-06-13 15:48 ` Catalin Marinas
1 sibling, 2 replies; 8+ messages in thread
From: Ben Greear @ 2013-06-11 19:52 UTC (permalink / raw)
To: Catalin Marinas; +Cc: Linux Kernel Mailing List, netdev
On 06/10/2013 03:32 PM, Catalin Marinas wrote:
> On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
>> We had a system go OOM while doing lots of wireless
>> stations. (System had 8GB of RAM, so I suspect a leak).
>>
>> I enabled kmemleak in a 3.9.5 (plus some local patches) and
>> I see the entries below. Any idea if these are real or not?
>>
>> unreferenced object 0xffff880212281c80 (size 128):
>> comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s)
>> hex dump (first 32 bytes):
>> 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
>> 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> backtrace:
>> [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
>> [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
>> [<ffffffff8118d605>] __kmalloc+0xf9/0x122
>> [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
>> [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
>> [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
>> [<ffffffff8152c992>] ops_init+0xd6/0xf7
>> [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
>> [<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
>> [<ffffffffa016db69>] 0xffffffffa016db69
>> [<ffffffffa016d109>] 0xffffffffa016d109
>> [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
>> [<ffffffff810f3985>] do_init_module+0x44/0x18f
>> [<ffffffff810f5da7>] load_module+0x14d1/0x168e
>> [<ffffffff810f6114>] sys_init_module+0xfd/0x101
>> [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
>
> No idea yet. You can try:
>
> echo clear > /sys/kernel/debug/kmemleak
>
> and see if there are more appearing after. All seem to have a common
> allocation path via debug_object_activate -> ... ->
> rcuhead_fixup_activate -> ... -> __debug_object_init.
I tried the command below, and it printed out quite a few things.
I'll try building a kernel without the extra SLUB debugging
to see if that helps.
Also, I read the kmemleak.txt documentation, but a question remains:
If I enable kmemleak at compile time, but disable it at boot
time using kmemleak=off, is there any significant runtime overhead?
[root@LEC2220-1 ~]# echo clear > /debug/kmemleak;sleep 60;echo scan > /debug/kmemleak; cat /debug/kmemleak
unreferenced object 0xffff88021867e450 (size 40):
comm "chrony-helper", pid 1138, jiffies 4294699268 (age 91773.781s)
hex dump (first 32 bytes):
d0 cf e5 18 02 88 ff ff 80 0b be 19 02 88 ff ff ................
01 00 00 00 00 00 00 00 f0 34 a3 12 02 88 ff ff .........4......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811acb71>] __d_free+0x56/0x5b
unreferenced object 0xffff880211893b50 (size 40):
comm "nmcli", pid 1178, jiffies 4294699660 (age 91773.390s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 f0 ac bd 19 02 88 ff ff ................
01 00 00 00 00 00 00 00 e0 6c 6e 01 02 88 ff ff .........ln.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119c05a>] __fput+0x1bb/0x1db
[<ffffffff8119c0ca>] ____fput+0xe/0x10
[<ffffffff810b48d5>] task_work_run+0x85/0xb0
[<ffffffff8109ccc1>] do_exit+0x3c9/0x978
[<ffffffff8109d2f3>] do_group_exit+0x83/0xae
....
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-11 19:52 ` Ben Greear
@ 2013-06-12 0:28 ` Ben Greear
2013-06-13 15:50 ` Catalin Marinas
2013-06-13 15:48 ` Catalin Marinas
1 sibling, 1 reply; 8+ messages in thread
From: Ben Greear @ 2013-06-12 0:28 UTC (permalink / raw)
To: Catalin Marinas; +Cc: Linux Kernel Mailing List, netdev
On 06/11/2013 12:52 PM, Ben Greear wrote:
> On 06/10/2013 03:32 PM, Catalin Marinas wrote:
>> On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
>>> We had a system go OOM while doing lots of wireless
>>> stations. (System had 8GB of RAM, so I suspect a leak).
>>>
>>> I enabled kmemleak in a 3.9.5 (plus some local patches) and
>>> I see the entries below. Any idea if these are real or not?
Most of this went away when I disabled SLUB debugging and other
kernel hacking options. The wifi cfg80211_inform_bss_frame
remains, however. I'll go dig some more on that tomorrow...didn't
see anything obvious at first glance.
But, perhaps there could be some improvements to
kmemleak to make it deal better with the various kernel
debugging features?
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-11 19:52 ` Ben Greear
2013-06-12 0:28 ` Ben Greear
@ 2013-06-13 15:48 ` Catalin Marinas
1 sibling, 0 replies; 8+ messages in thread
From: Catalin Marinas @ 2013-06-13 15:48 UTC (permalink / raw)
To: Ben Greear; +Cc: Linux Kernel Mailing List, netdev
On Tue, Jun 11, 2013 at 08:52:41PM +0100, Ben Greear wrote:
> On 06/10/2013 03:32 PM, Catalin Marinas wrote:
> > On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
> >> We had a system go OOM while doing lots of wireless
> >> stations. (System had 8GB of RAM, so I suspect a leak).
> >>
> >> I enabled kmemleak in a 3.9.5 (plus some local patches) and
> >> I see the entries below. Any idea if these are real or not?
> >>
> >> unreferenced object 0xffff880212281c80 (size 128):
> >> comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s)
> >> hex dump (first 32 bytes):
> >> 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
> >> 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> >> backtrace:
> >> [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
> >> [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
> >> [<ffffffff8118d605>] __kmalloc+0xf9/0x122
> >> [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
> >> [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
> >> [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
> >> [<ffffffff8152c992>] ops_init+0xd6/0xf7
> >> [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
> >> [<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
> >> [<ffffffffa016db69>] 0xffffffffa016db69
> >> [<ffffffffa016d109>] 0xffffffffa016d109
> >> [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
> >> [<ffffffff810f3985>] do_init_module+0x44/0x18f
> >> [<ffffffff810f5da7>] load_module+0x14d1/0x168e
> >> [<ffffffff810f6114>] sys_init_module+0xfd/0x101
> >> [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
> >
> > No idea yet. You can try:
> >
> > echo clear > /sys/kernel/debug/kmemleak
> >
> > and see if there are more appearing after. All seem to have a common
> > allocation path via debug_object_activate -> ... ->
> > rcuhead_fixup_activate -> ... -> __debug_object_init.
>
> I tried the command below, and it printed out quite a few things.
Can you send me your .config file? I can't reproduce this.
> Also, I read the kmemleak.txt documentation, but a question remains:
>
> If I enable kmemleak at compile time, but disable it at boot
> time using kmemleak=off, is there any significant runtime overhead?
You still get the callback into kmemleak for each allocation/freeing but
it doesn't walk the stack for the backtrace and it doesn't scan the
memory either. I would say the overhead is very small, probably
unnoticeable if you already have other debug options enabled.
--
Catalin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-12 0:28 ` Ben Greear
@ 2013-06-13 15:50 ` Catalin Marinas
2013-06-17 22:45 ` Ben Greear
0 siblings, 1 reply; 8+ messages in thread
From: Catalin Marinas @ 2013-06-13 15:50 UTC (permalink / raw)
To: Ben Greear; +Cc: Linux Kernel Mailing List, netdev
On Wed, Jun 12, 2013 at 01:28:13AM +0100, Ben Greear wrote:
> On 06/11/2013 12:52 PM, Ben Greear wrote:
> > On 06/10/2013 03:32 PM, Catalin Marinas wrote:
> >> On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
> >>> We had a system go OOM while doing lots of wireless
> >>> stations. (System had 8GB of RAM, so I suspect a leak).
> >>>
> >>> I enabled kmemleak in a 3.9.5 (plus some local patches) and
> >>> I see the entries below. Any idea if these are real or not?
>
> Most of this went away when I disabled SLUB debugging and other
> kernel hacking options. The wifi cfg80211_inform_bss_frame
> remains, however. I'll go dig some more on that tomorrow...didn't
> see anything obvious at first glance.
>
> But, perhaps there could be some improvements to
> kmemleak to make it deal better with the various kernel
> debugging features?
That's unrelated to the debugging features. Kmemleak cannot find
pointers to the allocated objects. They could be real leaks or it simply
doesn't scan the right memory where such pointers are stored. The debug
objects are stored in a list with the head as static memory, so it
should be scanned.
--
Catalin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: kmemleak reports in kernel 3.9.5+
2013-06-13 15:50 ` Catalin Marinas
@ 2013-06-17 22:45 ` Ben Greear
0 siblings, 0 replies; 8+ messages in thread
From: Ben Greear @ 2013-06-17 22:45 UTC (permalink / raw)
To: Catalin Marinas; +Cc: Linux Kernel Mailing List, netdev
On 06/13/2013 08:50 AM, Catalin Marinas wrote:
> On Wed, Jun 12, 2013 at 01:28:13AM +0100, Ben Greear wrote:
>> On 06/11/2013 12:52 PM, Ben Greear wrote:
>>> On 06/10/2013 03:32 PM, Catalin Marinas wrote:
>>>> On 10 June 2013 19:22, Ben Greear <greearb@candelatech.com> wrote:
>>>>> We had a system go OOM while doing lots of wireless
>>>>> stations. (System had 8GB of RAM, so I suspect a leak).
>>>>>
>>>>> I enabled kmemleak in a 3.9.5 (plus some local patches) and
>>>>> I see the entries below. Any idea if these are real or not?
>>
>> Most of this went away when I disabled SLUB debugging and other
>> kernel hacking options. The wifi cfg80211_inform_bss_frame
>> remains, however. I'll go dig some more on that tomorrow...didn't
>> see anything obvious at first glance.
>>
>> But, perhaps there could be some improvements to
>> kmemleak to make it deal better with the various kernel
>> debugging features?
>
> That's unrelated to the debugging features. Kmemleak cannot find
> pointers to the allocated objects. They could be real leaks or it simply
> doesn't scan the right memory where such pointers are stored. The debug
> objects are stored in a list with the head as static memory, so it
> should be scanned.
I re-ran a similar test just now, but I had added an explicit reference
for each of the items that I know this test causes to leak (I add them
to a global list.)
This means that memleak would no longer show them as leaked, and that
also cleaned up all of the strange leaks reported before.
So, I think kmemleak is at least mostly working properly, even with lots
of debugging options.
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-06-17 22:45 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-06-10 18:22 kmemleak reports in kernel 3.9.5+ Ben Greear
2013-06-10 22:32 ` Catalin Marinas
2013-06-11 18:36 ` Ben Greear
2013-06-11 19:52 ` Ben Greear
2013-06-12 0:28 ` Ben Greear
2013-06-13 15:50 ` Catalin Marinas
2013-06-17 22:45 ` Ben Greear
2013-06-13 15:48 ` Catalin Marinas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.