* Re: [kmemleak] unreferenced object 0xcd9c1a80 (size 192):
2018-02-12 5:47 ` Yonghong Song
@ 2018-02-12 8:28 ` Daniel Borkmann
2018-02-12 15:55 ` Alexei Starovoitov
2018-02-12 19:26 ` Mathieu Malaterre
1 sibling, 1 reply; 8+ messages in thread
From: Daniel Borkmann @ 2018-02-12 8:28 UTC (permalink / raw)
To: Yonghong Song, Mathieu Malaterre, Alexei Starovoitov
Cc: Alexei Starovoitov, LKML
On 02/12/2018 06:47 AM, Yonghong Song wrote:
> On 2/11/18 11:18 AM, Mathieu Malaterre wrote:
>> On Sun, Feb 11, 2018 at 5:54 PM, Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>> On Sun, Feb 11, 2018 at 7:24 AM, Mathieu Malaterre <malat@debian.org> wrote:
>>>> Alexei,
>>>>
>>>> Could you please comment on why I am seeing those memleaks being
>>>> reported on my ppc32 system ? Should they be marked as false positive
>>>> ?
>>>>
>>>> System is Mac Mini G4, git/master (4.15.0+), ppc.
>>>>
>>>> Thanks for your time
>>>>
>>>> $ dmesg
>>>> ...
>>>> [ 1281.504173] kmemleak: 36 new suspected memory leaks (see
>>>> /sys/kernel/debug/kmemleak)
>>>>
>>>> Where:
>>>>
>>>> # cat /sys/kernel/debug/kmemleak
>>>> unreferenced object 0xdee25000 (size 192):
>>>> comm "systemd", pid 1, jiffies 4294894348 (age 1438.580s)
>>>> hex dump (first 32 bytes):
>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 0c .V/.............
>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>> backtrace:
>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>> unreferenced object 0xdee25900 (size 192):
>>>> comm "systemd", pid 1, jiffies 4294894540 (age 1437.812s)
>>>> hex dump (first 32 bytes):
>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 08 .V/.............
>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>> backtrace:
>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>
>>> hmm. looks real. Is there a reproducer?
>>> Yonghong, lpm map not cleaning after itself?
>>
>> Not really. I simply boot up my machine and wait for the first kmemleak scan.
>
> I am not able to reproduce the issue. Tried with latest net-next on FC26 with kmemleak on. I only got this one after bootup,
> 'cat /sys/kernel/debug/kmemleak' or
> 'echo scan > /sys/kernel/debug/kmemleak
> cat /sys/kernel/debug/kmemleak':
>
> unreferenced object 0xffff99701a7386e0 (size 32):
> comm "mount", pid 1856, jiffies 4294669263 (age 98.440s)
> hex dump (first 32 bytes):
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> backtrace:
> [<000000004668ec00>] security_sb_parse_opts_str+0x36/0x50
> [<00000000a9807d2b>] parse_security_options+0x3d/0x60
> [<00000000cc1e1d58>] btrfs_mount_root+0x139/0x720
> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> [<0000000093ae5db7>] btrfs_mount+0x184/0x914
> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> [<000000003b67b9fc>] do_mount+0x5b9/0xc70
> [<00000000de4073a0>] SyS_mount+0x80/0xd0
> [<00000000fc5a968a>] do_syscall_64+0x5d/0x110
> [<000000003d61f5fc>] entry_SYSCALL_64_after_hwframe+0x21/0x86
> [<00000000458a6ffa>] 0xffffffffffffffff
>
> Not sure whether the above is a true issue or not.
>
> However, by inspecting the code, I do find the trie_free in lpm_trie.c
> may have missed freeing the trie memory.
>
> The change likes below should work:
> -bash-4.2$ git diff
> diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c
> index 7b469d1..cecb259 100644
> --- a/kernel/bpf/lpm_trie.c
> +++ b/kernel/bpf/lpm_trie.c
> @@ -589,6 +589,7 @@ static void trie_free(struct bpf_map *map)
>
> unlock:
> raw_spin_unlock(&trie->lock);
> + kfree(trie);
> }
>
> static int trie_get_next_key(struct bpf_map *map, void *_key, void *_next_key)
> -bash-4.2$
>
> Will propose a formal patch for this soon.
Agree, good catch, and I also think that this is the issue, since this
is what kmemleak reports in terms of size (192):
struct lpm_trie {
struct bpf_map map; /* 0 128 */
/* --- cacheline 2 boundary (128 bytes) --- */
struct lpm_trie_node * root; /* 128 8 */
size_t n_entries; /* 136 8 */
size_t max_prefixlen; /* 144 8 */
size_t data_size; /* 152 8 */
raw_spinlock_t lock; /* 160 4 */
/* size: 192, cachelines: 3, members: 6 */
/* padding: 28 */
};
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [kmemleak] unreferenced object 0xcd9c1a80 (size 192):
2018-02-12 8:28 ` Daniel Borkmann
@ 2018-02-12 15:55 ` Alexei Starovoitov
2018-02-12 17:00 ` Yonghong Song
0 siblings, 1 reply; 8+ messages in thread
From: Alexei Starovoitov @ 2018-02-12 15:55 UTC (permalink / raw)
To: Daniel Borkmann
Cc: Yonghong Song, Mathieu Malaterre, Alexei Starovoitov, LKML
On Mon, Feb 12, 2018 at 09:28:33AM +0100, Daniel Borkmann wrote:
> On 02/12/2018 06:47 AM, Yonghong Song wrote:
> > On 2/11/18 11:18 AM, Mathieu Malaterre wrote:
> >> On Sun, Feb 11, 2018 at 5:54 PM, Alexei Starovoitov
> >> <alexei.starovoitov@gmail.com> wrote:
> >>> On Sun, Feb 11, 2018 at 7:24 AM, Mathieu Malaterre <malat@debian.org> wrote:
> >>>> Alexei,
> >>>>
> >>>> Could you please comment on why I am seeing those memleaks being
> >>>> reported on my ppc32 system ? Should they be marked as false positive
> >>>> ?
> >>>>
> >>>> System is Mac Mini G4, git/master (4.15.0+), ppc.
> >>>>
> >>>> Thanks for your time
> >>>>
> >>>> $ dmesg
> >>>> ...
> >>>> [ 1281.504173] kmemleak: 36 new suspected memory leaks (see
> >>>> /sys/kernel/debug/kmemleak)
> >>>>
> >>>> Where:
> >>>>
> >>>> # cat /sys/kernel/debug/kmemleak
> >>>> unreferenced object 0xdee25000 (size 192):
> >>>> comm "systemd", pid 1, jiffies 4294894348 (age 1438.580s)
> >>>> hex dump (first 32 bytes):
> >>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 0c .V/.............
> >>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
> >>>> backtrace:
> >>>> [<6c69baf5>] trie_alloc+0xb0/0x150
> >>>> [<fa093284>] SyS_bpf+0x288/0x1458
> >>>> [<82182f53>] ret_from_syscall+0x0/0x38
> >>>> unreferenced object 0xdee25900 (size 192):
> >>>> comm "systemd", pid 1, jiffies 4294894540 (age 1437.812s)
> >>>> hex dump (first 32 bytes):
> >>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 08 .V/.............
> >>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
> >>>> backtrace:
> >>>> [<6c69baf5>] trie_alloc+0xb0/0x150
> >>>> [<fa093284>] SyS_bpf+0x288/0x1458
> >>>> [<82182f53>] ret_from_syscall+0x0/0x38
> >>>
> >>> hmm. looks real. Is there a reproducer?
> >>> Yonghong, lpm map not cleaning after itself?
> >>
> >> Not really. I simply boot up my machine and wait for the first kmemleak scan.
> >
> > I am not able to reproduce the issue. Tried with latest net-next on FC26 with kmemleak on. I only got this one after bootup,
> > 'cat /sys/kernel/debug/kmemleak' or
> > 'echo scan > /sys/kernel/debug/kmemleak
> > cat /sys/kernel/debug/kmemleak':
> >
> > unreferenced object 0xffff99701a7386e0 (size 32):
> > comm "mount", pid 1856, jiffies 4294669263 (age 98.440s)
> > hex dump (first 32 bytes):
> > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> > backtrace:
> > [<000000004668ec00>] security_sb_parse_opts_str+0x36/0x50
> > [<00000000a9807d2b>] parse_security_options+0x3d/0x60
> > [<00000000cc1e1d58>] btrfs_mount_root+0x139/0x720
> > [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> > [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> > [<0000000093ae5db7>] btrfs_mount+0x184/0x914
> > [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> > [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> > [<000000003b67b9fc>] do_mount+0x5b9/0xc70
> > [<00000000de4073a0>] SyS_mount+0x80/0xd0
> > [<00000000fc5a968a>] do_syscall_64+0x5d/0x110
> > [<000000003d61f5fc>] entry_SYSCALL_64_after_hwframe+0x21/0x86
> > [<00000000458a6ffa>] 0xffffffffffffffff
> >
> > Not sure whether the above is a true issue or not.
> >
> > However, by inspecting the code, I do find the trie_free in lpm_trie.c
> > may have missed freeing the trie memory.
> >
> > The change likes below should work:
> > -bash-4.2$ git diff
> > diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c
> > index 7b469d1..cecb259 100644
> > --- a/kernel/bpf/lpm_trie.c
> > +++ b/kernel/bpf/lpm_trie.c
> > @@ -589,6 +589,7 @@ static void trie_free(struct bpf_map *map)
> >
> > unlock:
> > raw_spin_unlock(&trie->lock);
> > + kfree(trie);
also looks like trie_free() is missing
synchronize_rcu() + rcu_barrier()
it doesn't wait for parallel lookup/update/delete to complete
before freeing the elements.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kmemleak] unreferenced object 0xcd9c1a80 (size 192):
2018-02-12 15:55 ` Alexei Starovoitov
@ 2018-02-12 17:00 ` Yonghong Song
0 siblings, 0 replies; 8+ messages in thread
From: Yonghong Song @ 2018-02-12 17:00 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann
Cc: Mathieu Malaterre, Alexei Starovoitov, LKML
On 2/12/18 7:55 AM, Alexei Starovoitov wrote:
> On Mon, Feb 12, 2018 at 09:28:33AM +0100, Daniel Borkmann wrote:
>> On 02/12/2018 06:47 AM, Yonghong Song wrote:
>>> On 2/11/18 11:18 AM, Mathieu Malaterre wrote:
>>>> On Sun, Feb 11, 2018 at 5:54 PM, Alexei Starovoitov
>>>> <alexei.starovoitov@gmail.com> wrote:
>>>>> On Sun, Feb 11, 2018 at 7:24 AM, Mathieu Malaterre <malat@debian.org> wrote:
>>>>>> Alexei,
>>>>>>
>>>>>> Could you please comment on why I am seeing those memleaks being
>>>>>> reported on my ppc32 system ? Should they be marked as false positive
>>>>>> ?
>>>>>>
>>>>>> System is Mac Mini G4, git/master (4.15.0+), ppc.
>>>>>>
>>>>>> Thanks for your time
>>>>>>
>>>>>> $ dmesg
>>>>>> ...
>>>>>> [ 1281.504173] kmemleak: 36 new suspected memory leaks (see
>>>>>> /sys/kernel/debug/kmemleak)
>>>>>>
>>>>>> Where:
>>>>>>
>>>>>> # cat /sys/kernel/debug/kmemleak
>>>>>> unreferenced object 0xdee25000 (size 192):
>>>>>> comm "systemd", pid 1, jiffies 4294894348 (age 1438.580s)
>>>>>> hex dump (first 32 bytes):
>>>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 0c .V/.............
>>>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>>>> backtrace:
>>>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>>>> unreferenced object 0xdee25900 (size 192):
>>>>>> comm "systemd", pid 1, jiffies 4294894540 (age 1437.812s)
>>>>>> hex dump (first 32 bytes):
>>>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 08 .V/.............
>>>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>>>> backtrace:
>>>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>>>
>>>>> hmm. looks real. Is there a reproducer?
>>>>> Yonghong, lpm map not cleaning after itself?
>>>>
>>>> Not really. I simply boot up my machine and wait for the first kmemleak scan.
>>>
>>> I am not able to reproduce the issue. Tried with latest net-next on FC26 with kmemleak on. I only got this one after bootup,
>>> 'cat /sys/kernel/debug/kmemleak' or
>>> 'echo scan > /sys/kernel/debug/kmemleak
>>> cat /sys/kernel/debug/kmemleak':
>>>
>>> unreferenced object 0xffff99701a7386e0 (size 32):
>>> comm "mount", pid 1856, jiffies 4294669263 (age 98.440s)
>>> hex dump (first 32 bytes):
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>>> backtrace:
>>> [<000000004668ec00>] security_sb_parse_opts_str+0x36/0x50
>>> [<00000000a9807d2b>] parse_security_options+0x3d/0x60
>>> [<00000000cc1e1d58>] btrfs_mount_root+0x139/0x720
>>> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
>>> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
>>> [<0000000093ae5db7>] btrfs_mount+0x184/0x914
>>> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
>>> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
>>> [<000000003b67b9fc>] do_mount+0x5b9/0xc70
>>> [<00000000de4073a0>] SyS_mount+0x80/0xd0
>>> [<00000000fc5a968a>] do_syscall_64+0x5d/0x110
>>> [<000000003d61f5fc>] entry_SYSCALL_64_after_hwframe+0x21/0x86
>>> [<00000000458a6ffa>] 0xffffffffffffffff
>>>
>>> Not sure whether the above is a true issue or not.
>>>
>>> However, by inspecting the code, I do find the trie_free in lpm_trie.c
>>> may have missed freeing the trie memory.
>>>
>>> The change likes below should work:
>>> -bash-4.2$ git diff
>>> diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c
>>> index 7b469d1..cecb259 100644
>>> --- a/kernel/bpf/lpm_trie.c
>>> +++ b/kernel/bpf/lpm_trie.c
>>> @@ -589,6 +589,7 @@ static void trie_free(struct bpf_map *map)
>>>
>>> unlock:
>>> raw_spin_unlock(&trie->lock);
>>> + kfree(trie);
>
> also looks like trie_free() is missing
> synchronize_rcu() + rcu_barrier()
> it doesn't wait for parallel lookup/update/delete to complete
> before freeing the elements.
Thanks, Alexei. I will address this in the patch as well.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kmemleak] unreferenced object 0xcd9c1a80 (size 192):
2018-02-12 5:47 ` Yonghong Song
2018-02-12 8:28 ` Daniel Borkmann
@ 2018-02-12 19:26 ` Mathieu Malaterre
1 sibling, 0 replies; 8+ messages in thread
From: Mathieu Malaterre @ 2018-02-12 19:26 UTC (permalink / raw)
To: Yonghong Song
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann, LKML
On Mon, Feb 12, 2018 at 6:47 AM, Yonghong Song <yhs@fb.com> wrote:
>
>
> On 2/11/18 11:18 AM, Mathieu Malaterre wrote:
>>
>> Hi,
>>
>> On Sun, Feb 11, 2018 at 5:54 PM, Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>>
>>> On Sun, Feb 11, 2018 at 7:24 AM, Mathieu Malaterre <malat@debian.org>
>>> wrote:
>>>>
>>>> Alexei,
>>>>
>>>> Could you please comment on why I am seeing those memleaks being
>>>> reported on my ppc32 system ? Should they be marked as false positive
>>>> ?
>>>>
>>>> System is Mac Mini G4, git/master (4.15.0+), ppc.
>>>>
>>>> Thanks for your time
>>>>
>>>> $ dmesg
>>>> ...
>>>> [ 1281.504173] kmemleak: 36 new suspected memory leaks (see
>>>> /sys/kernel/debug/kmemleak)
>>>>
>>>> Where:
>>>>
>>>> # cat /sys/kernel/debug/kmemleak
>>>> unreferenced object 0xdee25000 (size 192):
>>>> comm "systemd", pid 1, jiffies 4294894348 (age 1438.580s)
>>>> hex dump (first 32 bytes):
>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 0c .V/.............
>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>> backtrace:
>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>> unreferenced object 0xdee25900 (size 192):
>>>> comm "systemd", pid 1, jiffies 4294894540 (age 1437.812s)
>>>> hex dump (first 32 bytes):
>>>> c0 56 2f 88 00 00 00 00 00 00 00 0b 00 00 00 08 .V/.............
>>>> 00 00 00 08 00 00 00 01 00 00 00 01 00 00 00 01 ................
>>>> backtrace:
>>>> [<6c69baf5>] trie_alloc+0xb0/0x150
>>>> [<fa093284>] SyS_bpf+0x288/0x1458
>>>> [<82182f53>] ret_from_syscall+0x0/0x38
>>>
>>>
>>> hmm. looks real. Is there a reproducer?
>>> Yonghong, lpm map not cleaning after itself?
>>
>>
>> Not really. I simply boot up my machine and wait for the first kmemleak
>> scan.
>
>
> I am not able to reproduce the issue. Tried with latest net-next on FC26
> with kmemleak on. I only got this one after bootup,
> 'cat /sys/kernel/debug/kmemleak' or
> 'echo scan > /sys/kernel/debug/kmemleak
> cat /sys/kernel/debug/kmemleak':
>
> unreferenced object 0xffff99701a7386e0 (size 32):
> comm "mount", pid 1856, jiffies 4294669263 (age 98.440s)
> hex dump (first 32 bytes):
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> backtrace:
> [<000000004668ec00>] security_sb_parse_opts_str+0x36/0x50
> [<00000000a9807d2b>] parse_security_options+0x3d/0x60
> [<00000000cc1e1d58>] btrfs_mount_root+0x139/0x720
> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> [<0000000093ae5db7>] btrfs_mount+0x184/0x914
> [<00000000bdc4f1a3>] mount_fs+0x30/0x150
> [<00000000f189f1bd>] vfs_kern_mount.part.26+0x54/0x100
> [<000000003b67b9fc>] do_mount+0x5b9/0xc70
> [<00000000de4073a0>] SyS_mount+0x80/0xd0
> [<00000000fc5a968a>] do_syscall_64+0x5d/0x110
> [<000000003d61f5fc>] entry_SYSCALL_64_after_hwframe+0x21/0x86
> [<00000000458a6ffa>] 0xffffffffffffffff
>
> Not sure whether the above is a true issue or not.
>
> However, by inspecting the code, I do find the trie_free in lpm_trie.c
> may have missed freeing the trie memory.
>
> The change likes below should work:
> -bash-4.2$ git diff
> diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c
> index 7b469d1..cecb259 100644
> --- a/kernel/bpf/lpm_trie.c
> +++ b/kernel/bpf/lpm_trie.c
> @@ -589,6 +589,7 @@ static void trie_free(struct bpf_map *map)
>
> unlock:
> raw_spin_unlock(&trie->lock);
> + kfree(trie);
> }
>
> static int trie_get_next_key(struct bpf_map *map, void *_key, void
> *_next_key)
> -bash-4.2$
With this single patch added, system has been running for a couple of
hours, no memleak reported. So:
Tested-by: Mathieu Malaterre <malat@debian.org>
> Will propose a formal patch for this soon.
Will retest if needed.
Thanks !
-M
^ permalink raw reply [flat|nested] 8+ messages in thread