From: Rustam Kovhaev <rkovhaev@gmail.com>
To: Dmitry Vyukov <dvyukov@google.com>, andrii@kernel.org
Cc: syzbot <syzbot+f3694595248708227d35@syzkaller.appspotmail.com>,
andrii@kernel.org, Alexei Starovoitov <ast@kernel.org>,
bpf <bpf@vger.kernel.org>, Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Martin KaFai Lau <kafai@fb.com>, KP Singh <kpsingh@chromium.org>,
LKML <linux-kernel@vger.kernel.org>,
netdev <netdev@vger.kernel.org>, Song Liu <songliubraving@fb.com>,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Yonghong Song <yhs@fb.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: memory leak in bpf
Date: Wed, 7 Apr 2021 16:24:44 -0700 [thread overview]
Message-ID: <YG4/PEhZ9CnKo1K3@nuc10> (raw)
In-Reply-To: <CACT4Y+bvWyipjZ6P6gkno0ZHRWPJ-HFGiT3yECqQU37a0E_tgQ@mail.gmail.com>
On Mon, Mar 01, 2021 at 09:43:00PM +0100, Dmitry Vyukov wrote:
> On Mon, Mar 1, 2021 at 9:39 PM Rustam Kovhaev <rkovhaev@gmail.com> wrote:
> >
> > On Mon, Mar 01, 2021 at 08:05:42PM +0100, Dmitry Vyukov wrote:
> > > On Mon, Mar 1, 2021 at 5:21 PM Rustam Kovhaev <rkovhaev@gmail.com> wrote:
> > > >
> > > > On Wed, Dec 09, 2020 at 10:58:10PM -0800, syzbot wrote:
> > > > > syzbot has found a reproducer for the following issue on:
> > > > >
> > > > > HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check
> > > > > git tree: upstream
> > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000
> > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f
> > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35
> > > > > compiler: gcc (GCC) 10.1.0-syz 20200507
> > > > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000
> > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000
> > > > >
> > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > > Reported-by: syzbot+f3694595248708227d35@syzkaller.appspotmail.com
> > > > >
> > > > > Debian GNU/Linux 9 syzkaller ttyS0
> > > > > Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts.
> > > > > executing program
> > > > > executing program
> > > > > executing program
> > > > > BUG: memory leak
> > > > > unreferenced object 0xffff88810efccc80 (size 64):
> > > > > comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s)
> > > > > hex dump (first 32 bytes):
> > > > > c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................
> > > > > c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff .V?.....@.8.....
> > > > > backtrace:
> > > > > [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline]
> > > > > [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]
> > > > > [<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline]
> > > > > [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline]
> > > > > [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150
> > > > > [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline]
> > > > > [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline]
> > > > > [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381
> > > > > [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> > > > > [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > >
> > > > >
> > > >
> > > > i am pretty sure that this one is a false positive
> > > > the problem with reproducer is that it does not terminate all of the
> > > > child processes that it spawns
> > > >
> > > > i confirmed that it is a false positive by tracing __fput() and
> > > > bpf_map_release(), i ran reproducer, got kmemleak report, then i
> > > > manually killed those running leftover processes from reproducer and
> > > > then both functions were executed and memory was freed
> > > >
> > > > i am marking this one as:
> > > > #syz invalid
> > >
> > > Hi Rustam,
> > >
> > > Thanks for looking into this.
> > >
> > > I wonder how/where are these objects referenced? If they are not
> > > leaked and referenced somewhere, KMEMLEAK should not report them as
> > > leaks.
> > > So even if this is a false positive for BPF, this is a true positive
> > > bug and something to fix for KMEMLEAK ;)
> > > And syzbot will probably re-create this bug report soon as this still
> > > happens and is not a one-off thing.
> >
> > hi Dmitry, i haven't thought of it this way, but i guess you are right,
> > it is a kmemleak bug, ideally kmemleak should be aware that there are
> > still running processes holding references to bpf fd/anonymous inodes
> > which in their turn hold references to allocated bpf maps
>
> KMEMLEAK scans whole memory, so if there are pointers to the object
> anywhere in memory, KMEMLEAK should not report them as leaked. Running
> processes have no direct effect on KMEMLEAK logic.
> So the question is: where are these pointers to these objects? If we
> answer this, we can check how/why KMEMLEAK misses them. Are they
> mangled in some way?
thank you for your comments, they make sense, and indeed, the pointer
gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the
issue again, and Andrii had to look into the same bug, sorry about that.
if i am understanding this correctly here is what the fix should be:
---
kernel/bpf/ringbuf.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c
index f25b719ac786..30400e74abe2 100644
--- a/kernel/bpf/ringbuf.c
+++ b/kernel/bpf/ringbuf.c
@@ -8,6 +8,7 @@
#include <linux/vmalloc.h>
#include <linux/wait.h>
#include <linux/poll.h>
+#include <linux/kmemleak.h>
#include <uapi/linux/btf.h>
#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node)
rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages,
VM_ALLOC | VM_USERMAP, PAGE_KERNEL);
if (rb) {
+ kmemleak_not_leak((void *) pages);
rb->pages = pages;
rb->nr_pages = nr_pages;
return rb;
--
2.30.2
next prev parent reply other threads:[~2021-04-07 23:24 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-18 3:50 memory leak in bpf syzbot
2020-12-10 6:58 ` syzbot
2021-03-01 16:21 ` Rustam Kovhaev
2021-03-01 19:05 ` Dmitry Vyukov
2021-03-01 20:39 ` Rustam Kovhaev
2021-03-01 20:43 ` Dmitry Vyukov
2021-04-07 23:24 ` Rustam Kovhaev [this message]
2021-04-07 23:35 ` Andrii Nakryiko
2021-04-08 18:56 ` Rustam Kovhaev
2021-06-13 15:17 ` Rustam Kovhaev
2021-06-24 17:28 ` Rustam Kovhaev
2021-06-25 14:28 ` Dmitry Vyukov
2021-06-26 18:40 ` Rustam Kovhaev
2021-07-07 17:06 ` Andrii Nakryiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YG4/PEhZ9CnKo1K3@nuc10 \
--to=rkovhaev@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dvyukov@google.com \
--cc=gregkh@linuxfoundation.org \
--cc=john.fastabend@gmail.com \
--cc=kafai@fb.com \
--cc=kpsingh@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=syzbot+f3694595248708227d35@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).