From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17A92C43460 for ; Thu, 8 Apr 2021 18:56:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7A706023C for ; Thu, 8 Apr 2021 18:56:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232923AbhDHS4h (ORCPT ); Thu, 8 Apr 2021 14:56:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231676AbhDHS4e (ORCPT ); Thu, 8 Apr 2021 14:56:34 -0400 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B023C061760; Thu, 8 Apr 2021 11:56:23 -0700 (PDT) Received: by mail-pj1-x102b.google.com with SMTP id kk2-20020a17090b4a02b02900c777aa746fso1911149pjb.3; Thu, 08 Apr 2021 11:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=IexjwBFUHeHNYiYxUhNPM2AzfjEbH0Hum3i/l7HbIAo=; b=mPhDzdxKQHYvRyEAUvmiPOfCp+vgeyoP4BgErKeoyQAd46sX2YgRNheIVRe3Z8J6wh mqW72o2DVBXNptZZ2O6nZ58ypaSFxVxxFFfNPz4xfeN6pqRdndidLPpflBMgNp4BEwb6 e06/LxILWaBHoUwfHDwTG/pSte0AelCRm3AhUI1DqHutreRor8VLCkBUDRvMkyw7Apm6 8q3BQGDfUd89BLOK6R4OHHrBZiRsT9ma9FL6xHAHOlLMp4yoGZckgm5OmtuxKExY6b1S h2i50oa3AeLzeO3u+Gj0QUZbPQXMPvDb0pLChuWIBeDJosvFYCZ/rNnicsMzvzcUL0FS QoRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=IexjwBFUHeHNYiYxUhNPM2AzfjEbH0Hum3i/l7HbIAo=; b=XNIsTbEFhZqh2S/D88vXxQlIuwLPoCme3mUCLalH8fGpltPSN5Q6X531wY+x451fod R8nx+Ppo0ask5OnPFRaPsMcT9ltKMsLilNiGu0HUnAWFrjoByCwPHIEJGndyC0i7+sHI DW0W8I17PHJnW/fKRwy9UkxC8Ku00Yg1MSwwo8B0fMKgrPERjAKZR1cNckG7s1Cd/eYu V7kjVvabcSYLwNeQrrpEft052BPAsLTdF2RI+Ng4XiA9j8mQTUcv+TDjLTI+7bt924Bk uLzO4Dw00iF3ETz/CF1APgMYPGy/7ZYnl/bZs8U95n0fIllFNygxrX44DXuooZB13I1e 3ZYA== X-Gm-Message-State: AOAM533KwsxtfCmF9U04Cbx/QAIO89dKIdWD1dZ5VHzK3QSQO2/PouE/ Z0GJfo6OSqGqvW+h1kDrC60= X-Google-Smtp-Source: ABdhPJyb53zIKdPFu1C1EK8w0rYzGNzJsdT+WLA7sk56quBiVuqrDeAGUW/Z9x4iWhVXTKB1kY5Ujw== X-Received: by 2002:a17:90b:4a8a:: with SMTP id lp10mr9061898pjb.27.1617908182755; Thu, 08 Apr 2021 11:56:22 -0700 (PDT) Received: from nuc10 (104.36.148.139.aurocloud.com. [104.36.148.139]) by smtp.gmail.com with ESMTPSA id 7sm195496pfv.97.2021.04.08.11.56.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Apr 2021 11:56:22 -0700 (PDT) Date: Thu, 8 Apr 2021 11:56:15 -0700 From: Rustam Kovhaev To: Andrii Nakryiko Cc: Dmitry Vyukov , Andrii Nakryiko , syzbot , Alexei Starovoitov , bpf , Daniel Borkmann , John Fastabend , Martin KaFai Lau , KP Singh , LKML , netdev , Song Liu , syzkaller-bugs , Yonghong Song , Greg Kroah-Hartman Subject: Re: memory leak in bpf Message-ID: References: <000000000000911d3905b459824c@google.com> <000000000000e56a2605b616b2d9@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Apr 07, 2021 at 04:35:34PM -0700, Andrii Nakryiko wrote: > On Wed, Apr 7, 2021 at 4:24 PM Rustam Kovhaev wrote: > > > > On Mon, Mar 01, 2021 at 09:43:00PM +0100, Dmitry Vyukov wrote: > > > On Mon, Mar 1, 2021 at 9:39 PM Rustam Kovhaev wrote: > > > > > > > > On Mon, Mar 01, 2021 at 08:05:42PM +0100, Dmitry Vyukov wrote: > > > > > On Mon, Mar 1, 2021 at 5:21 PM Rustam Kovhaev wrote: > > > > > > > > > > > > On Wed, Dec 09, 2020 at 10:58:10PM -0800, syzbot wrote: > > > > > > > syzbot has found a reproducer for the following issue on: > > > > > > > > > > > > > > HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check > > > > > > > git tree: upstream > > > > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 > > > > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f > > > > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 > > > > > > > compiler: gcc (GCC) 10.1.0-syz 20200507 > > > > > > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 > > > > > > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000 > > > > > > > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > > > > Reported-by: syzbot+f3694595248708227d35@syzkaller.appspotmail.com > > > > > > > > > > > > > > Debian GNU/Linux 9 syzkaller ttyS0 > > > > > > > Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. > > > > > > > executing program > > > > > > > executing program > > > > > > > executing program > > > > > > > BUG: memory leak > > > > > > > unreferenced object 0xffff88810efccc80 (size 64): > > > > > > > comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) > > > > > > > hex dump (first 32 bytes): > > > > > > > c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ > > > > > > > c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff .V?.....@.8..... > > > > > > > backtrace: > > > > > > > [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] > > > > > > > [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline] > > > > > > > [<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] > > > > > > > [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] > > > > > > > [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 > > > > > > > [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] > > > > > > > [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] > > > > > > > [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 > > > > > > > [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 > > > > > > > [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > > > > > > > > > > > > > > > > > > > > > > > > > i am pretty sure that this one is a false positive > > > > > > the problem with reproducer is that it does not terminate all of the > > > > > > child processes that it spawns > > > > > > > > > > > > i confirmed that it is a false positive by tracing __fput() and > > > > > > bpf_map_release(), i ran reproducer, got kmemleak report, then i > > > > > > manually killed those running leftover processes from reproducer and > > > > > > then both functions were executed and memory was freed > > > > > > > > > > > > i am marking this one as: > > > > > > #syz invalid > > > > > > > > > > Hi Rustam, > > > > > > > > > > Thanks for looking into this. > > > > > > > > > > I wonder how/where are these objects referenced? If they are not > > > > > leaked and referenced somewhere, KMEMLEAK should not report them as > > > > > leaks. > > > > > So even if this is a false positive for BPF, this is a true positive > > > > > bug and something to fix for KMEMLEAK ;) > > > > > And syzbot will probably re-create this bug report soon as this still > > > > > happens and is not a one-off thing. > > > > > > > > hi Dmitry, i haven't thought of it this way, but i guess you are right, > > > > it is a kmemleak bug, ideally kmemleak should be aware that there are > > > > still running processes holding references to bpf fd/anonymous inodes > > > > which in their turn hold references to allocated bpf maps > > > > > > KMEMLEAK scans whole memory, so if there are pointers to the object > > > anywhere in memory, KMEMLEAK should not report them as leaked. Running > > > processes have no direct effect on KMEMLEAK logic. > > > So the question is: where are these pointers to these objects? If we > > > answer this, we can check how/why KMEMLEAK misses them. Are they > > > mangled in some way? > > thank you for your comments, they make sense, and indeed, the pointer > > gets vmaped. > > i should have looked into this sooner, becaused syzbot did trigger the > > issue again, and Andrii had to look into the same bug, sorry about that. > > No worries! I actually forgot about this thread :) Let's leave the > link to my today's investigation ([0]) just for completeness. > > [0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/ > > > if i am understanding this correctly here is what the fix should be: > > --- > > kernel/bpf/ringbuf.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c > > index f25b719ac786..30400e74abe2 100644 > > --- a/kernel/bpf/ringbuf.c > > +++ b/kernel/bpf/ringbuf.c > > @@ -8,6 +8,7 @@ > > #include > > #include > > #include > > +#include > > #include > > > > #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE) > > @@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) > > rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, > > VM_ALLOC | VM_USERMAP, PAGE_KERNEL); > > if (rb) { > > + kmemleak_not_leak((void *) pages); > > If that makes kmemleak happy, I have no problems with this. But maybe > leave some comment explaining why this is needed at all? > > And for my understanding, how vmap changes anything? Those pages are > still referenced from rb, which is referenced from some struct file in > the system. Sorry if that's a naive question. > valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further