From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3A14C433E2 for ; Fri, 11 Sep 2020 07:36:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0435422204 for ; Fri, 11 Sep 2020 07:36:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JiUND/gj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0435422204 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5234E6B0093; Fri, 11 Sep 2020 03:36:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D2546B0095; Fri, 11 Sep 2020 03:36:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C24F90000B; Fri, 11 Sep 2020 03:36:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0158.hostedemail.com [216.40.44.158]) by kanga.kvack.org (Postfix) with ESMTP id 26A576B0093 for ; Fri, 11 Sep 2020 03:36:11 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D6BE8824805A for ; Fri, 11 Sep 2020 07:36:10 +0000 (UTC) X-FDA: 77249972100.20.blow59_3703fd4270ec Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id ACA3A180C07A3 for ; Fri, 11 Sep 2020 07:36:10 +0000 (UTC) X-HE-Tag: blow59_3703fd4270ec X-Filterd-Recvd-Size: 7708 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Fri, 11 Sep 2020 07:36:10 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id v54so7126998qtj.7 for ; Fri, 11 Sep 2020 00:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=AgNHmGolg065FIPbyM/md78J/4QNU3wlyP9uV/8/RYg=; b=JiUND/gjxrV/gBkj143394Hm2srxooce7sGxGfCQSsiLfV4lvwc6tVZuZPj155tOU6 L3uPm0geiPdeXrRMHUqYgtM1qD7dGKHVMcQxgnkI2FGfUqQIOcs4r9zAl+XEbYLULF9c Dr5ibZj4jpH+Vo1ZDbAr29HeknsCBt1a77gCErDYBN1Xp6zKczaF5v3KAjaP2MRH2aT3 M/6dsFHremiT4SeSPVFW9cWRPT5HMEw2EVJFSVTaZ0/V/Wy5USdxStUEt0CtIL7ZlXia otL+xA3gFKF3xHSOh69ooIli779AmgDhoKeNtAUCnw5T6RPdmbPRd5Eod2PxXe0ZFbB2 4gyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=AgNHmGolg065FIPbyM/md78J/4QNU3wlyP9uV/8/RYg=; b=bpllpzMwOF21JRmHHZmnH+rew643ySbjhxbWpDsuz4DyXEN1CTwMww9A60rX1O0wHf CcLHpZhPSN+kr+78xWaX5hUJPV2jomcinuqikl8WSrExUNMqPedSNJUVFOiYsgw/vIEJ nHbb3mz4J8q/mfQPNMcoPSiw7zvxcCZQe79nemRbSy7/hI3gKhYblqW3td1G2XQ5SCPi 8Bgvy/BfW0J4UWRVmlkLvOrJkvgg2rY31QmOmyfNtYwPa1r7oTZX9y4Urg3oTOlg2hm3 cEwytWYYarNFP2StC9r7PkdPXxKvN7jhL/AZ+kc7KutKqO33uNdU4CS/iYNMWiF9+NCp VkCQ== X-Gm-Message-State: AOAM533tCAHGIMMty3LXxD/i3aWwj+3Qd4ap5BKgI57V2rkGQBKBAGDe yirG/AZItiZFsvazuLZsRaO5+A6R/QwPZbdSCnn7fg== X-Google-Smtp-Source: ABdhPJxHoNHd4QLdks06t0G4pvWYQGyvt9teUFNsQIlgL4vGramjgW27KW2RfbNH8qlUHaNUu+IXpXoAS+47ksJC4eQ= X-Received: by 2002:ac8:5215:: with SMTP id r21mr667601qtn.257.1599809769436; Fri, 11 Sep 2020 00:36:09 -0700 (PDT) MIME-Version: 1.0 References: <20200907134055.2878499-1-elver@google.com> <20200908153102.GB61807@elver.google.com> <20200908155631.GC61807@elver.google.com> In-Reply-To: <20200908155631.GC61807@elver.google.com> From: Dmitry Vyukov Date: Fri, 11 Sep 2020 09:35:58 +0200 Message-ID: Subject: Re: [PATCH RFC 00/10] KFENCE: A low-overhead sampling-based memory safety error detector To: Marco Elver Cc: Vlastimil Babka , Dave Hansen , Alexander Potapenko , Andrew Morton , Catalin Marinas , Christoph Lameter , David Rientjes , Joonsoo Kim , Mark Rutland , Pekka Enberg , "H. Peter Anvin" , "Paul E. McKenney" , Andrey Konovalov , Andrey Ryabinin , Andy Lutomirski , Borislav Petkov , Dave Hansen , Eric Dumazet , Greg Kroah-Hartman , Ingo Molnar , Jann Horn , Jonathan Corbet , Kees Cook , Peter Zijlstra , Qian Cai , Thomas Gleixner , Will Deacon , "the arch/x86 maintainers" , "open list:DOCUMENTATION" , LKML , kasan-dev , Linux ARM , Linux-MM Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: ACA3A180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 8, 2020 at 5:56 PM Marco Elver wrote: > > On Tue, Sep 08, 2020 at 05:36PM +0200, Vlastimil Babka wrote: > > On 9/8/20 5:31 PM, Marco Elver wrote: > > >> > > >> How much memory overhead does this end up having? I know it depends on > > >> the object size and so forth. But, could you give some real-world > > >> examples of memory consumption? Also, what's the worst case? Say I > > >> have a ton of worst-case-sized (32b) slab objects. Will I notice? > > > > > > KFENCE objects are limited (default 255). If we exhaust KFENCE's memory > > > pool, no more KFENCE allocations will occur. > > > Documentation/dev-tools/kfence.rst gives a formula to calculate the > > > KFENCE pool size: > > > > > > The total memory dedicated to the KFENCE memory pool can be computed as:: > > > > > > ( #objects + 1 ) * 2 * PAGE_SIZE > > > > > > Using the default config, and assuming a page size of 4 KiB, results in > > > dedicating 2 MiB to the KFENCE memory pool. > > > > > > Does that clarify this point? Or anything else that could help clarify > > > this? > > > > Hmm did you observe that with this limit, a long-running system would eventually > > converge to KFENCE memory pool being filled with long-aged objects, so there > > would be no space to sample new ones? > > Sure, that's a possibility. But remember that we're not trying to > deterministically detect bugs on 1 system (if you wanted that, you > should use KASAN), but a fleet of machines! The non-determinism of which > allocations will end up in KFENCE, will ensure we won't end up with a > fleet of machines of identical allocations. That's exactly what we're > after. Even if we eventually exhaust the pool, you'll still detect bugs > if there are any. > > If you are overly worried, either the sample interval or number of > available objects needs to be tweaked to be larger. The default of 255 > is quite conservative, and even using something larger on a modern > system is hardly noticeable. Choosing a sample interval & number of > objects should also factor in how many machines you plan to deploy this > on. Monitoring /sys/kernel/debug/kfence/stats can help you here. Hi Marco, I reviewed patches and they look good to me (minus some local comments that I've left). The main question/concern I have is what Vlastimil mentioned re long-aged objects. Is the default sample interval values reasonable for typical workloads? Do we have any guidelines on choosing the sample interval? Should it depend on workload/use pattern? By "reasonable" I mean if the pool will last long enough to still sample something after hours/days? Have you tried any experiments with some workload (both short-lived processes and long-lived processes/namespaces) capturing state of the pool? It can make sense to do to better understand dynamics. I suspect that the rate may need to be orders of magnitude lower. Also I am wondering about the boot process (both kernel and init). It's both inherently almost the same for the whole population of machines and inherently produces persistent objects. Should we lower the rate for the first minute of uptime? Or maybe make it proportional to uptime? I feel it's quite an important aspect. We can have this awesome idea and implementation, but radically lower its utility by using bad sampling value (which will have silent "failure mode" -- no bugs detected). But to make it clear: all of this does not conflict with the merge of the first version. Just having tunable sampling interval is good enough. We will get the ultimate understanding only when we start using it widely anyway.