From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 930BAC433E6 for ; Thu, 4 Mar 2021 19:03:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6157964F60 for ; Thu, 4 Mar 2021 19:03:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236185AbhCDTCo (ORCPT ); Thu, 4 Mar 2021 14:02:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233959AbhCDTCX (ORCPT ); Thu, 4 Mar 2021 14:02:23 -0500 Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B94A4C061756 for ; Thu, 4 Mar 2021 11:01:43 -0800 (PST) Received: by mail-ot1-x333.google.com with SMTP id g8so24667191otk.4 for ; Thu, 04 Mar 2021 11:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=NVSvNizI4KZN1UKsLkx5qjZ7v0EVfLOog8osH31hbuPP18B77Ed/U1DkJ+wnX8LNIc 97OleKF0shu4p7MuRnpkbjzD1oxfat5ge/CSKtqW2mE8Awgcd7VEs3CAGVs4G78siGJT VxNaFgsVdSOgBMq6eoBPf5c+taZPck9ZcssIyBepMk7JVIgtw9ZVsaNLM4rwF3/E9/jO 7KnOda3cN/yi9ANa5UMNvvDMJZ+tdOhmzWvhltmFxYatTuNZbktwwnvs7LYwUQ2eUNSz Chta7Uzah3jv3eflKadZzSdrUkCwcyRduia8ZdQmyWjr7NeHBkxUZzNxTSZvxJc+2Wvz 4Wcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=VUDN+SMYYv09D5dm1Exh1XzT1TMLODEYWLS2OiZfKEg4MGeLxRK3cKx0kcyMBhn1xk tnhgP/NTqTe4nK4AJ+73T6FKecvuc3/vMZXSwbs2jo/feyk3bO0oRRiJoAtlGbX2DkEQ Z+F+KbooxUoNMrzJ3gIDGcSQCzucsBy6egCQ+pyBwYVuGySvgPiOG+Q7u2sFPS9wzALs EMMc5SZjoXdItaSGfIC8zn1ukZTdLUu0AvBqzvS/0r+m8Ini88/NyRtJ6BV4Xpia5yZP 4RLEqulLxWVPVEMJd4HSwNfFPck0H91lYvb6o6as12Ka5CpbpL506NP5qWNH2T/8Sn3u T4yQ== X-Gm-Message-State: AOAM531g/gk1nNnqdqxcTh1bD52uYEm4AJUoPWlSytWaGSZHJVt9MctS 5XNnC+/ozUUjRQhri4JhmPnEME83y3U1VIiFiFyeIA== X-Google-Smtp-Source: ABdhPJzKjopu6ZYj7NmF3XXjx3un/RfQBHMPpkfQSADnXR012A/z7CvK/whzcqg9tH5PWdIP9RUXLf6upEy7KGUbdbI= X-Received: by 2002:a9d:7f11:: with SMTP id j17mr4694384otq.251.1614884502710; Thu, 04 Mar 2021 11:01:42 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> <20210304165923.GA60457@C02TD0UTHF1T.local> <20210304180154.GD60457@C02TD0UTHF1T.local> <20210304185148.GE60457@C02TD0UTHF1T.local> In-Reply-To: <20210304185148.GE60457@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 20:01:29 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Cc: Christophe Leroy , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , LKML , linuxppc-dev@lists.ozlabs.org, kasan-dev , Catalin Marinas , Will Deacon , Linux ARM , broonie@kernel.org, linux-toolchains@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 4 Mar 2021 at 19:51, Mark Rutland wrote: > On Thu, Mar 04, 2021 at 07:22:53PM +0100, Marco Elver wrote: > > On Thu, 4 Mar 2021 at 19:02, Mark Rutland wrote: > > > On Thu, Mar 04, 2021 at 06:25:33PM +0100, Marco Elver wrote: > > > > On Thu, Mar 04, 2021 at 04:59PM +0000, Mark Rutland wrote: > > > > > On Thu, Mar 04, 2021 at 04:30:34PM +0100, Marco Elver wrote: > > > > > > On Thu, 4 Mar 2021 at 15:57, Mark Rutland wrote: > > > > > > > [adding Mark Brown] > > > > > > > > > > > > > > The bigger problem here is that skipping is dodgy to begin with, and > > > > > > > this is still liable to break in some cases. One big concern is that > > > > > > > (especially with LTO) we cannot guarantee the compiler will not inline > > > > > > > or outline functions, causing the skipp value to be too large or too > > > > > > > small. That's liable to happen to callers, and in theory (though > > > > > > > unlikely in practice), portions of arch_stack_walk() or > > > > > > > stack_trace_save() could get outlined too. > > > > > > > > > > > > > > Unless we can get some strong guarantees from compiler folk such that we > > > > > > > can guarantee a specific function acts boundary for unwinding (and > > > > > > > doesn't itself get split, etc), the only reliable way I can think to > > > > > > > solve this requires an assembly trampoline. Whatever we do is liable to > > > > > > > need some invasive rework. > > > > > > > > > > > > Will LTO and friends respect 'noinline'? > > > > > > > > > > I hope so (and suspect we'd have more problems otherwise), but I don't > > > > > know whether they actually so. > > > > > > > > > > I suspect even with 'noinline' the compiler is permitted to outline > > > > > portions of a function if it wanted to (and IIUC it could still make > > > > > specialized copies in the absence of 'noclone'). > > > > > > > > > > > One thing I also noticed is that tail calls would also cause the stack > > > > > > trace to appear somewhat incomplete (for some of my tests I've > > > > > > disabled tail call optimizations). > > > > > > > > > > I assume you mean for a chain A->B->C where B tail-calls C, you get a > > > > > trace A->C? ... or is A going missing too? > > > > > > > > Correct, it's just the A->C outcome. > > > > > > I'd assumed that those cases were benign, e.g. for livepatching what > > > matters is what can be returned to, so B disappearing from the trace > > > isn't a problem there. > > > > > > Is the concern debugability, or is there a functional issue you have in > > > mind? > > > > For me, it's just been debuggability, and reliable test cases. > > > > > > > > Is there a way to also mark a function non-tail-callable? > > > > > > > > > > I think this can be bodged using __attribute__((optimize("$OPTIONS"))) > > > > > on a caller to inhibit TCO (though IIRC GCC doesn't reliably support > > > > > function-local optimization options), but I don't expect there's any way > > > > > to mark a callee as not being tail-callable. > > > > > > > > I don't think this is reliable. It'd be > > > > __attribute__((optimize("-fno-optimize-sibling-calls"))), but doesn't > > > > work if applied to the function we do not want to tail-call-optimize, > > > > but would have to be applied to the function that does the tail-calling. > > > > > > Yup; that's what I meant then I said you could do that on the caller but > > > not the callee. > > > > > > I don't follow why you'd want to put this on the callee, though, so I > > > think I'm missing something. Considering a set of functions in different > > > compilation units: > > > > > > A->B->C->D->E->F->G->H->I->J->K > > > > I was having this problem with KCSAN, where the compiler would > > tail-call-optimize __tsan_X instrumentation. > > Those are compiler-generated calls, right? When those are generated the > compilation unit (and whatever it has included) might not have provided > a prototype anyway, and the compiler has special knowledge of the > functions, so it feels like the compiler would need to inhibit TCO here > for this to be robust. For their intended usage subjecting them to TCO > doesn't seem to make sense AFAICT. > > I suspect that compilers have some way of handling that; otherwise I'd > expect to have heard stories of mcount/fentry calls getting TCO'd and > causing problems. So maybe there's an easy fix there? I agree, the compiler builtins should be handled by the compiler directly, perhaps that was a bad example. But we also have "explicit instrumentation", e.g. everything that's in . > > This would mean that KCSAN runtime functions ended up in the trace, > > but the function where the access happened would not. However, I don't > > care about the runtime functions, and instead want to see the function > > where the access happened. In that case, I'd like to just mark > > __tsan_X and any other kcsan instrumentation functions as > > do-not-tail-call-optimize, which would solve the problem. > > I understand why we don't want to TCO these calls, but given the calls > are implicitly generated, I strongly suspect it's better to fix the > implicit call generation to not be TCO'd to begin with. > > > The solution today is that when you compile a kernel with KCSAN, every > > instrumented TU is compiled with -fno-optimize-sibling-calls. The > > better solution would be to just mark KCSAN runtime functions somehow, > > but permit tail calling other things. Although, I probably still want > > to see the full trace, and would decide that having > > -fno-optimize-sibling-calls is a small price to pay in a > > debug-only-kernel to get complete traces. > > > > > ... if K were marked in this way, and J was compiled with visibility of > > > this, J would stick around, but J's callers might not, and so the a > > > trace might see: > > > > > > A->J->K > > > > > > ... do you just care about the final caller, i.e. you just need > > > certainty that J will be in the trace? > > > > Yes. But maybe it's a special problem that only sanitizers have. > > I reckon for basically any instrumentation we don't want calls to be > TCO'd, though I'm not immediately sure of cases beyond sanitizers and > mcount/fentry. Thinking about this more, I think it's all debugging tools. E.g. lockdep, if you lock/unlock at the end of a function, you might tail call into lockdep. If the compiler applies TCO, and lockdep determines there's a bug and then shows a trace, you'll have no idea where the actual bug is. The kernel has lots of debugging facilities that add instrumentation in this way. So perhaps it's a general debugging-tool problem (rather than just sanitizers). Thanks, -- Marco From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41292C433E0 for ; Thu, 4 Mar 2021 19:02:17 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A28264F3E for ; Thu, 4 Mar 2021 19:02:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A28264F3E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ds0bp4d7tz3dCF for ; Fri, 5 Mar 2021 06:02:14 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=NVSvNizI; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::334; helo=mail-ot1-x334.google.com; envelope-from=elver@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=NVSvNizI; dkim-atps=neutral Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ds0bK1ZZMz30LJ for ; Fri, 5 Mar 2021 06:01:46 +1100 (AEDT) Received: by mail-ot1-x334.google.com with SMTP id h22so28321140otr.6 for ; Thu, 04 Mar 2021 11:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=NVSvNizI4KZN1UKsLkx5qjZ7v0EVfLOog8osH31hbuPP18B77Ed/U1DkJ+wnX8LNIc 97OleKF0shu4p7MuRnpkbjzD1oxfat5ge/CSKtqW2mE8Awgcd7VEs3CAGVs4G78siGJT VxNaFgsVdSOgBMq6eoBPf5c+taZPck9ZcssIyBepMk7JVIgtw9ZVsaNLM4rwF3/E9/jO 7KnOda3cN/yi9ANa5UMNvvDMJZ+tdOhmzWvhltmFxYatTuNZbktwwnvs7LYwUQ2eUNSz Chta7Uzah3jv3eflKadZzSdrUkCwcyRduia8ZdQmyWjr7NeHBkxUZzNxTSZvxJc+2Wvz 4Wcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=ZiAIbth/JkEWeCnU6jSIABtuuseBU4TUBulbN767JeCrJoDlU5nT3V0HDc7RcmdX/Q YUar+mAblE8g/YNwmIeE7KsPkePqi8yA9+9AoSF1EABr0KxYhv6g52xiD583+sFtOMZd romIIxbmsL5jTLOdSeeAzEknNhLCOKZPkHwWjT29s9i00fTEy2hyVybxCe91ddYZ1j40 5tXgrt0cHnGEvN+2q627ZBPY35hJMDcZE5K3/dbA261rSacCV6+D6uHYq/BhaqWzcf9q C8YSMnXRnx4kDKNy1Gh8dNeD9AWmv7830SI7bYYVNBRyHkGlLP82AYfM5oswixrLd1CK 7m7Q== X-Gm-Message-State: AOAM530nHh+r4DvtPsHSNSdfcHLYFAeiVw67SXGHyJvETyEwihLe8Em1 ETwJHQ0g76JPVm2SmmDnG6KroeZUjn1cxoXJBRnBMg== X-Google-Smtp-Source: ABdhPJzKjopu6ZYj7NmF3XXjx3un/RfQBHMPpkfQSADnXR012A/z7CvK/whzcqg9tH5PWdIP9RUXLf6upEy7KGUbdbI= X-Received: by 2002:a9d:7f11:: with SMTP id j17mr4694384otq.251.1614884502710; Thu, 04 Mar 2021 11:01:42 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> <20210304165923.GA60457@C02TD0UTHF1T.local> <20210304180154.GD60457@C02TD0UTHF1T.local> <20210304185148.GE60457@C02TD0UTHF1T.local> In-Reply-To: <20210304185148.GE60457@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 20:01:29 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Will Deacon , LKML , broonie@kernel.org, Paul Mackerras , kasan-dev , linux-toolchains@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Linux ARM Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, 4 Mar 2021 at 19:51, Mark Rutland wrote: > On Thu, Mar 04, 2021 at 07:22:53PM +0100, Marco Elver wrote: > > On Thu, 4 Mar 2021 at 19:02, Mark Rutland wrote: > > > On Thu, Mar 04, 2021 at 06:25:33PM +0100, Marco Elver wrote: > > > > On Thu, Mar 04, 2021 at 04:59PM +0000, Mark Rutland wrote: > > > > > On Thu, Mar 04, 2021 at 04:30:34PM +0100, Marco Elver wrote: > > > > > > On Thu, 4 Mar 2021 at 15:57, Mark Rutland wrote: > > > > > > > [adding Mark Brown] > > > > > > > > > > > > > > The bigger problem here is that skipping is dodgy to begin with, and > > > > > > > this is still liable to break in some cases. One big concern is that > > > > > > > (especially with LTO) we cannot guarantee the compiler will not inline > > > > > > > or outline functions, causing the skipp value to be too large or too > > > > > > > small. That's liable to happen to callers, and in theory (though > > > > > > > unlikely in practice), portions of arch_stack_walk() or > > > > > > > stack_trace_save() could get outlined too. > > > > > > > > > > > > > > Unless we can get some strong guarantees from compiler folk such that we > > > > > > > can guarantee a specific function acts boundary for unwinding (and > > > > > > > doesn't itself get split, etc), the only reliable way I can think to > > > > > > > solve this requires an assembly trampoline. Whatever we do is liable to > > > > > > > need some invasive rework. > > > > > > > > > > > > Will LTO and friends respect 'noinline'? > > > > > > > > > > I hope so (and suspect we'd have more problems otherwise), but I don't > > > > > know whether they actually so. > > > > > > > > > > I suspect even with 'noinline' the compiler is permitted to outline > > > > > portions of a function if it wanted to (and IIUC it could still make > > > > > specialized copies in the absence of 'noclone'). > > > > > > > > > > > One thing I also noticed is that tail calls would also cause the stack > > > > > > trace to appear somewhat incomplete (for some of my tests I've > > > > > > disabled tail call optimizations). > > > > > > > > > > I assume you mean for a chain A->B->C where B tail-calls C, you get a > > > > > trace A->C? ... or is A going missing too? > > > > > > > > Correct, it's just the A->C outcome. > > > > > > I'd assumed that those cases were benign, e.g. for livepatching what > > > matters is what can be returned to, so B disappearing from the trace > > > isn't a problem there. > > > > > > Is the concern debugability, or is there a functional issue you have in > > > mind? > > > > For me, it's just been debuggability, and reliable test cases. > > > > > > > > Is there a way to also mark a function non-tail-callable? > > > > > > > > > > I think this can be bodged using __attribute__((optimize("$OPTIONS"))) > > > > > on a caller to inhibit TCO (though IIRC GCC doesn't reliably support > > > > > function-local optimization options), but I don't expect there's any way > > > > > to mark a callee as not being tail-callable. > > > > > > > > I don't think this is reliable. It'd be > > > > __attribute__((optimize("-fno-optimize-sibling-calls"))), but doesn't > > > > work if applied to the function we do not want to tail-call-optimize, > > > > but would have to be applied to the function that does the tail-calling. > > > > > > Yup; that's what I meant then I said you could do that on the caller but > > > not the callee. > > > > > > I don't follow why you'd want to put this on the callee, though, so I > > > think I'm missing something. Considering a set of functions in different > > > compilation units: > > > > > > A->B->C->D->E->F->G->H->I->J->K > > > > I was having this problem with KCSAN, where the compiler would > > tail-call-optimize __tsan_X instrumentation. > > Those are compiler-generated calls, right? When those are generated the > compilation unit (and whatever it has included) might not have provided > a prototype anyway, and the compiler has special knowledge of the > functions, so it feels like the compiler would need to inhibit TCO here > for this to be robust. For their intended usage subjecting them to TCO > doesn't seem to make sense AFAICT. > > I suspect that compilers have some way of handling that; otherwise I'd > expect to have heard stories of mcount/fentry calls getting TCO'd and > causing problems. So maybe there's an easy fix there? I agree, the compiler builtins should be handled by the compiler directly, perhaps that was a bad example. But we also have "explicit instrumentation", e.g. everything that's in . > > This would mean that KCSAN runtime functions ended up in the trace, > > but the function where the access happened would not. However, I don't > > care about the runtime functions, and instead want to see the function > > where the access happened. In that case, I'd like to just mark > > __tsan_X and any other kcsan instrumentation functions as > > do-not-tail-call-optimize, which would solve the problem. > > I understand why we don't want to TCO these calls, but given the calls > are implicitly generated, I strongly suspect it's better to fix the > implicit call generation to not be TCO'd to begin with. > > > The solution today is that when you compile a kernel with KCSAN, every > > instrumented TU is compiled with -fno-optimize-sibling-calls. The > > better solution would be to just mark KCSAN runtime functions somehow, > > but permit tail calling other things. Although, I probably still want > > to see the full trace, and would decide that having > > -fno-optimize-sibling-calls is a small price to pay in a > > debug-only-kernel to get complete traces. > > > > > ... if K were marked in this way, and J was compiled with visibility of > > > this, J would stick around, but J's callers might not, and so the a > > > trace might see: > > > > > > A->J->K > > > > > > ... do you just care about the final caller, i.e. you just need > > > certainty that J will be in the trace? > > > > Yes. But maybe it's a special problem that only sanitizers have. > > I reckon for basically any instrumentation we don't want calls to be > TCO'd, though I'm not immediately sure of cases beyond sanitizers and > mcount/fentry. Thinking about this more, I think it's all debugging tools. E.g. lockdep, if you lock/unlock at the end of a function, you might tail call into lockdep. If the compiler applies TCO, and lockdep determines there's a bug and then shows a trace, you'll have no idea where the actual bug is. The kernel has lots of debugging facilities that add instrumentation in this way. So perhaps it's a general debugging-tool problem (rather than just sanitizers). Thanks, -- Marco From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10BD1C433DB for ; Thu, 4 Mar 2021 19:04:06 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8B77A64F60 for ; Thu, 4 Mar 2021 19:04:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B77A64F60 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7W8zH5B1f/7SCEdwD71i68wuHkjV20P1aLI7rPjxHbg=; b=lZbMCpVfNJxeGqQtDEc5I/gRM MPbB48WcWcAN96Ow8i8wnO3JwIKdGBQGAyXrbzfFrzIgp8WirYWNQ3vj0ErVkO9F/47M3d3wRiuk7 lC8phSFyGKjfIw9kSW5IayT6gsFupANjMoyHXoWEebcdBf1nV8O2RPjE5mearzk5fOV8QWsYvGISf 9kUm8u1wdhJIm8L22NI5z70DAgWgAZfAXKQ0Q8Xs8/oUl8XZhIYtDjkMskgHqs7ThLu47V8Ew/kKN 6nKVd+ot/Os+uyat5RS5gstIpn24oZ87G+clnmlgOlo6vMwIk9vUUnjawjK6HiF+vMVQPG9l3yM+N zEc98AnFw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHtEN-009gEE-C8; Thu, 04 Mar 2021 19:01:51 +0000 Received: from mail-ot1-x336.google.com ([2607:f8b0:4864:20::336]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHtEG-009gCs-JU for linux-arm-kernel@lists.infradead.org; Thu, 04 Mar 2021 19:01:46 +0000 Received: by mail-ot1-x336.google.com with SMTP id b8so28273539oti.7 for ; Thu, 04 Mar 2021 11:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=NVSvNizI4KZN1UKsLkx5qjZ7v0EVfLOog8osH31hbuPP18B77Ed/U1DkJ+wnX8LNIc 97OleKF0shu4p7MuRnpkbjzD1oxfat5ge/CSKtqW2mE8Awgcd7VEs3CAGVs4G78siGJT VxNaFgsVdSOgBMq6eoBPf5c+taZPck9ZcssIyBepMk7JVIgtw9ZVsaNLM4rwF3/E9/jO 7KnOda3cN/yi9ANa5UMNvvDMJZ+tdOhmzWvhltmFxYatTuNZbktwwnvs7LYwUQ2eUNSz Chta7Uzah3jv3eflKadZzSdrUkCwcyRduia8ZdQmyWjr7NeHBkxUZzNxTSZvxJc+2Wvz 4Wcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0aoBG9bt7K2Nwml5Y4UALv6dR/n8jqT8XlWBeYmu5Iw=; b=lrx+HH5MU0Z5txKBFTqo3eA9PeV+yHhZLpNvVKGKRZqqSGH6jOtRj2Gg7JgE4WF4qQ TTh2lzcHPM7Fp3IgTGa9PsZ8U/Ebjn2W0qCcsPVMIRrNvDpUK2NLyFERfc3yGTKjK2nq YLt1NpDl8Cw9OvAxSil5hxicmdUvygPDql91bn2KOPahOc4ueKFLdIddV66xfDFon6z9 5ud8VOflTYVJTK2sN9Cb5jSVmsFoFpuMk0h4drUNjasv4kuo+X0+fNDPSZ5CciE/O+Sc ni+NH9q2ss1rBMmDN/SuU931PeywGCaegPhSmFCbXgntk+TXo4zOblwo0otljhlIjdgb tw7g== X-Gm-Message-State: AOAM533SFaaCrIUxoQ0kaBx2CyepfQCP3bQfah+VNVtFWfJ72bs+sbcA dK7g3xoEXpqUn27ijHFp63/LWHeJEb2Lf4KGhvw8YA== X-Google-Smtp-Source: ABdhPJzKjopu6ZYj7NmF3XXjx3un/RfQBHMPpkfQSADnXR012A/z7CvK/whzcqg9tH5PWdIP9RUXLf6upEy7KGUbdbI= X-Received: by 2002:a9d:7f11:: with SMTP id j17mr4694384otq.251.1614884502710; Thu, 04 Mar 2021 11:01:42 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> <20210304165923.GA60457@C02TD0UTHF1T.local> <20210304180154.GD60457@C02TD0UTHF1T.local> <20210304185148.GE60457@C02TD0UTHF1T.local> In-Reply-To: <20210304185148.GE60457@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 20:01:29 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Cc: Christophe Leroy , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , LKML , linuxppc-dev@lists.ozlabs.org, kasan-dev , Catalin Marinas , Will Deacon , Linux ARM , broonie@kernel.org, linux-toolchains@vger.kernel.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 4 Mar 2021 at 19:51, Mark Rutland wrote: > On Thu, Mar 04, 2021 at 07:22:53PM +0100, Marco Elver wrote: > > On Thu, 4 Mar 2021 at 19:02, Mark Rutland wrote: > > > On Thu, Mar 04, 2021 at 06:25:33PM +0100, Marco Elver wrote: > > > > On Thu, Mar 04, 2021 at 04:59PM +0000, Mark Rutland wrote: > > > > > On Thu, Mar 04, 2021 at 04:30:34PM +0100, Marco Elver wrote: > > > > > > On Thu, 4 Mar 2021 at 15:57, Mark Rutland wrote: > > > > > > > [adding Mark Brown] > > > > > > > > > > > > > > The bigger problem here is that skipping is dodgy to begin with, and > > > > > > > this is still liable to break in some cases. One big concern is that > > > > > > > (especially with LTO) we cannot guarantee the compiler will not inline > > > > > > > or outline functions, causing the skipp value to be too large or too > > > > > > > small. That's liable to happen to callers, and in theory (though > > > > > > > unlikely in practice), portions of arch_stack_walk() or > > > > > > > stack_trace_save() could get outlined too. > > > > > > > > > > > > > > Unless we can get some strong guarantees from compiler folk such that we > > > > > > > can guarantee a specific function acts boundary for unwinding (and > > > > > > > doesn't itself get split, etc), the only reliable way I can think to > > > > > > > solve this requires an assembly trampoline. Whatever we do is liable to > > > > > > > need some invasive rework. > > > > > > > > > > > > Will LTO and friends respect 'noinline'? > > > > > > > > > > I hope so (and suspect we'd have more problems otherwise), but I don't > > > > > know whether they actually so. > > > > > > > > > > I suspect even with 'noinline' the compiler is permitted to outline > > > > > portions of a function if it wanted to (and IIUC it could still make > > > > > specialized copies in the absence of 'noclone'). > > > > > > > > > > > One thing I also noticed is that tail calls would also cause the stack > > > > > > trace to appear somewhat incomplete (for some of my tests I've > > > > > > disabled tail call optimizations). > > > > > > > > > > I assume you mean for a chain A->B->C where B tail-calls C, you get a > > > > > trace A->C? ... or is A going missing too? > > > > > > > > Correct, it's just the A->C outcome. > > > > > > I'd assumed that those cases were benign, e.g. for livepatching what > > > matters is what can be returned to, so B disappearing from the trace > > > isn't a problem there. > > > > > > Is the concern debugability, or is there a functional issue you have in > > > mind? > > > > For me, it's just been debuggability, and reliable test cases. > > > > > > > > Is there a way to also mark a function non-tail-callable? > > > > > > > > > > I think this can be bodged using __attribute__((optimize("$OPTIONS"))) > > > > > on a caller to inhibit TCO (though IIRC GCC doesn't reliably support > > > > > function-local optimization options), but I don't expect there's any way > > > > > to mark a callee as not being tail-callable. > > > > > > > > I don't think this is reliable. It'd be > > > > __attribute__((optimize("-fno-optimize-sibling-calls"))), but doesn't > > > > work if applied to the function we do not want to tail-call-optimize, > > > > but would have to be applied to the function that does the tail-calling. > > > > > > Yup; that's what I meant then I said you could do that on the caller but > > > not the callee. > > > > > > I don't follow why you'd want to put this on the callee, though, so I > > > think I'm missing something. Considering a set of functions in different > > > compilation units: > > > > > > A->B->C->D->E->F->G->H->I->J->K > > > > I was having this problem with KCSAN, where the compiler would > > tail-call-optimize __tsan_X instrumentation. > > Those are compiler-generated calls, right? When those are generated the > compilation unit (and whatever it has included) might not have provided > a prototype anyway, and the compiler has special knowledge of the > functions, so it feels like the compiler would need to inhibit TCO here > for this to be robust. For their intended usage subjecting them to TCO > doesn't seem to make sense AFAICT. > > I suspect that compilers have some way of handling that; otherwise I'd > expect to have heard stories of mcount/fentry calls getting TCO'd and > causing problems. So maybe there's an easy fix there? I agree, the compiler builtins should be handled by the compiler directly, perhaps that was a bad example. But we also have "explicit instrumentation", e.g. everything that's in . > > This would mean that KCSAN runtime functions ended up in the trace, > > but the function where the access happened would not. However, I don't > > care about the runtime functions, and instead want to see the function > > where the access happened. In that case, I'd like to just mark > > __tsan_X and any other kcsan instrumentation functions as > > do-not-tail-call-optimize, which would solve the problem. > > I understand why we don't want to TCO these calls, but given the calls > are implicitly generated, I strongly suspect it's better to fix the > implicit call generation to not be TCO'd to begin with. > > > The solution today is that when you compile a kernel with KCSAN, every > > instrumented TU is compiled with -fno-optimize-sibling-calls. The > > better solution would be to just mark KCSAN runtime functions somehow, > > but permit tail calling other things. Although, I probably still want > > to see the full trace, and would decide that having > > -fno-optimize-sibling-calls is a small price to pay in a > > debug-only-kernel to get complete traces. > > > > > ... if K were marked in this way, and J was compiled with visibility of > > > this, J would stick around, but J's callers might not, and so the a > > > trace might see: > > > > > > A->J->K > > > > > > ... do you just care about the final caller, i.e. you just need > > > certainty that J will be in the trace? > > > > Yes. But maybe it's a special problem that only sanitizers have. > > I reckon for basically any instrumentation we don't want calls to be > TCO'd, though I'm not immediately sure of cases beyond sanitizers and > mcount/fentry. Thinking about this more, I think it's all debugging tools. E.g. lockdep, if you lock/unlock at the end of a function, you might tail call into lockdep. If the compiler applies TCO, and lockdep determines there's a bug and then shows a trace, you'll have no idea where the actual bug is. The kernel has lots of debugging facilities that add instrumentation in this way. So perhaps it's a general debugging-tool problem (rather than just sanitizers). Thanks, -- Marco _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel