From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8657EC636D4 for ; Mon, 6 Feb 2023 19:44:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230075AbjBFTow (ORCPT ); Mon, 6 Feb 2023 14:44:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230048AbjBFTov (ORCPT ); Mon, 6 Feb 2023 14:44:51 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88F4B1ABEE for ; Mon, 6 Feb 2023 11:44:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3311DB810F2 for ; Mon, 6 Feb 2023 19:44:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55251C4339B; Mon, 6 Feb 2023 19:44:46 +0000 (UTC) Date: Mon, 6 Feb 2023 14:44:44 -0500 From: Steven Rostedt To: Indu Bhagat Cc: linux-toolchains@vger.kernel.org, "Jose E. Marchesi" , Daan De Meyer , Kris Van Hees , Elena Zannoni , Ross Zwisler Subject: Re: Unwinding user-space programs in the kernel using SFrame fo Message-ID: <20230206144444.01862598@rorschach.local.home> In-Reply-To: <7dcae1d1-b0f5-a497-a473-26a43f1b1ad6@oracle.com> References: <7dcae1d1-b0f5-a497-a473-26a43f1b1ad6@oracle.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-toolchains@vger.kernel.org On Thu, 12 Jan 2023 12:30:26 -0800 Indu Bhagat wrote: > Hello, > > This email is to initiate discussion/collaboration on adding a new > user-space program unwinder in the kernel, an unwinder which uses the > SFrame format. This is just an FYI that I was looking to have this done too in a generic manner that perf, ftrace and BPF could use it (and anyone else). I sent a proposal about it to LSF/MM/BPF summit: https://lore.kernel.org/all/20230206103828.6efcb28f@rorschach.local.home/ -- Steve > > What is SFrame format? > SFrame is the Simple Frame format. It represents the minimal necessary > information needed for backtracing - i.e. Canonical Frame Address (CFA), > Frame Pointer (FP), and Return Address (RA). SFrame unwind information > is available in a section called .sframe, which is itself presented in a > new segment of its own, PT_GNU_SFRAME. SFrame format is supported for > AMD64 and AARCH64 (be/le) ABIs only. > > How can I experiment with the SFrame format support? > The support for SFrame format is available in binutils trunk. GNU > assembler when passed a --gsframe command line option, generates the > .sframe section. The GNU assembler uses the .cfi_* asm directives > emitted by the compiler to generate an .sframe section. GNU ld merges > the input .sframe sections as necessary, no explicit command line option > is needed. There is support in objdump/readelf as well, pass a --sframe > option to dump the .sframe section in textual format. > > Where can I find details about the format? > More details are available in the include/sframe.h in binutils repo > (https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=include/sframe.h). > SFrame spec is also present in the binutils trunk. Some more content > should be available online in the form of GNU Cauldron and LPC 2022 > presentations: this was talked about under the name "CTF Frame", but has > since been renamed to "SFrame". > > Why is SFrame based unwinder useful? > Having an unwinder for user-space programs based on SFrame format can be > useful: > - enabling -fno-omit-frame-pointers has performance implications and > other issues. > - Compared to .eh_frame info, SFrame is a simpler format to decode > and generate backtraces. SFrame unwinder itself, hence, is small and > simple > (https://github.com/oracle/binutils-gdb/blob/oracle/sframe-unwinder/libsframe/sframe-backtrace.c > is how an SFrame based unwinder can look like. This code uses libsframe > APIs like sframe_decode, sframe_find_fre, sframe_fre_get_fp_offset etc. > to generate backtraces.). > > There was some interest, at LPC 2022, in exploring an SFrame-based > userspace unwinder for the kernel. To get started on that, some > discussion on following items will be great. (Please feel free to > add/delete/correct any items; my knowledge about the kernel and its > internals remains limited). > > Userspace unwinder selection > ---------------------------- > IIUC, userspace unwinding is always frame-pointer based in the kernel. > This is unlike kernel-space unwinding where there are a set of unwinders > to chose from: say, for x86_64, UNWINDER_ORC / UNWINDER_FRAME_POINTER / > UNWINDER_GUESS. Additionally, for kernel stack unwinding, there is also > a framework in place to plug-and-play these different unwinders. > > For userspace stack unwinding, first, we may want to add new config > options, such that: > - USERSPACE_UNWINDER_SFRAME => This option enables the SFrame > unwinder for unwinding user stack traces as the first choice. User > programs must be built with SFrame support. If not, no SFrame section > will be present in the user program binary; In such a case, the > userspace unwinder defaults to frame pointer unwinding. > - USERSPACE_UNWINDER_FRAME_POINTER => userspace unwinding does frame > pointer based unwinding only. User programs must be built with frame > pointer preservation build flags to ensure useful stack traces. > > Second, regarding "the framework" needed for non-frame-pointer-based > unwinders, more thought is needed. > > Interface of the userspace unwinder > ----------------------------------- > * OPTION 1 > This one might be overly simplified but is an option. We add the > following stub: > > ... > if (check_sframe_state_p (current)) // checks for SFrame sections if > CONFIG_USER_UNWINDER_SFRAME is true > sframe_callchain_user (entry, regs); // current is implicit, > stores callchain entries as it unwinds using .sframe sections > ... > > in the following target APIs in x86_64 and aarch64 to give the desired > effect of "userspace unwinder selection" > -- perf_callchain_user in arch/x86/events/core.c > -- perf_callchain_user in in arch/arm64/kernel/perf_callchain.c > > where the functions look like: > static inline bool check_sframe_state_p(struct task_struct *task); > void sframe_callchain_user(struct perf_callchain_entry_ctx *entry, > struct pt_regs *regs); > > Here, sframe_callchain_user () will, first, perform an operation similar > to dl_iterate_phdr, because we need the location of the SFrame sections > for unwinding. This means, for every sframe_unwind() call, we go over > the memory mappings of "current" task_struct and find the locations of > the .sframe sections of the program + its DSOs from pages that contain > the ELF program headers. Next, using these SFrame sections, it will then > decode the SFrame section and unwind. > > * OPTION 2 > A possible optimization is to instead: > > 1. Cache some sframe related state, "struct *sframe_state", in the > "struct task_struct" (guarded by CONFIG_USER_UNWINDER_SFRAME), and > 2. Use an API like so "void sframe_callchain_user(struct *sframe_state, > struct perf_callchain_entry_ctx *entry, struct pt_regs *regs)" > This state (struct sframe_state) is simply put: data about the size and > addr of the text and SFrame segments of the program and its DSOs. > Ideally this state can be setup at task setup time and needs to be > updated only if there is any change in the DSOs (added or removed) [1]. > The size of the struct sframe_state itself is small here, as SFrame > sections can be decoded on-the-fly with no need for additional mallocs. > > [1] PS: That this detection of "add/delete of DSOs in a user program" is > possible in some efficient way in the kernel remains an assumption; I > still need to figure things out. Any inputs on this appreciated. > > Other framework > --------------- > The kernel stack unwinders adhere to some interface allowing them to be > used interchangeably. The requirements of the userspace unwinder are a > bit different though: not all user applications may be compiled with > SFrame support, which means there needs to be a way we fall back on the > frame-pointer based unwinder in the kernel for unwinding user programs. > > This requirement, however, does not mean that some framework changes > shouldn't be done now to make things work better. > > Any feedback/ideas are appreciated. I have also not been able yet to > evaluate what other impacts could this have on perf, if at all. > > Thanks