From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39D10FA373E for ; Fri, 21 Oct 2022 16:50:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230118AbiJUQuA (ORCPT ); Fri, 21 Oct 2022 12:50:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229913AbiJUQty (ORCPT ); Fri, 21 Oct 2022 12:49:54 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1791FE09E6 for ; Fri, 21 Oct 2022 09:49:51 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d24so2874616pls.4 for ; Fri, 21 Oct 2022 09:49:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=/veItqXlMJIQTcpDjM5JBzQqgSWpElwpdVZBwKN6hJc=; b=II+6tRfI3o4IWM6bK4CLHQMdChvkiK85yyUuB33AlvXPnQlJOzyfUurZ9ghhEh4Bcp hK5caeuJ7efRWUGlErbXOLzT/G5qtuLcOjUS2UwGz9qG0IEhT1Rem3kQkeE9vE+VvDAo 2qn+M6JzvenfZRqzI6B1y+kW4+qXWJYeQzJuo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/veItqXlMJIQTcpDjM5JBzQqgSWpElwpdVZBwKN6hJc=; b=GQfgSjB94Pqe1Jk6xBUBNM2Zipf1J+r6bvNT3npkLUjT+M3aU1W9QeyWez95qMtIp1 1aBskKsGTkWA/ZzYJj7AYR0fQjmWr3XeTW5/wLohqy7drKYY5ARg+zv5sNr7JiDVpEw9 OkG7+KX94hnonF32JtIrURzIC6eCTbep5YV/yWj+wbL5SpVBJojCcDFwzBPKDaH1PPBF bVmEUk1CersgRsd9iyNXatZm42N7BicDeumv7M03tDA39c9/nIZA7JQ3e5vJPTA28ZlV Cf4Rkux/BHF1vGGrKqnogiXxd1cH2KH6mhStwJlSE7sFfnL73BHS9kZWknWSE0Ns6GUa CzmQ== X-Gm-Message-State: ACrzQf2npdr+aCzvRt/Xpeu5v3T4iHmusjFstWtNhHKYOYqFpufC/QuO K4nyKC+Qh+9ZtdzD0vLnjsCQNWJPphHhcrPBKYspmw== X-Google-Smtp-Source: AMsMyM492XaifRbV0Xp7jrZHgBHM2zCIypWIvrKBrEXC2Ld4rwIq+kUfF1WZv/Xcr9iHc3ynJsuZUvoHdq/WuuH47xA= X-Received: by 2002:a17:902:f1cc:b0:186:6670:e3da with SMTP id e12-20020a170902f1cc00b001866670e3damr8654642plc.41.1666370990207; Fri, 21 Oct 2022 09:49:50 -0700 (PDT) MIME-Version: 1.0 References: <20220913162732.163631-1-xukuohai@huaweicloud.com> <970a25e4-9b79-9e0c-b338-ed1a934f2770@huawei.com> <2cb606b4-aa8b-e259-cdfd-1bfc61fd7c44@huawei.com> <7f34d333-3b2a-aea5-f411-d53be2c46eee@huawei.com> <20221005110707.55bd9354@gandalf.local.home> <20221005113019.18aeda76@gandalf.local.home> <20221006122922.53802a5c@gandalf.local.home> <20221021203158.4464ac19d8b19b6da6a40852@kernel.org> In-Reply-To: <20221021203158.4464ac19d8b19b6da6a40852@kernel.org> From: Florent Revest Date: Fri, 21 Oct 2022 18:49:38 +0200 Message-ID: Subject: Re: [PATCH bpf-next v2 0/4] Add ftrace direct call for arm64 To: Masami Hiramatsu Cc: Steven Rostedt , Xu Kuohai , Mark Rutland , Catalin Marinas , Daniel Borkmann , Xu Kuohai , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, Will Deacon , Jean-Philippe Brucker , Ingo Molnar , Oleg Nesterov , Alexei Starovoitov , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Pasha Tatashin , Ard Biesheuvel , Marc Zyngier , Guo Ren Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 21, 2022 at 1:32 PM Masami Hiramatsu wrote: > On Mon, 17 Oct 2022 19:55:06 +0200 > Florent Revest wrote: > > Mark finished an implementation of his per-callsite-ops and min-args > > branches (meaning that we can now skip the expensive ftrace's saving > > of all registers and iteration over all ops if only one is attached) > > - https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64-ftrace-call-ops-20221017 > > > > And Masami wrote similar patches to what I had originally done to > > fprobe in my branch: > > - https://github.com/mhiramat/linux/commits/kprobes/fprobe-update > > > > So I could rebase my previous "bpf on fprobe" branch on top of these: > > (as before, it's just good enough for benchmarking and to give a > > general sense of the idea, not for a thorough code review): > > - https://github.com/FlorentRevest/linux/commits/fprobe-min-args-3 > > > > And I could run the benchmarks against my rpi4. I have different > > baseline numbers as Xu so I ran everything again and tried to keep the > > format the same. "indirect call" refers to my branch I just linked and > > "direct call" refers to the series this is a reply to (Xu's work) > > Thanks for sharing the measurement results. Yes, fprobes/rethook > implementation is just porting the kretprobes implementation, thus > it may not be so optimized. > > BTW, I remember Wuqiang's patch for kretprobes. > > https://lore.kernel.org/all/20210830173324.32507-1-wuqiang.matt@bytedance.com/T/#u Oh that's a great idea, thanks for pointing it out Masami! > This is for the scalability fixing, but may possible to improve > the performance a bit. It is not hard to port to the recent kernel. > Can you try it too? I rebased it on my branch https://github.com/FlorentRevest/linux/commits/fprobe-min-args-3 And I got measurements again. Unfortunately it looks like this does not help :/ New benchmark results: https://paste.debian.net/1257856/ New perf report: https://paste.debian.net/1257859/ The fprobe based approach is still significantly slower than the direct call approach. > Anyway, eventually, I would like to remove the current kretprobe > based implementation and unify fexit hook with function-graph > tracer. It should make more better perfromance on it. That makes sense. :) How do you imagine the unified solution ? Would both the fgraph and fprobe APIs keep existing but under the hood one would be implemented on the other ? (or would one be gone ?) Would we replace the rethook freelist with the function graph's per-task shadow stacks ? (or the other way around ?)) > > Note that I can't really make sense of the perf report with indirect > > calls. it always reports it spent 12% of the time in > > rethook_trampoline_handler but I verified with both a WARN in that > > function and a breakpoint with a debugger, this function does *not* > > get called when running this "bench trig-fentry" benchmark. Also it > > wouldn't make sense for fprobe_handler to call it so I'm quite > > confused why perf would report this call and such a long time spent > > there. Anyone know what I could be missing here ? I made slight progress on this. If I put the vmlinux file in the cwd where I run perf report, the reports no longer contain references to rethook_trampoline_handler. Instead, they have a few 0xffff800008xxxxxx addresses under fprobe_handler. (like in the pastebin I just linked) It's still pretty weird because that range is the vmalloc area on arm64 and I don't understand why anything under fprobe_handler would execute there. However, I'm also definitely sure that these 12% are actually spent getting buffers from the rethook memory pool because if I replace rethook_try_get and rethook_recycle calls with the usage of a dummy static bss buffer (for the sake of benchmarking the "theoretical best case scenario") these weird perf report traces are gone and the 12% are saved. https://paste.debian.net/1257862/ This is why I would be interested in seeing rethook's memory pool reimplemented on top of something like https://lwn.net/Articles/788923/ If we get closer to the performance of the the theoretical best case scenario where getting a blob of memory is ~free (and I think it could be the case with a per task shadow stack like fgraph's), then a bpf on fprobe implementation would start to approach the performances of a direct called trampoline on arm64: https://paste.debian.net/1257863/