From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90E09C433FE for ; Mon, 17 Oct 2022 19:10:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229990AbiJQTKr (ORCPT ); Mon, 17 Oct 2022 15:10:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229778AbiJQTKn (ORCPT ); Mon, 17 Oct 2022 15:10:43 -0400 Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AB2A1A058 for ; Mon, 17 Oct 2022 12:10:42 -0700 (PDT) Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-12c8312131fso14374762fac.4 for ; Mon, 17 Oct 2022 12:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=s0HtQQEU1KD07cnfV5sqhhl7NC1nD6JeKl8vlD6X5bc=; b=J5ckQwk5f17OF8pQt66AjQUgpx2SA9UelXEToqmgB+xmj7dg9P51ytv03oHrB8CyF5 om0Q4KyUA14FQdFQpQ5+QiixeEV9pRRqN6t8OEr2bQHy+g43sDQwDNlQydb8ecf7aKok Mx4mSFKJN76E2vQop4KHqjsZG6lxMdK8ekteI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=s0HtQQEU1KD07cnfV5sqhhl7NC1nD6JeKl8vlD6X5bc=; b=PVlfeIvmCo/BYpom6IjX9SKJVoZXhFujLlRyuc4EKghN6Ow9pnen0Pz/WBHnVc5kue edt4Jzs6hMhgaDc1gypvSeywz8tguwZKkvIE0unbP0w7/mKhfh7GLZsYvl2/Dap7JLkp YdvQkjl18xD2iWYjaiT8WTPvVF2d+q0qAXjH6w0SMkLTX4w9bUUs4VaKAzAj4Lq8Bc1o HckEUuYNMnun9aS8xMQ6YoIss0P1gNYBw/URl5peuDsNfZ8D0NK9roS6LFBKNw6JyUd+ fOTGu0r3f+J4P/MtOTSR84H+drLJ4eKMaTS+a/m+PbSWlNRZKfnOLaGdO9GV5OYRfp/r Eejw== X-Gm-Message-State: ACrzQf1XKByM0FLmVBnaNuDK25NFZvzEH4QojVETj3kq/dwYEGx0Hk5y IGAYc9lcWpIVpy74QqgDQ9eM5O2ZP8QY62qYBEE5Q52B8k1YnQ== X-Google-Smtp-Source: AMsMyM7+SdOqBsDFuP6CDB2suF4jvA2csSIGb3xEPcNuj632CabHd6uGTXp96mMnVQYSqA0jiNYZWNpB7FENzjBkqEM= X-Received: by 2002:a17:90a:b903:b0:20d:a36b:6791 with SMTP id p3-20020a17090ab90300b0020da36b6791mr27474441pjr.26.1666033830692; Mon, 17 Oct 2022 12:10:30 -0700 (PDT) MIME-Version: 1.0 References: <20220913162732.163631-1-xukuohai@huaweicloud.com> <970a25e4-9b79-9e0c-b338-ed1a934f2770@huawei.com> <2cb606b4-aa8b-e259-cdfd-1bfc61fd7c44@huawei.com> <7f34d333-3b2a-aea5-f411-d53be2c46eee@huawei.com> <20221005110707.55bd9354@gandalf.local.home> <20221005113019.18aeda76@gandalf.local.home> <20221006122922.53802a5c@gandalf.local.home> <20221017144949.2b9dcdc5@gandalf.local.home> In-Reply-To: <20221017144949.2b9dcdc5@gandalf.local.home> From: Florent Revest Date: Mon, 17 Oct 2022 21:10:19 +0200 Message-ID: Subject: Re: [PATCH bpf-next v2 0/4] Add ftrace direct call for arm64 To: Steven Rostedt Cc: Xu Kuohai , Mark Rutland , Catalin Marinas , Daniel Borkmann , Xu Kuohai , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, Will Deacon , Jean-Philippe Brucker , Ingo Molnar , Oleg Nesterov , Alexei Starovoitov , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Pasha Tatashin , Ard Biesheuvel , Marc Zyngier , Guo Ren , Masami Hiramatsu Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Uhuh, apologies for my perf report formatting! I'll try to figure it out for next time, meanwhile you can find it better formatted here https://paste.debian.net/1257405/ On Mon, Oct 17, 2022 at 8:49 PM Steven Rostedt wrote: > > On Mon, 17 Oct 2022 19:55:06 +0200 > Florent Revest wrote: > > > Note that I can't really make sense of the perf report with indirect > > calls. it always reports it spent 12% of the time in > > rethook_trampoline_handler but I verified with both a WARN in that > > function and a breakpoint with a debugger, this function does *not* > > get called when running this "bench trig-fentry" benchmark. Also it > > wouldn't make sense for fprobe_handler to call it so I'm quite > > confused why perf would report this call and such a long time spent > > there. Anyone know what I could be missing here ? > > The trace shows __bpf_prog_exit, which I'm guessing is tracing the end of > the function. Right? Actually no, this function is called to end the context of a BPF program execution. Here it is called at the end of the fentry program (so still before the traced function). I hope the pastebin helps clarify this! > In which case I believe it must call rethook_trampoline_handler: > > -> fprobe_handler() /* Which could use some "unlikely()" to move disabled > paths out of the hot path */ > > /* And also calls rethook_try_get () which does a cmpxchg! */ > > -> ret_hook() > -> arch_rethook_prepare() > Sets regs->lr = arch_rethook_trampoline > > On return of the function, it jumps to arch_rethook_trampoline() > > -> arch_rethook_trampoline() > -> arch_rethook_trampoline_callback() > -> rethook_trampoline_handler() This is indeed what happens when an fexit program is also attached. But when running "bench trig-fentry", only an fentry program is attached so bpf_fprobe_entry returns a non-zero value and fprobe doesn't call rethook_hook. Also, in this situation arch_rethook_trampoline is called by the traced function's return but in the perf report, iiuc, it shows up as being called from fprobe_handler and that should never happen. I wonder if this is some sort of stack unwinding artifact during the perf record? > So I do not know how it wouldn't trigger the WARNING or breakpoint if you > added it there. By the way, the WARNING does trigger if I also attach an fexit program (then rethook_hook is called). But I made sure we skip the whole rethook logic if no fexit program is attached so bench trig-fentry should not go through rethook_trampoline_handler.