From: Xu Kuohai <xukuohai@huawei.com> To: <bpf@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>, <linux-kselftest@vger.kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <ast@kernel.org>, Zi Shen Lim <zlim.lnx@gmail.com>, Andrii Nakryiko <andrii@kernel.org>, Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>, John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>, "David S . Miller" <davem@davemloft.net>, Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>, David Ahern <dsahern@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>, Shuah Khan <shuah@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Ard Biesheuvel <ardb@kernel.org>, Pasha Tatashin <pasha.tatashin@soleen.com>, Peter Collingbourne <pcc@google.com>, Daniel Kiss <daniel.kiss@arm.com>, Sudeep Holla <sudeep.holla@arm.com>, Steven Price <steven.price@arm.com>, Marc Zyngier <maz@kernel.org>, Mark Brown <broonie@kernel.org>, Kumar Kartikeya Dwivedi <memxor@gmail.com>, Delyan Kratunov <delyank@fb.com> Subject: [PATCH bpf-next v2 1/6] arm64: ftrace: Add ftrace direct call support Date: Thu, 14 Apr 2022 12:22:15 -0400 [thread overview] Message-ID: <20220414162220.1985095-2-xukuohai@huawei.com> (raw) In-Reply-To: <20220414162220.1985095-1-xukuohai@huawei.com> Add ftrace direct support for arm64. 1. When there is custom trampoline only, replace the fentry nop to a jump instruction that jumps directly to the custom trampoline. 2. When ftrace trampoline and custom trampoline coexist, jump from fentry to ftrace trampoline first, then jump to custom trampoline when ftrace trampoline exits. The current unused register pt_regs->orig_x0 is used as an intermediary for jumping from ftrace trampoline to custom trampoline. Signed-off-by: Xu Kuohai <xukuohai@huawei.com> Acked-by: Song Liu <songliubraving@fb.com> --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/ftrace.h | 10 ++++++++++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry-ftrace.S | 18 +++++++++++++++--- 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 57c4c995965f..81cc330daafc 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -177,6 +177,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ + if DYNAMIC_FTRACE_WITH_REGS select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \ if DYNAMIC_FTRACE_WITH_REGS select HAVE_EFFICIENT_UNALIGNED_ACCESS diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 1494cfa8639b..3a363d6a3bd0 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -78,6 +78,16 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) return addr; } +static inline void arch_ftrace_set_direct_caller(struct pt_regs *regs, + unsigned long addr) +{ + /* + * Place custom trampoline address in regs->orig_x0 to let ftrace + * trampoline jump to it. + */ + regs->orig_x0 = addr; +} + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS struct dyn_ftrace; int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1197e7679882..b1ed0bf01c59 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -80,6 +80,7 @@ int main(void) DEFINE(S_SDEI_TTBR1, offsetof(struct pt_regs, sdei_ttbr1)); DEFINE(S_PMR_SAVE, offsetof(struct pt_regs, pmr_save)); DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); + DEFINE(S_ORIG_X0, offsetof(struct pt_regs, orig_x0)); DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs)); BLANK(); #ifdef CONFIG_COMPAT diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index e535480a4069..dfe62c55e3a2 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -60,6 +60,9 @@ str x29, [sp, #S_FP] .endif + /* Set orig_x0 to zero */ + str xzr, [sp, #S_ORIG_X0] + /* Save the callsite's SP and LR */ add x10, sp, #(PT_REGS_SIZE + 16) stp x9, x10, [sp, #S_LR] @@ -119,12 +122,21 @@ ftrace_common_return: /* Restore the callsite's FP, LR, PC */ ldr x29, [sp, #S_FP] ldr x30, [sp, #S_LR] - ldr x9, [sp, #S_PC] - + ldr x10, [sp, #S_PC] + + ldr x11, [sp, #S_ORIG_X0] + cbz x11, 1f + /* Set x9 to parent ip before jump to custom trampoline */ + mov x9, x30 + /* Set lr to self ip */ + ldr x30, [sp, #S_PC] + /* Set x10 (used for return address) to custom trampoline */ + mov x10, x11 +1: /* Restore the callsite's SP */ add sp, sp, #PT_REGS_SIZE + 16 - ret x9 + ret x10 SYM_CODE_END(ftrace_common) #ifdef CONFIG_FUNCTION_GRAPH_TRACER -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
WARNING: multiple messages have this Message-ID (diff)
From: Xu Kuohai <xukuohai@huawei.com> To: <bpf@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>, <linux-kselftest@vger.kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <ast@kernel.org>, Zi Shen Lim <zlim.lnx@gmail.com>, Andrii Nakryiko <andrii@kernel.org>, Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>, John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>, "David S . Miller" <davem@davemloft.net>, Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>, David Ahern <dsahern@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>, Shuah Khan <shuah@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Ard Biesheuvel <ardb@kernel.org>, Pasha Tatashin <pasha.tatashin@soleen.com>, Peter Collingbourne <pcc@google.com>, Daniel Kiss <daniel.kiss@arm.com>, Sudeep Holla <sudeep.holla@arm.com>, Steven Price <steven.price@arm.com>, Marc Zyngier <maz@kernel.org>, Mark Brown <broonie@kernel.org>, Kumar Kartikeya Dwivedi <memxor@gmail.com>, Delyan Kratunov <delyank@fb.com> Subject: [PATCH bpf-next v2 1/6] arm64: ftrace: Add ftrace direct call support Date: Thu, 14 Apr 2022 12:22:15 -0400 [thread overview] Message-ID: <20220414162220.1985095-2-xukuohai@huawei.com> (raw) In-Reply-To: <20220414162220.1985095-1-xukuohai@huawei.com> Add ftrace direct support for arm64. 1. When there is custom trampoline only, replace the fentry nop to a jump instruction that jumps directly to the custom trampoline. 2. When ftrace trampoline and custom trampoline coexist, jump from fentry to ftrace trampoline first, then jump to custom trampoline when ftrace trampoline exits. The current unused register pt_regs->orig_x0 is used as an intermediary for jumping from ftrace trampoline to custom trampoline. Signed-off-by: Xu Kuohai <xukuohai@huawei.com> Acked-by: Song Liu <songliubraving@fb.com> --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/ftrace.h | 10 ++++++++++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/entry-ftrace.S | 18 +++++++++++++++--- 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 57c4c995965f..81cc330daafc 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -177,6 +177,8 @@ config ARM64 select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS \ if $(cc-option,-fpatchable-function-entry=2) + select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \ + if DYNAMIC_FTRACE_WITH_REGS select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \ if DYNAMIC_FTRACE_WITH_REGS select HAVE_EFFICIENT_UNALIGNED_ACCESS diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index 1494cfa8639b..3a363d6a3bd0 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -78,6 +78,16 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) return addr; } +static inline void arch_ftrace_set_direct_caller(struct pt_regs *regs, + unsigned long addr) +{ + /* + * Place custom trampoline address in regs->orig_x0 to let ftrace + * trampoline jump to it. + */ + regs->orig_x0 = addr; +} + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS struct dyn_ftrace; int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 1197e7679882..b1ed0bf01c59 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -80,6 +80,7 @@ int main(void) DEFINE(S_SDEI_TTBR1, offsetof(struct pt_regs, sdei_ttbr1)); DEFINE(S_PMR_SAVE, offsetof(struct pt_regs, pmr_save)); DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe)); + DEFINE(S_ORIG_X0, offsetof(struct pt_regs, orig_x0)); DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs)); BLANK(); #ifdef CONFIG_COMPAT diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index e535480a4069..dfe62c55e3a2 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -60,6 +60,9 @@ str x29, [sp, #S_FP] .endif + /* Set orig_x0 to zero */ + str xzr, [sp, #S_ORIG_X0] + /* Save the callsite's SP and LR */ add x10, sp, #(PT_REGS_SIZE + 16) stp x9, x10, [sp, #S_LR] @@ -119,12 +122,21 @@ ftrace_common_return: /* Restore the callsite's FP, LR, PC */ ldr x29, [sp, #S_FP] ldr x30, [sp, #S_LR] - ldr x9, [sp, #S_PC] - + ldr x10, [sp, #S_PC] + + ldr x11, [sp, #S_ORIG_X0] + cbz x11, 1f + /* Set x9 to parent ip before jump to custom trampoline */ + mov x9, x30 + /* Set lr to self ip */ + ldr x30, [sp, #S_PC] + /* Set x10 (used for return address) to custom trampoline */ + mov x10, x11 +1: /* Restore the callsite's SP */ add sp, sp, #PT_REGS_SIZE + 16 - ret x9 + ret x10 SYM_CODE_END(ftrace_common) #ifdef CONFIG_FUNCTION_GRAPH_TRACER -- 2.30.2
next prev parent reply other threads:[~2022-04-14 16:11 UTC|newest] Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-14 16:22 [PATCH bpf-next v2 0/6] bpf trampoline for arm64 Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai [this message] 2022-04-14 16:22 ` [PATCH bpf-next v2 1/6] arm64: ftrace: Add ftrace direct call support Xu Kuohai 2022-04-14 16:22 ` [PATCH bpf-next v2 2/6] ftrace: Fix deadloop caused by direct call in ftrace selftest Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-20 23:24 ` Steven Rostedt 2022-04-20 23:24 ` Steven Rostedt 2022-04-21 3:01 ` Xu Kuohai 2022-04-21 3:01 ` Xu Kuohai 2022-04-14 16:22 ` [PATCH bpf-next v2 3/6] bpf: Move is_valid_bpf_tramp_flags() to the public trampoline code Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-14 16:22 ` [PATCH bpf-next v2 4/6] bpf, arm64: Impelment bpf_arch_text_poke() for arm64 Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-15 2:34 ` Hou Tao 2022-04-15 2:34 ` Hou Tao 2022-04-15 3:39 ` Xu Kuohai 2022-04-15 3:39 ` Xu Kuohai 2022-04-22 10:54 ` Jakub Sitnicki 2022-04-22 10:54 ` Jakub Sitnicki 2022-04-24 5:05 ` Xu Kuohai 2022-04-24 5:05 ` Xu Kuohai 2022-04-25 14:26 ` Jakub Sitnicki 2022-04-25 14:26 ` Jakub Sitnicki 2022-04-26 4:01 ` Xu Kuohai 2022-04-26 4:01 ` Xu Kuohai 2022-04-14 16:22 ` [PATCH bpf-next v2 5/6] bpf, arm64: bpf trampoline " Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-15 17:12 ` Andrii Nakryiko 2022-04-15 17:12 ` Andrii Nakryiko 2022-04-16 1:57 ` Xu Kuohai 2022-04-16 1:57 ` Xu Kuohai 2022-04-20 7:43 ` Xu Kuohai 2022-04-20 7:43 ` Xu Kuohai 2022-04-20 11:42 ` KP Singh 2022-04-20 11:42 ` KP Singh 2022-04-14 16:22 ` [PATCH bpf-next v2 6/6] selftests/bpf: Fix trivial typo in fentry_fexit.c Xu Kuohai 2022-04-14 16:22 ` Xu Kuohai 2022-04-15 2:37 ` [PATCH bpf-next v2 0/6] bpf trampoline for arm64 Hou Tao 2022-04-15 2:37 ` Hou Tao 2022-04-15 3:47 ` Xu Kuohai 2022-04-15 3:47 ` Xu Kuohai
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220414162220.1985095-2-xukuohai@huawei.com \ --to=xukuohai@huawei.com \ --cc=andrii@kernel.org \ --cc=ardb@kernel.org \ --cc=ast@kernel.org \ --cc=bp@alien8.de \ --cc=bpf@vger.kernel.org \ --cc=broonie@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=daniel.kiss@arm.com \ --cc=daniel@iogearbox.net \ --cc=dave.hansen@linux.intel.com \ --cc=davem@davemloft.net \ --cc=delyank@fb.com \ --cc=dsahern@kernel.org \ --cc=hpa@zytor.com \ --cc=john.fastabend@gmail.com \ --cc=kafai@fb.com \ --cc=kpsingh@kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-kselftest@vger.kernel.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=memxor@gmail.com \ --cc=mingo@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=pasha.tatashin@soleen.com \ --cc=pcc@google.com \ --cc=rostedt@goodmis.org \ --cc=shuah@kernel.org \ --cc=songliubraving@fb.com \ --cc=steven.price@arm.com \ --cc=sudeep.holla@arm.com \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ --cc=x86@kernel.org \ --cc=yhs@fb.com \ --cc=yoshfuji@linux-ipv6.org \ --cc=zlim.lnx@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.