From: Xu Kuohai <xukuohai@huawei.com> To: <bpf@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>, <linux-kselftest@vger.kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <ast@kernel.org>, Zi Shen Lim <zlim.lnx@gmail.com>, Andrii Nakryiko <andrii@kernel.org>, Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>, John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>, "David S . Miller" <davem@davemloft.net>, Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>, David Ahern <dsahern@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>, Shuah Khan <shuah@kernel.org>, Jakub Kicinski <kuba@kernel.org>, Jesper Dangaard Brouer <hawk@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Pasha Tatashin <pasha.tatashin@soleen.com>, Ard Biesheuvel <ardb@kernel.org>, Daniel Kiss <daniel.kiss@arm.com>, Steven Price <steven.price@arm.com>, Sudeep Holla <sudeep.holla@arm.com>, Marc Zyngier <maz@kernel.org>, Peter Collingbourne <pcc@google.com>, Mark Brown <broonie@kernel.org>, Delyan Kratunov <delyank@fb.com>, Kumar Kartikeya Dwivedi <memxor@gmail.com> Subject: [PATCH bpf-next v3 5/7] bpf, arm64: Support to poke bpf prog Date: Sun, 24 Apr 2022 11:40:26 -0400 [thread overview] Message-ID: <20220424154028.1698685-6-xukuohai@huawei.com> (raw) In-Reply-To: <20220424154028.1698685-1-xukuohai@huawei.com> 1. Set up the bpf prog entry in the same way as fentry to support trampoline. Now bpf prog entry looks like this: bti c // if BTI enabled mov x9, x30 // save lr nop // to be replaced with jump instruction paciasp // if PAC enabled 2. Update bpf_arch_text_poke() to poke bpf prog. If the instruction to be poked is bpf prog's first instruction, skip to the nop instruction in the prog entry. Signed-off-by: Xu Kuohai <xukuohai@huawei.com> --- arch/arm64/net/bpf_jit.h | 1 + arch/arm64/net/bpf_jit_comp.c | 41 +++++++++++++++++++++++++++-------- 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index 194c95ccc1cf..1c4b0075a3e2 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -270,6 +270,7 @@ #define A64_BTI_C A64_HINT(AARCH64_INSN_HINT_BTIC) #define A64_BTI_J A64_HINT(AARCH64_INSN_HINT_BTIJ) #define A64_BTI_JC A64_HINT(AARCH64_INSN_HINT_BTIJC) +#define A64_NOP A64_HINT(AARCH64_INSN_HINT_NOP) /* DMB */ #define A64_DMB_ISH aarch64_insn_gen_dmb(AARCH64_INSN_MB_ISH) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 3f9bdfec54c4..293bdefc5d0c 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -237,14 +237,23 @@ static bool is_lsi_offset(int offset, int scale) return true; } -/* Tail call offset to jump into */ -#if IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) || \ - IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) -#define PROLOGUE_OFFSET 9 +#if IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) +#define BTI_INSNS 1 +#else +#define BTI_INSNS 0 +#endif + +#if IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) +#define PAC_INSNS 1 #else -#define PROLOGUE_OFFSET 8 +#define PAC_INSNS 0 #endif +/* Tail call offset to jump into */ +#define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) +/* Offset of nop instruction in bpf prog entry to be poked */ +#define POKE_OFFSET (BTI_INSNS + 1) + static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) { const struct bpf_prog *prog = ctx->prog; @@ -281,12 +290,15 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) * */ + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) + emit(A64_BTI_C, ctx); + + emit(A64_MOV(1, A64_R(9), A64_LR), ctx); + emit(A64_NOP, ctx); + /* Sign lr */ if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) emit(A64_PACIASP, ctx); - /* BTI landing pad */ - else if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - emit(A64_BTI_C, ctx); /* Save FP and LR registers to stay align with ARM64 AAPCS */ emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); @@ -1552,9 +1564,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, u32 old_insn; u32 new_insn; u32 replaced; + unsigned long offset = ~0UL; enum aarch64_insn_branch_type branch_type; + char namebuf[KSYM_NAME_LEN]; - if (!is_bpf_text_address((long)ip)) + if (!__bpf_address_lookup((unsigned long)ip, NULL, &offset, namebuf)) /* Only poking bpf text is supported. Since kernel function * entry is set up by ftrace, we reply on ftrace to poke kernel * functions. For kernel funcitons, bpf_arch_text_poke() is only @@ -1565,6 +1579,15 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, */ return -EINVAL; + /* bpf entry */ + if (offset == 0UL) + /* skip to the nop instruction in bpf prog entry: + * bti c // if BTI enabled + * mov x9, x30 + * nop + */ + ip = (u32 *)ip + POKE_OFFSET; + if (poke_type == BPF_MOD_CALL) branch_type = AARCH64_INSN_BRANCH_LINK; else -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
WARNING: multiple messages have this Message-ID (diff)
From: Xu Kuohai <xukuohai@huawei.com> To: <bpf@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org>, <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>, <linux-kselftest@vger.kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <ast@kernel.org>, Zi Shen Lim <zlim.lnx@gmail.com>, Andrii Nakryiko <andrii@kernel.org>, Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>, Yonghong Song <yhs@fb.com>, John Fastabend <john.fastabend@gmail.com>, KP Singh <kpsingh@kernel.org>, "David S . Miller" <davem@davemloft.net>, Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>, David Ahern <dsahern@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, <x86@kernel.org>, <hpa@zytor.com>, Shuah Khan <shuah@kernel.org>, Jakub Kicinski <kuba@kernel.org>, Jesper Dangaard Brouer <hawk@kernel.org>, Mark Rutland <mark.rutland@arm.com>, Pasha Tatashin <pasha.tatashin@soleen.com>, Ard Biesheuvel <ardb@kernel.org>, Daniel Kiss <daniel.kiss@arm.com>, Steven Price <steven.price@arm.com>, Sudeep Holla <sudeep.holla@arm.com>, Marc Zyngier <maz@kernel.org>, Peter Collingbourne <pcc@google.com>, Mark Brown <broonie@kernel.org>, Delyan Kratunov <delyank@fb.com>, Kumar Kartikeya Dwivedi <memxor@gmail.com> Subject: [PATCH bpf-next v3 5/7] bpf, arm64: Support to poke bpf prog Date: Sun, 24 Apr 2022 11:40:26 -0400 [thread overview] Message-ID: <20220424154028.1698685-6-xukuohai@huawei.com> (raw) In-Reply-To: <20220424154028.1698685-1-xukuohai@huawei.com> 1. Set up the bpf prog entry in the same way as fentry to support trampoline. Now bpf prog entry looks like this: bti c // if BTI enabled mov x9, x30 // save lr nop // to be replaced with jump instruction paciasp // if PAC enabled 2. Update bpf_arch_text_poke() to poke bpf prog. If the instruction to be poked is bpf prog's first instruction, skip to the nop instruction in the prog entry. Signed-off-by: Xu Kuohai <xukuohai@huawei.com> --- arch/arm64/net/bpf_jit.h | 1 + arch/arm64/net/bpf_jit_comp.c | 41 +++++++++++++++++++++++++++-------- 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index 194c95ccc1cf..1c4b0075a3e2 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -270,6 +270,7 @@ #define A64_BTI_C A64_HINT(AARCH64_INSN_HINT_BTIC) #define A64_BTI_J A64_HINT(AARCH64_INSN_HINT_BTIJ) #define A64_BTI_JC A64_HINT(AARCH64_INSN_HINT_BTIJC) +#define A64_NOP A64_HINT(AARCH64_INSN_HINT_NOP) /* DMB */ #define A64_DMB_ISH aarch64_insn_gen_dmb(AARCH64_INSN_MB_ISH) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 3f9bdfec54c4..293bdefc5d0c 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -237,14 +237,23 @@ static bool is_lsi_offset(int offset, int scale) return true; } -/* Tail call offset to jump into */ -#if IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) || \ - IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) -#define PROLOGUE_OFFSET 9 +#if IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) +#define BTI_INSNS 1 +#else +#define BTI_INSNS 0 +#endif + +#if IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL) +#define PAC_INSNS 1 #else -#define PROLOGUE_OFFSET 8 +#define PAC_INSNS 0 #endif +/* Tail call offset to jump into */ +#define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) +/* Offset of nop instruction in bpf prog entry to be poked */ +#define POKE_OFFSET (BTI_INSNS + 1) + static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) { const struct bpf_prog *prog = ctx->prog; @@ -281,12 +290,15 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) * */ + if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) + emit(A64_BTI_C, ctx); + + emit(A64_MOV(1, A64_R(9), A64_LR), ctx); + emit(A64_NOP, ctx); + /* Sign lr */ if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) emit(A64_PACIASP, ctx); - /* BTI landing pad */ - else if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - emit(A64_BTI_C, ctx); /* Save FP and LR registers to stay align with ARM64 AAPCS */ emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); @@ -1552,9 +1564,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, u32 old_insn; u32 new_insn; u32 replaced; + unsigned long offset = ~0UL; enum aarch64_insn_branch_type branch_type; + char namebuf[KSYM_NAME_LEN]; - if (!is_bpf_text_address((long)ip)) + if (!__bpf_address_lookup((unsigned long)ip, NULL, &offset, namebuf)) /* Only poking bpf text is supported. Since kernel function * entry is set up by ftrace, we reply on ftrace to poke kernel * functions. For kernel funcitons, bpf_arch_text_poke() is only @@ -1565,6 +1579,15 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, */ return -EINVAL; + /* bpf entry */ + if (offset == 0UL) + /* skip to the nop instruction in bpf prog entry: + * bti c // if BTI enabled + * mov x9, x30 + * nop + */ + ip = (u32 *)ip + POKE_OFFSET; + if (poke_type == BPF_MOD_CALL) branch_type = AARCH64_INSN_BRANCH_LINK; else -- 2.30.2
next prev parent reply other threads:[~2022-04-24 15:30 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-24 15:40 [PATCH bpf-next v3 0/7] bpf trampoline for arm64 Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-04-24 15:40 ` [PATCH bpf-next v3 1/7] arm64: ftrace: Add ftrace direct call support Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-04-24 15:40 ` [PATCH bpf-next v3 2/7] ftrace: Fix deadloop caused by direct call in ftrace selftest Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-04-25 15:05 ` Steven Rostedt 2022-04-25 15:05 ` Steven Rostedt 2022-04-26 7:36 ` Xu Kuohai 2022-04-26 7:36 ` Xu Kuohai 2022-04-24 15:40 ` [PATCH bpf-next v3 3/7] bpf: Move is_valid_bpf_tramp_flags() to the public trampoline code Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-04-24 15:40 ` [PATCH bpf-next v3 4/7] bpf, arm64: Impelment bpf_arch_text_poke() for arm64 Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-05-10 11:45 ` Jakub Sitnicki 2022-05-10 11:45 ` Jakub Sitnicki 2022-05-11 3:18 ` Xu Kuohai 2022-05-11 3:18 ` Xu Kuohai 2022-05-13 14:59 ` Mark Rutland 2022-05-13 14:59 ` Mark Rutland 2022-05-16 6:55 ` Xu Kuohai 2022-05-16 6:55 ` Xu Kuohai 2022-05-16 7:18 ` Mark Rutland 2022-05-16 7:18 ` Mark Rutland 2022-05-16 7:58 ` Xu Kuohai 2022-05-16 7:58 ` Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai [this message] 2022-04-24 15:40 ` [PATCH bpf-next v3 5/7] bpf, arm64: Support to poke bpf prog Xu Kuohai 2022-05-10 9:36 ` Jakub Sitnicki 2022-05-10 9:36 ` Jakub Sitnicki 2022-05-11 3:12 ` Xu Kuohai 2022-05-11 3:12 ` Xu Kuohai 2022-05-12 10:54 ` Jakub Sitnicki 2022-05-12 10:54 ` Jakub Sitnicki 2022-04-24 15:40 ` [PATCH bpf-next v3 6/7] bpf, arm64: bpf trampoline for arm64 Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai 2022-04-24 15:40 ` [PATCH bpf-next v3 7/7] selftests/bpf: Fix trivial typo in fentry_fexit.c Xu Kuohai 2022-04-24 15:40 ` Xu Kuohai
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220424154028.1698685-6-xukuohai@huawei.com \ --to=xukuohai@huawei.com \ --cc=andrii@kernel.org \ --cc=ardb@kernel.org \ --cc=ast@kernel.org \ --cc=bp@alien8.de \ --cc=bpf@vger.kernel.org \ --cc=broonie@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=daniel.kiss@arm.com \ --cc=daniel@iogearbox.net \ --cc=dave.hansen@linux.intel.com \ --cc=davem@davemloft.net \ --cc=delyank@fb.com \ --cc=dsahern@kernel.org \ --cc=hawk@kernel.org \ --cc=hpa@zytor.com \ --cc=john.fastabend@gmail.com \ --cc=kafai@fb.com \ --cc=kpsingh@kernel.org \ --cc=kuba@kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-kselftest@vger.kernel.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=memxor@gmail.com \ --cc=mingo@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=pasha.tatashin@soleen.com \ --cc=pcc@google.com \ --cc=rostedt@goodmis.org \ --cc=shuah@kernel.org \ --cc=songliubraving@fb.com \ --cc=steven.price@arm.com \ --cc=sudeep.holla@arm.com \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ --cc=x86@kernel.org \ --cc=yhs@fb.com \ --cc=yoshfuji@linux-ipv6.org \ --cc=zlim.lnx@gmail.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.