From: "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> To: Michael Ellerman <mpe@ellerman.id.au>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <alexei.starovoitov@gmail.com> Cc: Jiri Olsa <jolsa@redhat.com>, ykaliuta@redhat.com, Christophe Leroy <christophe.leroy@csgroup.eu>, song@kernel.org, johan.almbladh@anyfinetworks.com, Hari Bathini <hbathini@linux.ibm.com>, <bpf@vger.kernel.org>, <linuxppc-dev@lists.ozlabs.org> Subject: [PATCH 07/13] powerpc/bpf: Handle large branch ranges with BPF_EXIT Date: Thu, 6 Jan 2022 17:15:11 +0530 [thread overview] Message-ID: <e64c43e43f1ba8fbd38cda2b29fec3ad81a438f3.1641468127.git.naveen.n.rao@linux.vnet.ibm.com> (raw) In-Reply-To: <cover.1641468127.git.naveen.n.rao@linux.vnet.ibm.com> In some scenarios, it is possible that the program epilogue is outside the branch range for a BPF_EXIT instruction. Instead of rejecting such programs, emit epilogue as an alternate exit point from the program. Track the location of the same so that subsequent exits can take either of the two paths. Reported-by: Jordan Niethe <jniethe5@gmail.com> Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> --- arch/powerpc/net/bpf_jit.h | 2 ++ arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++++- arch/powerpc/net/bpf_jit_comp32.c | 7 +++++-- arch/powerpc/net/bpf_jit_comp64.c | 7 +++++-- 4 files changed, 33 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h index 9cdd33d6be4cc0..3b5c44c0b6638d 100644 --- a/arch/powerpc/net/bpf_jit.h +++ b/arch/powerpc/net/bpf_jit.h @@ -151,6 +151,7 @@ struct codegen_context { unsigned int stack_size; int b2p[ARRAY_SIZE(b2p)]; unsigned int exentry_idx; + unsigned int alt_exit_addr; }; #ifdef CONFIG_PPC32 @@ -186,6 +187,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx); void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); void bpf_jit_realloc_regs(struct codegen_context *ctx); +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr); int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx, int insn_idx, int jmp_off, int dst_reg); diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index 56dd1f4e3e4447..141e64585b6458 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -89,6 +89,22 @@ static int bpf_jit_fixup_addresses(struct bpf_prog *fp, u32 *image, return 0; } +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr) +{ + if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) { + PPC_JMP(exit_addr); + } else if (ctx->alt_exit_addr) { + if (WARN_ON(!is_offset_in_branch_range((long)ctx->alt_exit_addr - (ctx->idx * 4)))) + return -1; + PPC_JMP(ctx->alt_exit_addr); + } else { + ctx->alt_exit_addr = ctx->idx * 4; + bpf_jit_build_epilogue(image, ctx); + } + + return 0; +} + struct powerpc64_jit_data { struct bpf_binary_header *header; u32 *addrs; @@ -177,8 +193,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) * If we have seen a tail call, we need a second pass. * This is because bpf_jit_emit_common_epilogue() is called * from bpf_jit_emit_tail_call() with a not yet stable ctx->seen. + * We also need a second pass if we ended up with too large + * a program so as to ensure BPF_EXIT branches are in range. */ - if (cgctx.seen & SEEN_TAILCALL) { + if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) { cgctx.idx = 0; if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) { fp = org_fp; @@ -193,6 +211,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) * calculate total size from idx. */ bpf_jit_build_prologue(0, &cgctx); + addrs[fp->len] = cgctx.idx * 4; bpf_jit_build_epilogue(0, &cgctx); fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN * 4; @@ -233,6 +252,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) for (pass = 1; pass < 3; pass++) { /* Now build the prologue, body code & epilogue for real. */ cgctx.idx = 0; + cgctx.alt_exit_addr = 0; bpf_jit_build_prologue(code_base, &cgctx); if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) { bpf_jit_binary_free(bpf_hdr); diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c index 72c2c47612964d..8c918db4c2c486 100644 --- a/arch/powerpc/net/bpf_jit_comp32.c +++ b/arch/powerpc/net/bpf_jit_comp32.c @@ -929,8 +929,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * the epilogue. If we _are_ the last instruction, * we'll just fall through to the epilogue. */ - if (i != flen - 1) - PPC_JMP(exit_addr); + if (i != flen - 1) { + ret = bpf_jit_emit_exit_insn(image, ctx, _R0, exit_addr); + if (ret) + return ret; + } /* else fall through to the epilogue */ break; diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 2b291d435d2e26..48d2ca3fe126dd 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -867,8 +867,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * the epilogue. If we _are_ the last instruction, * we'll just fall through to the epilogue. */ - if (i != flen - 1) - PPC_JMP(exit_addr); + if (i != flen - 1) { + ret = bpf_jit_emit_exit_insn(image, ctx, b2p[TMP_REG_1], exit_addr); + if (ret) + return ret; + } /* else fall through to the epilogue */ break; -- 2.34.1
WARNING: multiple messages have this Message-ID (diff)
From: "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> To: Michael Ellerman <mpe@ellerman.id.au>, Daniel Borkmann <daniel@iogearbox.net>, Alexei Starovoitov <alexei.starovoitov@gmail.com> Cc: ykaliuta@redhat.com, johan.almbladh@anyfinetworks.com, linuxppc-dev@lists.ozlabs.org, song@kernel.org, bpf@vger.kernel.org, Jiri Olsa <jolsa@redhat.com>, Hari Bathini <hbathini@linux.ibm.com> Subject: [PATCH 07/13] powerpc/bpf: Handle large branch ranges with BPF_EXIT Date: Thu, 6 Jan 2022 17:15:11 +0530 [thread overview] Message-ID: <e64c43e43f1ba8fbd38cda2b29fec3ad81a438f3.1641468127.git.naveen.n.rao@linux.vnet.ibm.com> (raw) In-Reply-To: <cover.1641468127.git.naveen.n.rao@linux.vnet.ibm.com> In some scenarios, it is possible that the program epilogue is outside the branch range for a BPF_EXIT instruction. Instead of rejecting such programs, emit epilogue as an alternate exit point from the program. Track the location of the same so that subsequent exits can take either of the two paths. Reported-by: Jordan Niethe <jniethe5@gmail.com> Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> --- arch/powerpc/net/bpf_jit.h | 2 ++ arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++++- arch/powerpc/net/bpf_jit_comp32.c | 7 +++++-- arch/powerpc/net/bpf_jit_comp64.c | 7 +++++-- 4 files changed, 33 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h index 9cdd33d6be4cc0..3b5c44c0b6638d 100644 --- a/arch/powerpc/net/bpf_jit.h +++ b/arch/powerpc/net/bpf_jit.h @@ -151,6 +151,7 @@ struct codegen_context { unsigned int stack_size; int b2p[ARRAY_SIZE(b2p)]; unsigned int exentry_idx; + unsigned int alt_exit_addr; }; #ifdef CONFIG_PPC32 @@ -186,6 +187,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx); void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx); void bpf_jit_realloc_regs(struct codegen_context *ctx); +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr); int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx, int insn_idx, int jmp_off, int dst_reg); diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index 56dd1f4e3e4447..141e64585b6458 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -89,6 +89,22 @@ static int bpf_jit_fixup_addresses(struct bpf_prog *fp, u32 *image, return 0; } +int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr) +{ + if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) { + PPC_JMP(exit_addr); + } else if (ctx->alt_exit_addr) { + if (WARN_ON(!is_offset_in_branch_range((long)ctx->alt_exit_addr - (ctx->idx * 4)))) + return -1; + PPC_JMP(ctx->alt_exit_addr); + } else { + ctx->alt_exit_addr = ctx->idx * 4; + bpf_jit_build_epilogue(image, ctx); + } + + return 0; +} + struct powerpc64_jit_data { struct bpf_binary_header *header; u32 *addrs; @@ -177,8 +193,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) * If we have seen a tail call, we need a second pass. * This is because bpf_jit_emit_common_epilogue() is called * from bpf_jit_emit_tail_call() with a not yet stable ctx->seen. + * We also need a second pass if we ended up with too large + * a program so as to ensure BPF_EXIT branches are in range. */ - if (cgctx.seen & SEEN_TAILCALL) { + if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) { cgctx.idx = 0; if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) { fp = org_fp; @@ -193,6 +211,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) * calculate total size from idx. */ bpf_jit_build_prologue(0, &cgctx); + addrs[fp->len] = cgctx.idx * 4; bpf_jit_build_epilogue(0, &cgctx); fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN * 4; @@ -233,6 +252,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) for (pass = 1; pass < 3; pass++) { /* Now build the prologue, body code & epilogue for real. */ cgctx.idx = 0; + cgctx.alt_exit_addr = 0; bpf_jit_build_prologue(code_base, &cgctx); if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) { bpf_jit_binary_free(bpf_hdr); diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c index 72c2c47612964d..8c918db4c2c486 100644 --- a/arch/powerpc/net/bpf_jit_comp32.c +++ b/arch/powerpc/net/bpf_jit_comp32.c @@ -929,8 +929,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * the epilogue. If we _are_ the last instruction, * we'll just fall through to the epilogue. */ - if (i != flen - 1) - PPC_JMP(exit_addr); + if (i != flen - 1) { + ret = bpf_jit_emit_exit_insn(image, ctx, _R0, exit_addr); + if (ret) + return ret; + } /* else fall through to the epilogue */ break; diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 2b291d435d2e26..48d2ca3fe126dd 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -867,8 +867,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * * the epilogue. If we _are_ the last instruction, * we'll just fall through to the epilogue. */ - if (i != flen - 1) - PPC_JMP(exit_addr); + if (i != flen - 1) { + ret = bpf_jit_emit_exit_insn(image, ctx, b2p[TMP_REG_1], exit_addr); + if (ret) + return ret; + } /* else fall through to the epilogue */ break; -- 2.34.1
next prev parent reply other threads:[~2022-01-06 11:46 UTC|newest] Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-06 11:45 [PATCH 00/13] powerpc/bpf: Some fixes and updates Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 01/13] bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack() Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-07 10:21 ` Daniel Borkmann 2022-01-07 10:21 ` Daniel Borkmann 2022-01-10 8:57 ` Christophe Leroy 2022-01-10 8:57 ` Christophe Leroy 2022-01-10 10:36 ` Naveen N. Rao 2022-01-10 10:36 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 02/13] powerpc32/bpf: Fix codegen for bpf-to-bpf calls Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-10 9:06 ` Christophe Leroy 2022-01-10 9:06 ` Christophe Leroy 2022-01-10 10:52 ` Naveen N. Rao 2022-01-10 10:52 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 03/13] powerpc/bpf: Update ldimm64 instructions during extra pass Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-08 14:45 ` Jiri Olsa 2022-01-08 14:45 ` Jiri Olsa 2022-01-10 9:27 ` Christophe Leroy 2022-01-10 9:27 ` Christophe Leroy 2022-01-10 10:56 ` Naveen N. Rao 2022-01-10 10:56 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 04/13] tools/bpf: Rename 'struct event' to avoid naming conflict Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-07 10:21 ` Daniel Borkmann 2022-01-07 10:21 ` Daniel Borkmann 2022-01-06 11:45 ` [PATCH 05/13] powerpc/bpf: Skip branch range validation during first pass Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 06/13] powerpc/bpf: Emit a single branch instruction for known short branch ranges Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao [this message] 2022-01-06 11:45 ` [PATCH 07/13] powerpc/bpf: Handle large branch ranges with BPF_EXIT Naveen N. Rao 2022-01-06 11:45 ` [PATCH 08/13] powerpc64/bpf: Limit 'ldbrx' to processors compliant with ISA v2.06 Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 09/13] powerpc64/bpf: Do not save/restore LR on each call to bpf_stf_barrier() Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 10/13] powerpc64/bpf: Use r12 for constant blinding Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 11/13] powerpc64/bpf elfv2: Setup kernel TOC in r2 on entry Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-10 9:20 ` Christophe Leroy 2022-01-10 9:20 ` Christophe Leroy 2022-01-11 10:31 ` Naveen N. Rao 2022-01-11 10:31 ` Naveen N. Rao 2022-01-11 14:35 ` Christophe Leroy 2022-01-11 14:35 ` Christophe Leroy 2022-01-11 14:43 ` Christophe Leroy 2022-01-14 11:17 ` Naveen N. Rao 2022-01-14 11:17 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 12/13] powerpc64/bpf elfv1: Do not load TOC before calling functions Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 11:45 ` [PATCH 13/13] powerpc64/bpf: Optimize instruction sequence used for function calls Naveen N. Rao 2022-01-06 11:45 ` Naveen N. Rao 2022-01-06 21:46 ` [PATCH 00/13] powerpc/bpf: Some fixes and updates Daniel Borkmann 2022-01-06 21:46 ` Daniel Borkmann 2022-01-07 7:36 ` Naveen N. Rao 2022-01-07 7:36 ` Naveen N. Rao 2022-01-07 10:20 ` Daniel Borkmann 2022-01-07 10:20 ` Daniel Borkmann 2022-01-10 3:47 ` Michael Ellerman 2022-01-10 3:47 ` Michael Ellerman 2022-01-16 10:41 ` Michael Ellerman 2022-01-16 10:41 ` Michael Ellerman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=e64c43e43f1ba8fbd38cda2b29fec3ad81a438f3.1641468127.git.naveen.n.rao@linux.vnet.ibm.com \ --to=naveen.n.rao@linux.vnet.ibm.com \ --cc=alexei.starovoitov@gmail.com \ --cc=bpf@vger.kernel.org \ --cc=christophe.leroy@csgroup.eu \ --cc=daniel@iogearbox.net \ --cc=hbathini@linux.ibm.com \ --cc=johan.almbladh@anyfinetworks.com \ --cc=jolsa@redhat.com \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=mpe@ellerman.id.au \ --cc=song@kernel.org \ --cc=ykaliuta@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.