* [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support @ 2019-12-09 17:31 Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Björn Töpel ` (8 more replies) 0 siblings, 9 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf Hi! This series contain one non-critical fix, support for far jumps. and some optimizations for the BPF JIT. Previously, the JIT only supported 12b branch targets for conditional branches, and 21b for unconditional branches. Starting with this series, 32b branching is supported. As part of supporting far jumps, branch relaxation was introduced. The idea is to start with a pessimistic jump (e.g. auipc/jalr) and for each pass the JIT will have an opportunity to pick a better instruction (e.g. jal) and shrink the image. Instead of two passes, the JIT requires more passes. It typically converges after 3 passes. The optimizations mentioned in the subject are for calls and tail calls. In the tail call generation we can save one instruction by using the offset in jalr. Calls are optimized by doing (auipc)/jal(r) relative jumps instead of loading the entire absolute address and doing jalr. This required that the JIT image allocator was made RISC-V specific, so we can ensure that the JIT image and the kernel text are in range (32b). The last two patches of the series is not critical to the series, but are two UAPI build issues for BPF events. A closer look from the RV-folks would be much appreciated. The test_bpf.ko module and test_verifier from selftests pass all tests. RISC-V is still missing proper kprobe and tracepoint support, so a lot of BPF selftests cannot be run. Thanks, Björn Björn Töpel (8): riscv, bpf: fix broken BPF tail calls riscv, bpf: add support for far branching riscv, bpf: add support for far jumps and exits riscv, bpf: optimize BPF tail calls riscv, bpf: provide RISC-V specific JIT image alloc/free riscv, bpf: optimize calls riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs riscv, perf: add arch specific perf_arch_bpf_user_pt_regs arch/riscv/include/asm/perf_event.h | 4 + arch/riscv/include/asm/pgtable.h | 4 + arch/riscv/include/uapi/asm/bpf_perf_event.h | 9 + arch/riscv/net/bpf_jit_comp.c | 511 +++++++++++-------- tools/include/uapi/asm/bpf_perf_event.h | 2 + 5 files changed, 310 insertions(+), 220 deletions(-) create mode 100644 arch/riscv/include/uapi/asm/bpf_perf_event.h -- 2.20.1 ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 2/8] riscv, bpf: add support for far branching Björn Töpel ` (7 subsequent siblings) 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf The BPF JIT incorrectly clobbered the a0 register, and did not flag usage of s5 register when BPF stack was being used. Fixes: 2353ecc6f91f ("bpf, riscv: add BPF JIT for RV64G") Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/net/bpf_jit_comp.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 5451ef3845f2..1606ebd49666 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -120,6 +120,11 @@ static bool seen_reg(int reg, struct rv_jit_context *ctx) return false; } +static void mark_fp(struct rv_jit_context *ctx) +{ + __set_bit(RV_CTX_F_SEEN_S5, &ctx->flags); +} + static void mark_call(struct rv_jit_context *ctx) { __set_bit(RV_CTX_F_SEEN_CALL, &ctx->flags); @@ -596,7 +601,8 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) emit(rv_addi(RV_REG_SP, RV_REG_SP, stack_adjust), ctx); /* Set return value. */ - emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); + if (reg == RV_REG_RA) + emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); } @@ -1426,6 +1432,10 @@ static void build_prologue(struct rv_jit_context *ctx) { int stack_adjust = 0, store_offset, bpf_stack_adjust; + bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); + if (bpf_stack_adjust) + mark_fp(ctx); + if (seen_reg(RV_REG_RA, ctx)) stack_adjust += 8; stack_adjust += 8; /* RV_REG_FP */ @@ -1443,7 +1453,6 @@ static void build_prologue(struct rv_jit_context *ctx) stack_adjust += 8; stack_adjust = round_up(stack_adjust, 16); - bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); stack_adjust += bpf_stack_adjust; store_offset = stack_adjust - 8; -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 21:08 ` Luke Nelson 2019-12-10 5:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits Björn Töpel ` (6 subsequent siblings) 8 siblings, 2 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf This commit adds branch relaxation to the BPF JIT, and with that support for far (offset greater than 12b) branching. The branch relaxation requires more than two passes to converge. For most programs it is three passes, but for larger programs it can be more. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/net/bpf_jit_comp.c | 352 ++++++++++++++++++---------------- 1 file changed, 188 insertions(+), 164 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 1606ebd49666..e2f6724b64f7 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -461,6 +461,11 @@ static u32 rv_amoadd_d(u8 rd, u8 rs2, u8 rs1, u8 aq, u8 rl) return rv_amo_insn(0, aq, rl, rs2, rs1, 3, rd, 0x2f); } +static u32 rv_auipc(u8 rd, u32 imm31_12) +{ + return rv_u_insn(imm31_12, rd, 0x17); +} + static bool is_12b_int(s64 val) { return -(1 << 11) <= val && val < (1 << 11); @@ -484,7 +489,7 @@ static bool is_32b_int(s64 val) static int is_12b_check(int off, int insn) { if (!is_12b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 12b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -494,7 +499,7 @@ static int is_12b_check(int off, int insn) static int is_13b_check(int off, int insn) { if (!is_13b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 13b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -504,7 +509,7 @@ static int is_13b_check(int off, int insn) static int is_21b_check(int off, int insn) { if (!is_21b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 21b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -550,10 +555,13 @@ static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) emit(rv_addi(rd, rd, lower), ctx); } -static int rv_offset(int bpf_to, int bpf_from, struct rv_jit_context *ctx) +static int rv_offset(int insn, int off, struct rv_jit_context *ctx) { - int from = ctx->offset[bpf_from] - 1, to = ctx->offset[bpf_to]; + int from, to; + off++; /* BPF branch is from PC+1, RV is from PC */ + from = (insn > 0) ? ctx->offset[insn - 1] : 0; + to = (insn + off > 0) ? ctx->offset[insn + off - 1] : 0; return (to - from) << 2; } @@ -693,13 +701,6 @@ static void init_regs(u8 *rd, u8 *rs, const struct bpf_insn *insn, *rs = bpf_to_rv_reg(insn->src_reg, ctx); } -static int rv_offset_check(int *rvoff, s16 off, int insn, - struct rv_jit_context *ctx) -{ - *rvoff = rv_offset(insn + off, insn, ctx); - return is_13b_check(*rvoff, insn); -} - static void emit_zext_32_rd_rs(u8 *rd, u8 *rs, struct rv_jit_context *ctx) { emit(rv_addi(RV_REG_T2, *rd, 0), ctx); @@ -732,13 +733,122 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } +/* return -1 or inverted cond */ +static int invert_bpf_cond(u8 cond) +{ + switch (cond) { + case BPF_JEQ: + return BPF_JNE; + case BPF_JGT: + return BPF_JLE; + case BPF_JLT: + return BPF_JGE; + case BPF_JGE: + return BPF_JLT; + case BPF_JLE: + return BPF_JGT; + case BPF_JNE: + return BPF_JEQ; + case BPF_JSGT: + return BPF_JSLE; + case BPF_JSLT: + return BPF_JSGE; + case BPF_JSGE: + return BPF_JSLT; + case BPF_JSLE: + return BPF_JSGT; + } + return -1; +} + +static void emit_bcc(u8 cond, u8 rd, u8 rs, int rvoff, + struct rv_jit_context *ctx) +{ + switch (cond) { + case BPF_JEQ: + emit(rv_beq(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JGT: + emit(rv_bltu(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JLT: + emit(rv_bltu(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JGE: + emit(rv_bgeu(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JLE: + emit(rv_bgeu(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JNE: + emit(rv_bne(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSGT: + emit(rv_blt(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JSLT: + emit(rv_blt(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSGE: + emit(rv_bge(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSLE: + emit(rv_bge(rs, rd, rvoff >> 1), ctx); + } +} + +static void emit_branch(u8 cond, u8 rd, u8 rs, int insn, int rvoff, + struct rv_jit_context *ctx) +{ + s64 upper, lower; + + if (is_13b_int(rvoff)) { + emit_bcc(cond, rd, rs, rvoff, ctx); + return; + } + + /* Adjust for jal */ + rvoff -= 4; + + /* Transform, e.g.: + * bne rd,rs,foo + * to + * beq rd,rs,<.L1> + * (auipc foo) + * jal(r) foo + * .L1 + */ + cond = invert_bpf_cond(cond); + if (is_21b_int(rvoff)) { + emit_bcc(cond, rd, rs, 8, ctx); + emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + return; + } + + /* 32b No need for an additional rvoff adjustment, since we + * get that from the auipc at PC', where PC = PC' + 4. + */ + upper = (rvoff + (1 << 11)) >> 12; + lower = rvoff & 0xfff; + + emit_bcc(cond, rd, rs, 12, ctx); + emit(rv_auipc(RV_REG_T1, upper), ctx); + emit(rv_jalr(RV_REG_ZERO, RV_REG_T1, lower), ctx); +} + +static bool is_signed_bpf_cond(u8 cond) +{ + return cond == BPF_JSGT || cond == BPF_JSLT || + cond == BPF_JSGE || cond == BPF_JSLE; +} + static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, bool extra_pass) { bool is64 = BPF_CLASS(insn->code) == BPF_ALU64 || BPF_CLASS(insn->code) == BPF_JMP; + int s, e, rvoff, i = insn - ctx->prog->insnsi; struct bpf_prog_aux *aux = ctx->prog->aux; - int rvoff, i = insn - ctx->prog->insnsi; u8 rd = -1, rs = -1, code = insn->code; s16 off = insn->off; s32 imm = insn->imm; @@ -1006,7 +1116,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: - rvoff = rv_offset(i + off, i, ctx); + rvoff = rv_offset(i, off, ctx); if (!is_21b_int(rvoff)) { pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", i, rvoff); @@ -1019,194 +1129,96 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* IF (dst COND src) JUMP off */ case BPF_JMP | BPF_JEQ | BPF_X: case BPF_JMP32 | BPF_JEQ | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_beq(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGT | BPF_X: case BPF_JMP32 | BPF_JGT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bltu(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLT | BPF_X: case BPF_JMP32 | BPF_JLT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bltu(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGE | BPF_X: case BPF_JMP32 | BPF_JGE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bgeu(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLE | BPF_X: case BPF_JMP32 | BPF_JLE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bgeu(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JNE | BPF_X: case BPF_JMP32 | BPF_JNE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bne(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGT | BPF_X: case BPF_JMP32 | BPF_JSGT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_blt(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLT | BPF_X: case BPF_JMP32 | BPF_JSLT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_blt(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGE | BPF_X: case BPF_JMP32 | BPF_JSGE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bge(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLE | BPF_X: case BPF_JMP32 | BPF_JSLE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bge(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSET | BPF_X: case BPF_JMP32 | BPF_JSET | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_and(RV_REG_T1, rd, rs), ctx); - emit(rv_bne(RV_REG_T1, RV_REG_ZERO, rvoff >> 1), ctx); + rvoff = rv_offset(i, off, ctx); + if (!is64) { + s = ctx->ninsns; + if (is_signed_bpf_cond(BPF_OP(code))) + emit_sext_32_rd_rs(&rd, &rs, ctx); + else + emit_zext_32_rd_rs(&rd, &rs, ctx); + e = ctx->ninsns; + + /* Adjust for extra insns */ + rvoff -= (e - s) << 2; + } + + if (BPF_OP(code) == BPF_JSET) { + /* Adjust for and */ + rvoff -= 4; + emit(rv_and(RV_REG_T1, rd, rs), ctx); + emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, i, rvoff, + ctx); + } else { + emit_branch(BPF_OP(code), rd, rs, i, rvoff, ctx); + } break; /* IF (dst COND imm) JUMP off */ case BPF_JMP | BPF_JEQ | BPF_K: case BPF_JMP32 | BPF_JEQ | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_beq(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGT | BPF_K: case BPF_JMP32 | BPF_JGT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bltu(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLT | BPF_K: case BPF_JMP32 | BPF_JLT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bltu(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGE | BPF_K: case BPF_JMP32 | BPF_JGE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bgeu(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLE | BPF_K: case BPF_JMP32 | BPF_JLE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bgeu(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JNE | BPF_K: case BPF_JMP32 | BPF_JNE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bne(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGT | BPF_K: case BPF_JMP32 | BPF_JSGT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_blt(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLT | BPF_K: case BPF_JMP32 | BPF_JSLT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_blt(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGE | BPF_K: case BPF_JMP32 | BPF_JSGE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_bge(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLE | BPF_K: case BPF_JMP32 | BPF_JSLE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_bge(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSET | BPF_K: case BPF_JMP32 | BPF_JSET | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; + rvoff = rv_offset(i, off, ctx); + s = ctx->ninsns; emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_and(RV_REG_T1, rd, RV_REG_T1), ctx); - emit(rv_bne(RV_REG_T1, RV_REG_ZERO, rvoff >> 1), ctx); + if (!is64) { + if (is_signed_bpf_cond(BPF_OP(code))) + emit_sext_32_rd(&rd, ctx); + else + emit_zext_32_rd_t1(&rd, ctx); + } + e = ctx->ninsns; + + /* Adjust for extra insns */ + rvoff -= (e - s) << 2; + + if (BPF_OP(code) == BPF_JSET) { + /* Adjust for and */ + rvoff -= 4; + emit(rv_and(RV_REG_T1, rd, RV_REG_T1), ctx); + emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, i, rvoff, + ctx); + } else { + emit_branch(BPF_OP(code), rd, RV_REG_T1, i, rvoff, ctx); + } break; /* function call */ @@ -1557,6 +1569,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { bool tmp_blinded = false, extra_pass = false; struct bpf_prog *tmp, *orig_prog = prog; + int pass = 0, prev_ninsns = 0, i; struct rv_jit_data *jit_data; struct rv_jit_context *ctx; unsigned int image_size; @@ -1596,15 +1609,25 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = orig_prog; goto out_offset; } + for (i = 0; i < prog->len; i++) { + prev_ninsns += 32; + ctx->offset[i] = prev_ninsns; + } - /* First pass generates the ctx->offset, but does not emit an image. */ - if (build_body(ctx, extra_pass)) { - prog = orig_prog; - goto out_offset; + for (i = 0; i < 16; i++) { + pass++; + ctx->ninsns = 0; + if (build_body(ctx, extra_pass)) { + prog = orig_prog; + goto out_offset; + } + build_prologue(ctx); + ctx->epilogue_offset = ctx->ninsns; + build_epilogue(ctx); + if (ctx->ninsns == prev_ninsns) + break; + prev_ninsns = ctx->ninsns; } - build_prologue(ctx); - ctx->epilogue_offset = ctx->ninsns; - build_epilogue(ctx); /* Allocate image, now that we know the size. */ image_size = sizeof(u32) * ctx->ninsns; @@ -1619,6 +1642,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) /* Second, real pass, that acutally emits the image. */ ctx->insns = (u32 *)jit_data->image; skip_init_ctx: + pass++; ctx->ninsns = 0; build_prologue(ctx); @@ -1630,7 +1654,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) build_epilogue(ctx); if (bpf_jit_enable > 1) - bpf_jit_dump(prog->len, image_size, 2, ctx->insns); + bpf_jit_dump(prog->len, image_size, pass, ctx->insns); prog->bpf_func = (void *)ctx->insns; prog->jited = 1; -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 17:31 ` [PATCH bpf-next 2/8] riscv, bpf: add support for far branching Björn Töpel @ 2019-12-09 21:08 ` Luke Nelson 2019-12-09 21:27 ` Daniel Borkmann 2019-12-10 5:32 ` Björn Töpel 2019-12-10 5:31 ` Björn Töpel 1 sibling, 2 replies; 16+ messages in thread From: Luke Nelson @ 2019-12-09 21:08 UTC (permalink / raw) To: Björn Töpel Cc: Daniel Borkmann, Netdev, Alexei Starovoitov, linux-riscv, bpf, Xi Wang On Mon, Dec 9, 2019 at 9:32 AM Björn Töpel <bjorn.topel@gmail.com> wrote: > > This commit adds branch relaxation to the BPF JIT, and with that > support for far (offset greater than 12b) branching. > > The branch relaxation requires more than two passes to converge. For > most programs it is three passes, but for larger programs it can be > more. > > Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> We have been developing a formal verification tool for BPF JIT compilers, which we have used in the past to find bugs in the RV64 and x32 BPF JITs: https://unsat.cs.washington.edu/projects/serval/ Recently I added support for verifying the JIT for branch and jump instructions, and thought it a good opportunity to verify these patches that add support for far jumps and branching. I ported these patches to our tool and ran verification, which didn't find any bugs according to our specification of BPF and RISC-V. The tool and code are publicly available, and you can read a more detailed writeup of the results here: https://github.com/uw-unsat/bpf-jit-verif/tree/far-jump-review Currently the tool works on a manually translated version of the JIT from C to Rosette, but we are experimenting with ways of making this process more automated. Reviewed-by: Luke Nelson <lukenels@cs.washington.edu> Cc: Xi Wang <xi.wang@gmail.com> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 21:08 ` Luke Nelson @ 2019-12-09 21:27 ` Daniel Borkmann 2019-12-10 7:02 ` Luke Nelson 2019-12-10 5:32 ` Björn Töpel 1 sibling, 1 reply; 16+ messages in thread From: Daniel Borkmann @ 2019-12-09 21:27 UTC (permalink / raw) To: Luke Nelson, Björn Töpel Cc: Netdev, linux-riscv, bpf, Alexei Starovoitov, Xi Wang On 12/9/19 10:08 PM, Luke Nelson wrote: [...] > We have been developing a formal verification tool for BPF JIT > compilers, which we have used in the past to find bugs in the RV64 > and x32 BPF JITs: > > https://unsat.cs.washington.edu/projects/serval/ > > Recently I added support for verifying the JIT for branch and jump > instructions, and thought it a good opportunity to verify these > patches that add support for far jumps and branching. > > I ported these patches to our tool and ran verification, which > didn't find any bugs according to our specification of BPF and > RISC-V. > > The tool and code are publicly available, and you can read a more > detailed writeup of the results here: > > https://github.com/uw-unsat/bpf-jit-verif/tree/far-jump-review > > Currently the tool works on a manually translated version of the > JIT from C to Rosette, but we are experimenting with ways of making > this process more automated. This is awesome work! Did you also check for other architectures aside from riscv and x86-32, e.g. x86-64 or arm64? It would be great if we could add such verification tool under tools/bpf/ which would then take the in-tree JIT-code as-is for its analysis and potentially even trigger a run out of BPF selftests. Any thoughts whether such path would be feasible wrt serval? > Reviewed-by: Luke Nelson <lukenels@cs.washington.edu> > Cc: Xi Wang <xi.wang@gmail.com> Thanks, Daniel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 21:27 ` Daniel Borkmann @ 2019-12-10 7:02 ` Luke Nelson 0 siblings, 0 replies; 16+ messages in thread From: Luke Nelson @ 2019-12-10 7:02 UTC (permalink / raw) To: Daniel Borkmann Cc: Björn Töpel, Alexei Starovoitov, Netdev, linux-riscv, bpf, Xi Wang On Mon, Dec 9, 2019 at 1:27 PM Daniel Borkmann <daniel@iogearbox.net> wrote: > > This is awesome work! Did you also check for other architectures aside > from riscv and x86-32, e.g. x86-64 or arm64? We haven't tried on x86-64 or arm64 yet, but we plan to in the future and are looking at ways to minimize the effort required to port verification to new architectures. > It would be great if we could add such verification tool under tools/bpf/ > which would then take the in-tree JIT-code as-is for its analysis and > potentially even trigger a run out of BPF selftests. Any thoughts whether > such path would be feasible wrt serval? Right now the verification requires manual translation of the JIT implementation in C to Rosette for verification, which makes it difficult to integrate into existing tests. Were currently working on automating this process to be able to verify the C implementation directly. If this works out, it'd be awesome to integrate into the selftests in some way. Will keep you posted. Thanks, Luke ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 21:08 ` Luke Nelson 2019-12-09 21:27 ` Daniel Borkmann @ 2019-12-10 5:32 ` Björn Töpel 1 sibling, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-10 5:32 UTC (permalink / raw) To: Luke Nelson Cc: Daniel Borkmann, Netdev, Alexei Starovoitov, linux-riscv, bpf, Xi Wang On Mon, 9 Dec 2019 at 22:08, Luke Nelson <lukenels@cs.washington.edu> wrote: > > On Mon, Dec 9, 2019 at 9:32 AM Björn Töpel <bjorn.topel@gmail.com> wrote: > > > > This commit adds branch relaxation to the BPF JIT, and with that > > support for far (offset greater than 12b) branching. > > > > The branch relaxation requires more than two passes to converge. For > > most programs it is three passes, but for larger programs it can be > > more. > > > > Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> > > We have been developing a formal verification tool for BPF JIT > compilers, which we have used in the past to find bugs in the RV64 > and x32 BPF JITs: > > https://unsat.cs.washington.edu/projects/serval/ > > Recently I added support for verifying the JIT for branch and jump > instructions, and thought it a good opportunity to verify these > patches that add support for far jumps and branching. > > I ported these patches to our tool and ran verification, which > didn't find any bugs according to our specification of BPF and > RISC-V. > > The tool and code are publicly available, and you can read a more > detailed writeup of the results here: > > https://github.com/uw-unsat/bpf-jit-verif/tree/far-jump-review > > Currently the tool works on a manually translated version of the > JIT from C to Rosette, but we are experimenting with ways of making > this process more automated. > > > Reviewed-by: Luke Nelson <lukenels@cs.washington.edu> > Cc: Xi Wang <xi.wang@gmail.com> Wow! Very cool! Thanks a bunch for this! ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 2/8] riscv, bpf: add support for far branching 2019-12-09 17:31 ` [PATCH bpf-next 2/8] riscv, bpf: add support for far branching Björn Töpel 2019-12-09 21:08 ` Luke Nelson @ 2019-12-10 5:31 ` Björn Töpel 1 sibling, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-10 5:31 UTC (permalink / raw) To: Daniel Borkmann, Alexei Starovoitov, Netdev; +Cc: linux-riscv, bpf On Mon, 9 Dec 2019 at 18:31, Björn Töpel <bjorn.topel@gmail.com> wrote: > > This commit adds branch relaxation to the BPF JIT, and with that > support for far (offset greater than 12b) branching. > > The branch relaxation requires more than two passes to converge. For > most programs it is three passes, but for larger programs it can be > more. > > Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> > --- > arch/riscv/net/bpf_jit_comp.c | 352 ++++++++++++++++++---------------- [...] > +} > + > +static void emit_branch(u8 cond, u8 rd, u8 rs, int insn, int rvoff, The "insn" is not used. I'll do a respin. ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 2/8] riscv, bpf: add support for far branching Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 21:15 ` Luke Nelson 2019-12-09 17:31 ` [PATCH bpf-next 4/8] riscv, bpf: optimize BPF tail calls Björn Töpel ` (5 subsequent siblings) 8 siblings, 1 reply; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf This commit add support for far (offset > 21b) jumps and exits. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/net/bpf_jit_comp.c | 37 ++++++++++++++++------------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index e2f6724b64f7..e9cc9832ac2c 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -506,16 +506,6 @@ static int is_13b_check(int off, int insn) return 0; } -static int is_21b_check(int off, int insn) -{ - if (!is_21b_int(off)) { - pr_err("bpf-jit: insn=%d 21b < offset=%d not supported yet!\n", - insn, (int)off); - return -1; - } - return 0; -} - static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) { /* Note that the immediate from the add is sign-extended, @@ -733,6 +723,21 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } +static void emit_jump_and_link(u8 rd, int rvoff, struct rv_jit_context *ctx) +{ + s64 upper, lower; + + if (is_21b_int(rvoff)) { + emit(rv_jal(rd, rvoff >> 1), ctx); + return; + } + + upper = (rvoff + (1 << 11)) >> 12; + lower = rvoff & 0xfff; + emit(rv_auipc(RV_REG_T1, upper), ctx); + emit(rv_jalr(rd, RV_REG_T1, lower), ctx); +} + /* return -1 or inverted cond */ static int invert_bpf_cond(u8 cond) { @@ -1117,13 +1122,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: rvoff = rv_offset(i, off, ctx); - if (!is_21b_int(rvoff)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", - i, rvoff); - return -1; - } - - emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); break; /* IF (dst COND src) JUMP off */ @@ -1261,9 +1260,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, break; rvoff = epilogue_offset(ctx); - if (is_21b_check(rvoff, i)) - return -1; - emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); break; /* dst = imm64 */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits 2019-12-09 17:31 ` [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits Björn Töpel @ 2019-12-09 21:15 ` Luke Nelson 0 siblings, 0 replies; 16+ messages in thread From: Luke Nelson @ 2019-12-09 21:15 UTC (permalink / raw) To: Björn Töpel Cc: Daniel Borkmann, Netdev, Alexei Starovoitov, linux-riscv, bpf, Xi Wang On Mon, Dec 9, 2019 at 9:32 AM Björn Töpel <bjorn.topel@gmail.com> wrote: > > This commit add support for far (offset > 21b) jumps and exits. > > Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> Similar to the other patch for far branching, we also used our tool to formally verify this patch for far jumps: https://github.com/uw-unsat/bpf-jit-verif/tree/far-jump-review Reviewed-by: Luke Nelson <lukenels@cs.washington.edu> Cc: Xi Wang <xi.wang@gmail.com> ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH bpf-next 4/8] riscv, bpf: optimize BPF tail calls 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (2 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 5/8] riscv, bpf: provide RISC-V specific JIT image alloc/free Björn Töpel ` (4 subsequent siblings) 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf Remove one addi, and instead use the offset part of jalr. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/net/bpf_jit_comp.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index e9cc9832ac2c..cbcb33613d1d 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -562,7 +562,7 @@ static int epilogue_offset(struct rv_jit_context *ctx) return (to - from) << 2; } -static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) +static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) { int stack_adjust = ctx->stack_size, store_offset = stack_adjust - 8; @@ -599,9 +599,11 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) emit(rv_addi(RV_REG_SP, RV_REG_SP, stack_adjust), ctx); /* Set return value. */ - if (reg == RV_REG_RA) + if (!is_tail_call) emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); - emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); + emit(rv_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, + is_tail_call ? 4 : 0), /* skip TCC init */ + ctx); } static void emit_zext_32(u8 reg, struct rv_jit_context *ctx) @@ -664,9 +666,8 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) if (is_12b_check(off, insn)) return -1; emit(rv_ld(RV_REG_T3, off, RV_REG_T2), ctx); - emit(rv_addi(RV_REG_T3, RV_REG_T3, 4), ctx); emit(rv_addi(RV_REG_TCC, RV_REG_T1, 0), ctx); - __build_epilogue(RV_REG_T3, ctx); + __build_epilogue(true, ctx); return 0; } @@ -1520,7 +1521,7 @@ static void build_prologue(struct rv_jit_context *ctx) static void build_epilogue(struct rv_jit_context *ctx) { - __build_epilogue(RV_REG_RA, ctx); + __build_epilogue(false, ctx); } static int build_body(struct rv_jit_context *ctx, bool extra_pass) -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH bpf-next 5/8] riscv, bpf: provide RISC-V specific JIT image alloc/free 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (3 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 4/8] riscv, bpf: optimize BPF tail calls Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 6/8] riscv, bpf: optimize calls Björn Töpel ` (3 subsequent siblings) 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf This commit makes sure that the JIT images is kept close to the kernel text, so BPF calls can use relative calling with auipc/jalr or jal instead of loading the full 64-bit address and jalr. The BPF JIT image region is 128 MB before the kernel text. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/include/asm/pgtable.h | 4 ++++ arch/riscv/net/bpf_jit_comp.c | 13 +++++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 7ff0ed4f292e..cc3f49415620 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -404,6 +404,10 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, #define VMALLOC_END (PAGE_OFFSET - 1) #define VMALLOC_START (PAGE_OFFSET - VMALLOC_SIZE) +#define BPF_JIT_REGION_SIZE (SZ_128M) +#define BPF_JIT_REGION_START (PAGE_OFFSET - BPF_JIT_REGION_SIZE) +#define BPF_JIT_REGION_END (VMALLOC_END) + /* * Roughly size the vmemmap space to be large enough to fit enough * struct pages to map half the virtual address space. Then diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index cbcb33613d1d..f7b1ae3a968f 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -1672,3 +1672,16 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) tmp : orig_prog); return prog; } + +void *bpf_jit_alloc_exec(unsigned long size) +{ + return __vmalloc_node_range(size, PAGE_SIZE, BPF_JIT_REGION_START, + BPF_JIT_REGION_END, GFP_KERNEL, + PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + __builtin_return_address(0)); +} + +void bpf_jit_free_exec(void *addr) +{ + return vfree(addr); +} -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH bpf-next 6/8] riscv, bpf: optimize calls 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (4 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 5/8] riscv, bpf: provide RISC-V specific JIT image alloc/free Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 7/8] riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs Björn Töpel ` (2 subsequent siblings) 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf Instead of using emit_imm() and emit_jalr() which can expand to six instructions, start using jal or auipc+jalr. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/net/bpf_jit_comp.c | 101 +++++++++++++++++++++------------- 1 file changed, 64 insertions(+), 37 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index f7b1ae3a968f..260df0a87fd3 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -724,11 +724,12 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } -static void emit_jump_and_link(u8 rd, int rvoff, struct rv_jit_context *ctx) +static void emit_jump_and_link(u8 rd, s64 rvoff, bool force_jalr, + struct rv_jit_context *ctx) { s64 upper, lower; - if (is_21b_int(rvoff)) { + if (rvoff && is_21b_int(rvoff) && !force_jalr) { emit(rv_jal(rd, rvoff >> 1), ctx); return; } @@ -848,6 +849,28 @@ static bool is_signed_bpf_cond(u8 cond) cond == BPF_JSGE || cond == BPF_JSLE; } +static int emit_call(bool fixed, u64 addr, struct rv_jit_context *ctx) +{ + s64 off = 0; + u64 ip; + u8 rd; + + if (addr && ctx->insns) { + ip = (u64)(long)(ctx->insns + ctx->ninsns); + off = addr - ip; + if (!is_32b_int(off)) { + pr_err("bpf-jit: target call addr %pK is out of range\n", + (void *)addr); + return -ERANGE; + } + } + + emit_jump_and_link(RV_REG_RA, off, !fixed, ctx); + rd = bpf_to_rv_reg(BPF_REG_0, ctx); + emit(rv_addi(rd, RV_REG_A0, 0), ctx); + return 0; +} + static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, bool extra_pass) { @@ -1123,7 +1146,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: rvoff = rv_offset(i, off, ctx); - emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); break; /* IF (dst COND src) JUMP off */ @@ -1225,7 +1248,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_JMP | BPF_CALL: { bool fixed; - int i, ret; + int ret; u64 addr; mark_call(ctx); @@ -1233,20 +1256,9 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, &fixed); if (ret < 0) return ret; - if (fixed) { - emit_imm(RV_REG_T1, addr, ctx); - } else { - i = ctx->ninsns; - emit_imm(RV_REG_T1, addr, ctx); - for (i = ctx->ninsns - i; i < 8; i++) { - /* nop */ - emit(rv_addi(RV_REG_ZERO, RV_REG_ZERO, 0), - ctx); - } - } - emit(rv_jalr(RV_REG_RA, RV_REG_T1, 0), ctx); - rd = bpf_to_rv_reg(BPF_REG_0, ctx); - emit(rv_addi(rd, RV_REG_A0, 0), ctx); + ret = emit_call(fixed, addr, ctx); + if (ret) + return ret; break; } /* tail call */ @@ -1261,7 +1273,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, break; rvoff = epilogue_offset(ctx); - emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); break; /* dst = imm64 */ @@ -1524,7 +1536,7 @@ static void build_epilogue(struct rv_jit_context *ctx) __build_epilogue(false, ctx); } -static int build_body(struct rv_jit_context *ctx, bool extra_pass) +static int build_body(struct rv_jit_context *ctx, bool extra_pass, int *offset) { const struct bpf_prog *prog = ctx->prog; int i; @@ -1536,12 +1548,12 @@ static int build_body(struct rv_jit_context *ctx, bool extra_pass) ret = emit_insn(insn, ctx, extra_pass); if (ret > 0) { i++; - if (ctx->insns == NULL) - ctx->offset[i] = ctx->ninsns; + if (offset) + offset[i] = ctx->ninsns; continue; } - if (ctx->insns == NULL) - ctx->offset[i] = ctx->ninsns; + if (offset) + offset[i] = ctx->ninsns; if (ret) return ret; } @@ -1569,8 +1581,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) struct bpf_prog *tmp, *orig_prog = prog; int pass = 0, prev_ninsns = 0, i; struct rv_jit_data *jit_data; + unsigned int image_size = 0; struct rv_jit_context *ctx; - unsigned int image_size; if (!prog->jit_requested) return orig_prog; @@ -1615,36 +1627,51 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) for (i = 0; i < 16; i++) { pass++; ctx->ninsns = 0; - if (build_body(ctx, extra_pass)) { + if (build_body(ctx, extra_pass, ctx->offset)) { prog = orig_prog; goto out_offset; } build_prologue(ctx); ctx->epilogue_offset = ctx->ninsns; build_epilogue(ctx); - if (ctx->ninsns == prev_ninsns) - break; + + if (ctx->ninsns == prev_ninsns) { + if (jit_data->header) + break; + + image_size = sizeof(u32) * ctx->ninsns; + jit_data->header = + bpf_jit_binary_alloc(image_size, + &jit_data->image, + sizeof(u32), + bpf_fill_ill_insns); + if (!jit_data->header) { + prog = orig_prog; + goto out_offset; + } + + ctx->insns = (u32 *)jit_data->image; + /* Now, when the image is allocated, the image + * can potentially shrink more (auipc/jalr -> + * jal). + */ + } prev_ninsns = ctx->ninsns; } - /* Allocate image, now that we know the size. */ - image_size = sizeof(u32) * ctx->ninsns; - jit_data->header = bpf_jit_binary_alloc(image_size, &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + if (i == 16) { + pr_err("bpf-jit: image did not converge in <%d passes!\n", i); + bpf_jit_binary_free(jit_data->header); prog = orig_prog; goto out_offset; } - /* Second, real pass, that acutally emits the image. */ - ctx->insns = (u32 *)jit_data->image; skip_init_ctx: pass++; ctx->ninsns = 0; build_prologue(ctx); - if (build_body(ctx, extra_pass)) { + if (build_body(ctx, extra_pass, NULL)) { bpf_jit_binary_free(jit_data->header); prog = orig_prog; goto out_offset; -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH bpf-next 7/8] riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (5 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 6/8] riscv, bpf: optimize calls Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 8/8] riscv, perf: add arch specific perf_arch_bpf_user_pt_regs Björn Töpel 2019-12-19 21:59 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Palmer Dabbelt 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf Add missing uapi header the BPF_PROG_TYPE_PERF_EVENT programs by exporting struct user_regs_struct instead of struct pt_regs which is in-kernel only. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/include/uapi/asm/bpf_perf_event.h | 9 +++++++++ tools/include/uapi/asm/bpf_perf_event.h | 2 ++ 2 files changed, 11 insertions(+) create mode 100644 arch/riscv/include/uapi/asm/bpf_perf_event.h diff --git a/arch/riscv/include/uapi/asm/bpf_perf_event.h b/arch/riscv/include/uapi/asm/bpf_perf_event.h new file mode 100644 index 000000000000..6cb1c2823288 --- /dev/null +++ b/arch/riscv/include/uapi/asm/bpf_perf_event.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI__ASM_BPF_PERF_EVENT_H__ +#define _UAPI__ASM_BPF_PERF_EVENT_H__ + +#include <asm/ptrace.h> + +typedef struct user_regs_struct bpf_user_pt_regs_t; + +#endif /* _UAPI__ASM_BPF_PERF_EVENT_H__ */ diff --git a/tools/include/uapi/asm/bpf_perf_event.h b/tools/include/uapi/asm/bpf_perf_event.h index 13a58531e6fa..39acc149d843 100644 --- a/tools/include/uapi/asm/bpf_perf_event.h +++ b/tools/include/uapi/asm/bpf_perf_event.h @@ -2,6 +2,8 @@ #include "../../arch/arm64/include/uapi/asm/bpf_perf_event.h" #elif defined(__s390__) #include "../../arch/s390/include/uapi/asm/bpf_perf_event.h" +#elif defined(__riscv) +#include "../../arch/riscv/include/uapi/asm/bpf_perf_event.h" #else #include <uapi/asm-generic/bpf_perf_event.h> #endif -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH bpf-next 8/8] riscv, perf: add arch specific perf_arch_bpf_user_pt_regs 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (6 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 7/8] riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs Björn Töpel @ 2019-12-09 17:31 ` Björn Töpel 2019-12-19 21:59 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Palmer Dabbelt 8 siblings, 0 replies; 16+ messages in thread From: Björn Töpel @ 2019-12-09 17:31 UTC (permalink / raw) To: daniel, ast, netdev; +Cc: Björn Töpel, linux-riscv, bpf RISC-V was missing a proper perf_arch_bpf_user_pt_regs macro for CONFIG_PERF_EVENT builds. Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> --- arch/riscv/include/asm/perf_event.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/riscv/include/asm/perf_event.h b/arch/riscv/include/asm/perf_event.h index aefbfaa6a781..0234048b12bc 100644 --- a/arch/riscv/include/asm/perf_event.h +++ b/arch/riscv/include/asm/perf_event.h @@ -82,4 +82,8 @@ struct riscv_pmu { int irq; }; +#ifdef CONFIG_PERF_EVENTS +#define perf_arch_bpf_user_pt_regs(regs) (struct user_regs_struct *)regs +#endif + #endif /* _ASM_RISCV_PERF_EVENT_H */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel ` (7 preceding siblings ...) 2019-12-09 17:31 ` [PATCH bpf-next 8/8] riscv, perf: add arch specific perf_arch_bpf_user_pt_regs Björn Töpel @ 2019-12-19 21:59 ` Palmer Dabbelt 8 siblings, 0 replies; 16+ messages in thread From: Palmer Dabbelt @ 2019-12-19 21:59 UTC (permalink / raw) To: Björn Töpel Cc: daniel, netdev, ast, Björn Töpel, linux-riscv, bpf On Mon, 09 Dec 2019 09:31:29 PST (-0800), Björn Töpel wrote: > The BPF JIT incorrectly clobbered the a0 register, and did not flag > usage of s5 register when BPF stack was being used. > > Fixes: 2353ecc6f91f ("bpf, riscv: add BPF JIT for RV64G") > Signed-off-by: Björn Töpel <bjorn.topel@gmail.com> > --- > arch/riscv/net/bpf_jit_comp.c | 13 +++++++++++-- > 1 file changed, 11 insertions(+), 2 deletions(-) > > diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c > index 5451ef3845f2..1606ebd49666 100644 > --- a/arch/riscv/net/bpf_jit_comp.c > +++ b/arch/riscv/net/bpf_jit_comp.c > @@ -120,6 +120,11 @@ static bool seen_reg(int reg, struct rv_jit_context *ctx) > return false; > } > > +static void mark_fp(struct rv_jit_context *ctx) > +{ > + __set_bit(RV_CTX_F_SEEN_S5, &ctx->flags); > +} > + > static void mark_call(struct rv_jit_context *ctx) > { > __set_bit(RV_CTX_F_SEEN_CALL, &ctx->flags); > @@ -596,7 +601,8 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) > > emit(rv_addi(RV_REG_SP, RV_REG_SP, stack_adjust), ctx); > /* Set return value. */ > - emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); > + if (reg == RV_REG_RA) > + emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); > emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); > } > > @@ -1426,6 +1432,10 @@ static void build_prologue(struct rv_jit_context *ctx) > { > int stack_adjust = 0, store_offset, bpf_stack_adjust; > > + bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); > + if (bpf_stack_adjust) > + mark_fp(ctx); > + > if (seen_reg(RV_REG_RA, ctx)) > stack_adjust += 8; > stack_adjust += 8; /* RV_REG_FP */ > @@ -1443,7 +1453,6 @@ static void build_prologue(struct rv_jit_context *ctx) > stack_adjust += 8; > > stack_adjust = round_up(stack_adjust, 16); > - bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); > stack_adjust += bpf_stack_adjust; > > store_offset = stack_adjust - 8; Reviewed-by: Palmer Dabbelt <palmerdabbelt@google.com> and I'm assuming these are going in through a BPF tree, so also Acked-by: Palmer Dabbelt <palmerdabbelt@google.com> but LMK if you want it through the RISC-V tree. ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-12-19 22:03 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-12-09 17:31 [PATCH bpf-next 0/8] riscv: BPF JIT fix, optimizations and far jumps support Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 2/8] riscv, bpf: add support for far branching Björn Töpel 2019-12-09 21:08 ` Luke Nelson 2019-12-09 21:27 ` Daniel Borkmann 2019-12-10 7:02 ` Luke Nelson 2019-12-10 5:32 ` Björn Töpel 2019-12-10 5:31 ` Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 3/8] riscv, bpf: add support for far jumps and exits Björn Töpel 2019-12-09 21:15 ` Luke Nelson 2019-12-09 17:31 ` [PATCH bpf-next 4/8] riscv, bpf: optimize BPF tail calls Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 5/8] riscv, bpf: provide RISC-V specific JIT image alloc/free Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 6/8] riscv, bpf: optimize calls Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 7/8] riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs Björn Töpel 2019-12-09 17:31 ` [PATCH bpf-next 8/8] riscv, perf: add arch specific perf_arch_bpf_user_pt_regs Björn Töpel 2019-12-19 21:59 ` [PATCH bpf-next 1/8] riscv, bpf: fix broken BPF tail calls Palmer Dabbelt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).