* [PATCH v2 0/8] bpf powerpc: Add BPF_PROBE_MEM support in powerpc JIT compiler
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev, Hari Bathini
Patch #1 & #2 are simple cleanup patches. Patch #3 refactors JIT
compiler code with the aim to simplify adding BPF_PROBE_MEM support.
Patch #4 introduces PPC_RAW_BRANCH() macro instead of open coding
branch instruction. Patch #5 & #7 add BPF_PROBE_MEM support for PPC64
& PPC32 JIT compilers respectively. Patch #6 & #8 add explicit
addr > TASK_SIZE_MAX check to handle bad userspace pointers for
PPC64 & PPC32 cases respectively.
Link to v1 posted by Ravi:
https://lore.kernel.org/all/20210706073211.349889-1-ravi.bangoria@linux.ibm.com/
("[PATCH 0/4] bpf powerpc: Add BPF_PROBE_MEM support for 64bit JIT")
Hari Bathini (4):
bpf powerpc: refactor JIT compiler code
powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
bpf ppc32: Add BPF_PROBE_MEM support for JIT
bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
Ravi Bangoria (4):
bpf powerpc: Remove unused SEEN_STACK
bpf powerpc: Remove extra_pass from bpf_jit_build_body()
bpf ppc64: Add BPF_PROBE_MEM support for JIT
bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
arch/powerpc/include/asm/ppc-opcode.h | 2 +
arch/powerpc/net/bpf_jit.h | 19 ++--
arch/powerpc/net/bpf_jit_comp.c | 75 ++++++++++++++--
arch/powerpc/net/bpf_jit_comp32.c | 123 +++++++++++++++++++++-----
arch/powerpc/net/bpf_jit_comp64.c | 114 ++++++++++++++++--------
5 files changed, 260 insertions(+), 73 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH v2 0/8] bpf powerpc: Add BPF_PROBE_MEM support in powerpc JIT compiler
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai, Hari Bathini
Patch #1 & #2 are simple cleanup patches. Patch #3 refactors JIT
compiler code with the aim to simplify adding BPF_PROBE_MEM support.
Patch #4 introduces PPC_RAW_BRANCH() macro instead of open coding
branch instruction. Patch #5 & #7 add BPF_PROBE_MEM support for PPC64
& PPC32 JIT compilers respectively. Patch #6 & #8 add explicit
addr > TASK_SIZE_MAX check to handle bad userspace pointers for
PPC64 & PPC32 cases respectively.
Link to v1 posted by Ravi:
https://lore.kernel.org/all/20210706073211.349889-1-ravi.bangoria@linux.ibm.com/
("[PATCH 0/4] bpf powerpc: Add BPF_PROBE_MEM support for 64bit JIT")
Hari Bathini (4):
bpf powerpc: refactor JIT compiler code
powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
bpf ppc32: Add BPF_PROBE_MEM support for JIT
bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
Ravi Bangoria (4):
bpf powerpc: Remove unused SEEN_STACK
bpf powerpc: Remove extra_pass from bpf_jit_build_body()
bpf ppc64: Add BPF_PROBE_MEM support for JIT
bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
arch/powerpc/include/asm/ppc-opcode.h | 2 +
arch/powerpc/net/bpf_jit.h | 19 ++--
arch/powerpc/net/bpf_jit_comp.c | 75 ++++++++++++++--
arch/powerpc/net/bpf_jit_comp32.c | 123 +++++++++++++++++++++-----
arch/powerpc/net/bpf_jit_comp64.c | 114 ++++++++++++++++--------
5 files changed, 260 insertions(+), 73 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH v2 1/8] bpf powerpc: Remove unused SEEN_STACK
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev,
Ravi Bangoria
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
SEEN_STACK is unused on PowerPC. Remove it. Also, have
SEEN_TAILCALL use 0x40000000.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
---
* No changes in v2.
arch/powerpc/net/bpf_jit.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 99fad093f43e..d6267e93027a 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -116,8 +116,7 @@ static inline bool is_nearbranch(int offset)
#define COND_LE (CR0_GT | COND_CMP_FALSE)
#define SEEN_FUNC 0x20000000 /* might call external helpers */
-#define SEEN_STACK 0x40000000 /* uses BPF stack */
-#define SEEN_TAILCALL 0x80000000 /* uses tail calls */
+#define SEEN_TAILCALL 0x40000000 /* uses tail calls */
#define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */
#define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 1/8] bpf powerpc: Remove unused SEEN_STACK
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
SEEN_STACK is unused on PowerPC. Remove it. Also, have
SEEN_TAILCALL use 0x40000000.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
---
* No changes in v2.
arch/powerpc/net/bpf_jit.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 99fad093f43e..d6267e93027a 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -116,8 +116,7 @@ static inline bool is_nearbranch(int offset)
#define COND_LE (CR0_GT | COND_CMP_FALSE)
#define SEEN_FUNC 0x20000000 /* might call external helpers */
-#define SEEN_STACK 0x40000000 /* uses BPF stack */
-#define SEEN_TAILCALL 0x80000000 /* uses tail calls */
+#define SEEN_TAILCALL 0x40000000 /* uses tail calls */
#define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */
#define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 2/8] bpf powerpc: Remove extra_pass from bpf_jit_build_body()
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev,
Ravi Bangoria
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
In case of extra_pass, usual JIT passes are always skipped. So,
extra_pass is always false while calling bpf_jit_build_body() and
thus it can be removed.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
---
Changes in v2:
* Updated the changelog wording a bit.
arch/powerpc/net/bpf_jit.h | 2 +-
arch/powerpc/net/bpf_jit_comp.c | 6 +++---
arch/powerpc/net/bpf_jit_comp32.c | 4 ++--
arch/powerpc/net/bpf_jit_comp64.c | 4 ++--
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index d6267e93027a..411c63d945c7 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -166,7 +166,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass);
+ u32 *addrs);
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 53aefee3fe70..c5c9e8ad1de7 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -149,7 +149,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
/* Scouting faux-generate pass 0 */
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
/* We hit something illegal or unsupported. */
fp = org_fp;
goto out_addrs;
@@ -162,7 +162,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
*/
if (cgctx.seen & SEEN_TAILCALL) {
cgctx.idx = 0;
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
fp = org_fp;
goto out_addrs;
}
@@ -210,7 +210,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
/* Now build the prologue, body code & epilogue for real. */
cgctx.idx = 0;
bpf_jit_build_prologue(code_base, &cgctx);
- bpf_jit_build_body(fp, code_base, &cgctx, addrs, extra_pass);
+ bpf_jit_build_body(fp, code_base, &cgctx, addrs);
bpf_jit_build_epilogue(code_base, &cgctx);
if (bpf_jit_enable > 1)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index beb12cbc8c29..b60b59426a24 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass)
+ u32 *addrs)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -860,7 +860,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_JMP | BPF_CALL:
ctx->seen |= SEEN_FUNC;
- ret = bpf_jit_get_func_addr(fp, &insn[i], extra_pass,
+ ret = bpf_jit_get_func_addr(fp, &insn[i], false,
&func_addr, &func_addr_fixed);
if (ret < 0)
return ret;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index b87a63dba9c8..2a87da50d9a4 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -272,7 +272,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass)
+ u32 *addrs)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -769,7 +769,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_JMP | BPF_CALL:
ctx->seen |= SEEN_FUNC;
- ret = bpf_jit_get_func_addr(fp, &insn[i], extra_pass,
+ ret = bpf_jit_get_func_addr(fp, &insn[i], false,
&func_addr, &func_addr_fixed);
if (ret < 0)
return ret;
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 2/8] bpf powerpc: Remove extra_pass from bpf_jit_build_body()
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
In case of extra_pass, usual JIT passes are always skipped. So,
extra_pass is always false while calling bpf_jit_build_body() and
thus it can be removed.
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
---
Changes in v2:
* Updated the changelog wording a bit.
arch/powerpc/net/bpf_jit.h | 2 +-
arch/powerpc/net/bpf_jit_comp.c | 6 +++---
arch/powerpc/net/bpf_jit_comp32.c | 4 ++--
arch/powerpc/net/bpf_jit_comp64.c | 4 ++--
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index d6267e93027a..411c63d945c7 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -166,7 +166,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass);
+ u32 *addrs);
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 53aefee3fe70..c5c9e8ad1de7 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -149,7 +149,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
/* Scouting faux-generate pass 0 */
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
/* We hit something illegal or unsupported. */
fp = org_fp;
goto out_addrs;
@@ -162,7 +162,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
*/
if (cgctx.seen & SEEN_TAILCALL) {
cgctx.idx = 0;
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs, false)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
fp = org_fp;
goto out_addrs;
}
@@ -210,7 +210,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
/* Now build the prologue, body code & epilogue for real. */
cgctx.idx = 0;
bpf_jit_build_prologue(code_base, &cgctx);
- bpf_jit_build_body(fp, code_base, &cgctx, addrs, extra_pass);
+ bpf_jit_build_body(fp, code_base, &cgctx, addrs);
bpf_jit_build_epilogue(code_base, &cgctx);
if (bpf_jit_enable > 1)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index beb12cbc8c29..b60b59426a24 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass)
+ u32 *addrs)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -860,7 +860,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_JMP | BPF_CALL:
ctx->seen |= SEEN_FUNC;
- ret = bpf_jit_get_func_addr(fp, &insn[i], extra_pass,
+ ret = bpf_jit_get_func_addr(fp, &insn[i], false,
&func_addr, &func_addr_fixed);
if (ret < 0)
return ret;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index b87a63dba9c8..2a87da50d9a4 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -272,7 +272,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs, bool extra_pass)
+ u32 *addrs)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -769,7 +769,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_JMP | BPF_CALL:
ctx->seen |= SEEN_FUNC;
- ret = bpf_jit_get_func_addr(fp, &insn[i], extra_pass,
+ ret = bpf_jit_get_func_addr(fp, &insn[i], false,
&func_addr, &func_addr_fixed);
if (ret < 0)
return ret;
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev, Hari Bathini
Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to refactor a bit of JITing code.
arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
2 files changed, 68 insertions(+), 58 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index b60b59426a24..c8ae14c316e3 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
u32 exit_addr = addrs[flen];
for (i = 0; i < flen; i++) {
- u32 code = insn[i].code;
u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
- u32 dst_reg_h = dst_reg - 1;
u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
- u32 src_reg_h = src_reg - 1;
u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
+ u32 true_cond, code = insn[i].code;
+ u32 dst_reg_h = dst_reg - 1;
+ u32 src_reg_h = src_reg - 1;
+ u32 size = BPF_SIZE(code);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
u64 func_addr;
- u32 true_cond;
/*
* addrs[] maps a BPF bytecode address into a real offset from
@@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/*
* BPF_LDX
*/
- case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
- EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
- EMIT(PPC_RAW_LI(dst_reg_h, 0));
- break;
- case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
- EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
- EMIT(PPC_RAW_LI(dst_reg_h, 0));
- break;
- case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
+ /* dst = *(u8 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_B:
+ /* dst = *(u16 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_H:
+ /* dst = *(u32 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_W:
+ /* dst = *(u64 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_DW:
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
+ break;
+ case BPF_DW:
+ EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
+ break;
+ }
+
+ if ((size != BPF_DW) && !fp->aux->verifier_zext)
EMIT(PPC_RAW_LI(dst_reg_h, 0));
break;
- case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
- EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
- break;
/*
* Doubleword load
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 2a87da50d9a4..78b28f25555c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
u32 exit_addr = addrs[flen];
for (i = 0; i < flen; i++) {
- u32 code = insn[i].code;
u32 dst_reg = b2p[insn[i].dst_reg];
u32 src_reg = b2p[insn[i].src_reg];
+ u32 tmp_idx, code = insn[i].code;
+ u32 size = BPF_SIZE(code);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
- u64 func_addr;
- u64 imm64;
+ u64 func_addr, imm64;
u32 true_cond;
- u32 tmp_idx;
/*
* addrs[] maps a BPF bytecode address into a real offset from
@@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
+ if ((size == BPF_B) || (size == BPF_H))
+ EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
+ else /* size == BPF_W || size == BPF_DW */
+ PPC_LI32(b2p[TMP_REG_1], imm);
src_reg = b2p[TMP_REG_1];
}
- PPC_BPF_STL(src_reg, dst_reg, off);
+
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
+ break;
+ case BPF_DW:
+ PPC_BPF_STL(src_reg, dst_reg, off);
+ break;
+ }
break;
/*
@@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
- EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
- EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
- PPC_BPF_LL(dst_reg, src_reg, off);
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
+ break;
+ case BPF_DW:
+ PPC_BPF_LL(dst_reg, src_reg, off);
+ break;
+ }
+
+ if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
+ addrs[++i] = ctx->idx * 4;
break;
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai, Hari Bathini
Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to refactor a bit of JITing code.
arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
2 files changed, 68 insertions(+), 58 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index b60b59426a24..c8ae14c316e3 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
u32 exit_addr = addrs[flen];
for (i = 0; i < flen; i++) {
- u32 code = insn[i].code;
u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
- u32 dst_reg_h = dst_reg - 1;
u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
- u32 src_reg_h = src_reg - 1;
u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
+ u32 true_cond, code = insn[i].code;
+ u32 dst_reg_h = dst_reg - 1;
+ u32 src_reg_h = src_reg - 1;
+ u32 size = BPF_SIZE(code);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
u64 func_addr;
- u32 true_cond;
/*
* addrs[] maps a BPF bytecode address into a real offset from
@@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/*
* BPF_LDX
*/
- case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
- EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
- EMIT(PPC_RAW_LI(dst_reg_h, 0));
- break;
- case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
- EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
- EMIT(PPC_RAW_LI(dst_reg_h, 0));
- break;
- case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
- if (!fp->aux->verifier_zext)
+ /* dst = *(u8 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_B:
+ /* dst = *(u16 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_H:
+ /* dst = *(u32 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_W:
+ /* dst = *(u64 *)(ul) (src + off) */
+ case BPF_LDX | BPF_MEM | BPF_DW:
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
+ break;
+ case BPF_DW:
+ EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
+ break;
+ }
+
+ if ((size != BPF_DW) && !fp->aux->verifier_zext)
EMIT(PPC_RAW_LI(dst_reg_h, 0));
break;
- case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
- EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
- break;
/*
* Doubleword load
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 2a87da50d9a4..78b28f25555c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
u32 exit_addr = addrs[flen];
for (i = 0; i < flen; i++) {
- u32 code = insn[i].code;
u32 dst_reg = b2p[insn[i].dst_reg];
u32 src_reg = b2p[insn[i].src_reg];
+ u32 tmp_idx, code = insn[i].code;
+ u32 size = BPF_SIZE(code);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
- u64 func_addr;
- u64 imm64;
+ u64 func_addr, imm64;
u32 true_cond;
- u32 tmp_idx;
/*
* addrs[] maps a BPF bytecode address into a real offset from
@@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
- if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
- src_reg = b2p[TMP_REG_1];
- }
- EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
- break;
case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
+ if ((size == BPF_B) || (size == BPF_H))
+ EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
+ else /* size == BPF_W || size == BPF_DW */
+ PPC_LI32(b2p[TMP_REG_1], imm);
src_reg = b2p[TMP_REG_1];
}
- PPC_BPF_STL(src_reg, dst_reg, off);
+
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
+ break;
+ case BPF_DW:
+ PPC_BPF_STL(src_reg, dst_reg, off);
+ break;
+ }
break;
/*
@@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
- EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
- EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
- EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
- if (insn_is_zext(&insn[i + 1]))
- addrs[++i] = ctx->idx * 4;
- break;
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
- PPC_BPF_LL(dst_reg, src_reg, off);
+ switch (size) {
+ case BPF_B:
+ EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
+ break;
+ case BPF_H:
+ EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
+ break;
+ case BPF_W:
+ EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
+ break;
+ case BPF_DW:
+ PPC_BPF_LL(dst_reg, src_reg, off);
+ break;
+ }
+
+ if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
+ addrs[++i] = ctx->idx * 4;
break;
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 4/8] powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev, Hari Bathini
Define and use PPC_RAW_BRANCH() macro instead of open coding it. This
macro is used while adding BPF_PROBE_MEM support.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to introduce PPC_RAW_BRANCH() macro.
arch/powerpc/include/asm/ppc-opcode.h | 2 ++
arch/powerpc/net/bpf_jit.h | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
index baea657bc868..f50213e2a3e0 100644
--- a/arch/powerpc/include/asm/ppc-opcode.h
+++ b/arch/powerpc/include/asm/ppc-opcode.h
@@ -566,6 +566,8 @@
#define PPC_RAW_MTSPR(spr, d) (0x7c0003a6 | ___PPC_RS(d) | __PPC_SPR(spr))
#define PPC_RAW_EIEIO() (0x7c0006ac)
+#define PPC_RAW_BRANCH(addr) (PPC_INST_BRANCH | ((addr) & 0x03fffffc))
+
/* Deal with instructions that older assemblers aren't aware of */
#define PPC_BCCTR_FLUSH stringify_in_c(.long PPC_INST_BCCTR_FLUSH)
#define PPC_CP_ABORT stringify_in_c(.long PPC_RAW_CP_ABORT)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 411c63d945c7..0c8f885b8f48 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -24,8 +24,8 @@
#define EMIT(instr) PLANT_INSTR(image, ctx->idx, instr)
/* Long jump; (unconditional 'branch') */
-#define PPC_JMP(dest) EMIT(PPC_INST_BRANCH | \
- (((dest) - (ctx->idx * 4)) & 0x03fffffc))
+#define PPC_JMP(dest) EMIT(PPC_RAW_BRANCH((dest) - (ctx->idx * 4)))
+
/* blr; (unconditional 'branch' with link) to absolute address */
#define PPC_BL_ABS(dest) EMIT(PPC_INST_BL | \
(((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 4/8] powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai, Hari Bathini
Define and use PPC_RAW_BRANCH() macro instead of open coding it. This
macro is used while adding BPF_PROBE_MEM support.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to introduce PPC_RAW_BRANCH() macro.
arch/powerpc/include/asm/ppc-opcode.h | 2 ++
arch/powerpc/net/bpf_jit.h | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
index baea657bc868..f50213e2a3e0 100644
--- a/arch/powerpc/include/asm/ppc-opcode.h
+++ b/arch/powerpc/include/asm/ppc-opcode.h
@@ -566,6 +566,8 @@
#define PPC_RAW_MTSPR(spr, d) (0x7c0003a6 | ___PPC_RS(d) | __PPC_SPR(spr))
#define PPC_RAW_EIEIO() (0x7c0006ac)
+#define PPC_RAW_BRANCH(addr) (PPC_INST_BRANCH | ((addr) & 0x03fffffc))
+
/* Deal with instructions that older assemblers aren't aware of */
#define PPC_BCCTR_FLUSH stringify_in_c(.long PPC_INST_BCCTR_FLUSH)
#define PPC_CP_ABORT stringify_in_c(.long PPC_RAW_CP_ABORT)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 411c63d945c7..0c8f885b8f48 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -24,8 +24,8 @@
#define EMIT(instr) PLANT_INSTR(image, ctx->idx, instr)
/* Long jump; (unconditional 'branch') */
-#define PPC_JMP(dest) EMIT(PPC_INST_BRANCH | \
- (((dest) - (ctx->idx * 4)) & 0x03fffffc))
+#define PPC_JMP(dest) EMIT(PPC_RAW_BRANCH((dest) - (ctx->idx * 4)))
+
/* blr; (unconditional 'branch' with link) to absolute address */
#define PPC_BL_ABS(dest) EMIT(PPC_INST_BL | \
(((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 5/8] bpf ppc64: Add BPF_PROBE_MEM support for JIT
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev,
Ravi Bangoria, Hari Bathini
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
BPF load instruction with BPF_PROBE_MEM mode can cause a fault
inside kernel. Append exception table for such instructions
within BPF program.
Unlike other archs which uses extable 'fixup' field to pass dest_reg
and nip, BPF exception table on PowerPC follows the generic PowerPC
exception table design, where it populates both fixup and extable
sections within BPF program. fixup section contains two instructions,
first instruction clears dest_reg and 2nd jumps to next instruction
in the BPF code. extable 'insn' field contains relative offset of
the instruction and 'fixup' field contains relative offset of the
fixup entry. Example layout of BPF program with extable present:
+------------------+
| |
| |
0x4020 -->| ld r27,4(r3) |
| |
| |
0x40ac -->| lwz r3,0(r4) |
| |
| |
|------------------|
0x4280 -->| li r27,0 | \ fixup entry
| b 0x4024 | /
0x4288 -->| li r3,0 |
| b 0x40b0 |
|------------------|
0x4290 -->| insn=0xfffffd90 | \ extable entry
| fixup=0xffffffec | /
0x4298 -->| insn=0xfffffe14 |
| fixup=0xffffffec |
+------------------+
(Addresses shown here are chosen random, not real)
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* Used JITing code after refactoring.
* Replaced 'xor reg,reg,reg' with 'li reg,0' where appropriate.
* Avoided unnecessary init during declaration.
arch/powerpc/net/bpf_jit.h | 5 ++-
arch/powerpc/net/bpf_jit_comp.c | 25 ++++++++++----
arch/powerpc/net/bpf_jit_comp32.c | 2 +-
arch/powerpc/net/bpf_jit_comp64.c | 57 ++++++++++++++++++++++++++++++-
4 files changed, 80 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 0c8f885b8f48..6357c71c26eb 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -141,8 +141,11 @@ struct codegen_context {
unsigned int idx;
unsigned int stack_size;
int b2p[ARRAY_SIZE(b2p)];
+ unsigned int exentry_idx;
};
+#define BPF_FIXUP_LEN 8 /* Two instructions */
+
static inline void bpf_flush_icache(void *start, void *end)
{
smp_wmb(); /* smp write barrier */
@@ -166,7 +169,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs);
+ u32 *addrs, int pass);
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index c5c9e8ad1de7..e92bd79d3bac 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -101,6 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
struct bpf_prog *tmp_fp;
bool bpf_blinded = false;
bool extra_pass = false;
+ u32 extable_len;
+ u32 fixup_len;
if (!fp->jit_requested)
return org_fp;
@@ -131,7 +133,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
image = jit_data->image;
bpf_hdr = jit_data->header;
proglen = jit_data->proglen;
- alloclen = proglen + FUNCTION_DESCR_SIZE;
extra_pass = true;
goto skip_init_ctx;
}
@@ -149,7 +150,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
/* Scouting faux-generate pass 0 */
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
/* We hit something illegal or unsupported. */
fp = org_fp;
goto out_addrs;
@@ -162,7 +163,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
*/
if (cgctx.seen & SEEN_TAILCALL) {
cgctx.idx = 0;
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
fp = org_fp;
goto out_addrs;
}
@@ -177,8 +178,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
bpf_jit_build_prologue(0, &cgctx);
bpf_jit_build_epilogue(0, &cgctx);
+ fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN;
+ extable_len = fp->aux->num_exentries * sizeof(struct exception_table_entry);
+
proglen = cgctx.idx * 4;
- alloclen = proglen + FUNCTION_DESCR_SIZE;
+ alloclen = proglen + FUNCTION_DESCR_SIZE + fixup_len + extable_len;
bpf_hdr = bpf_jit_binary_alloc(alloclen, &image, 4, bpf_jit_fill_ill_insns);
if (!bpf_hdr) {
@@ -186,6 +190,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
goto out_addrs;
}
+ if (extable_len) {
+ fp->aux->extable = (void *)image + FUNCTION_DESCR_SIZE +
+ proglen + fixup_len;
+ }
+
skip_init_ctx:
code_base = (u32 *)(image + FUNCTION_DESCR_SIZE);
@@ -210,7 +219,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
/* Now build the prologue, body code & epilogue for real. */
cgctx.idx = 0;
bpf_jit_build_prologue(code_base, &cgctx);
- bpf_jit_build_body(fp, code_base, &cgctx, addrs);
+ if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) {
+ bpf_jit_binary_free(bpf_hdr);
+ fp = org_fp;
+ goto out_addrs;
+ }
bpf_jit_build_epilogue(code_base, &cgctx);
if (bpf_jit_enable > 1)
@@ -234,7 +247,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
fp->bpf_func = (void *)image;
fp->jited = 1;
- fp->jited_len = alloclen;
+ fp->jited_len = proglen + FUNCTION_DESCR_SIZE;
bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + (bpf_hdr->pages * PAGE_SIZE));
bpf_jit_binary_lock_ro(bpf_hdr);
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index c8ae14c316e3..94641b7be387 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs)
+ u32 *addrs, int pass)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 78b28f25555c..2fc10995f243 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -270,9 +270,54 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* out: */
}
+/*
+ * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
+ * this function, as this only applies to BPF_PROBE_MEM, for now.
+ */
+static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
+ struct codegen_context *ctx, int dst_reg)
+{
+ off_t offset;
+ unsigned long pc;
+ struct exception_table_entry *ex;
+ u32 *fixup;
+
+ /* Populate extable entries only in the last pass */
+ if (pass != 2)
+ return 0;
+
+ if (!fp->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
+ return -EINVAL;
+
+ pc = (unsigned long)&image[ctx->idx - 1];
+
+ fixup = (void *)fp->aux->extable -
+ (fp->aux->num_exentries * BPF_FIXUP_LEN) +
+ (ctx->exentry_idx * BPF_FIXUP_LEN);
+
+ fixup[0] = PPC_RAW_LI(dst_reg, 0);
+ fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
+
+ ex = &fp->aux->extable[ctx->exentry_idx];
+
+ offset = pc - (long)&ex->insn;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->insn = offset;
+
+ offset = (long)fixup - (long)&ex->fixup;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->fixup = offset;
+
+ ctx->exentry_idx++;
+ return 0;
+}
+
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs)
+ u32 *addrs, int pass)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -714,12 +759,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_H:
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_W:
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
@@ -737,6 +786,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
addrs[++i] = ctx->idx * 4;
+
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
+ if (ret)
+ return ret;
+ }
break;
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 5/8] bpf ppc64: Add BPF_PROBE_MEM support for JIT
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai, Hari Bathini
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
BPF load instruction with BPF_PROBE_MEM mode can cause a fault
inside kernel. Append exception table for such instructions
within BPF program.
Unlike other archs which uses extable 'fixup' field to pass dest_reg
and nip, BPF exception table on PowerPC follows the generic PowerPC
exception table design, where it populates both fixup and extable
sections within BPF program. fixup section contains two instructions,
first instruction clears dest_reg and 2nd jumps to next instruction
in the BPF code. extable 'insn' field contains relative offset of
the instruction and 'fixup' field contains relative offset of the
fixup entry. Example layout of BPF program with extable present:
+------------------+
| |
| |
0x4020 -->| ld r27,4(r3) |
| |
| |
0x40ac -->| lwz r3,0(r4) |
| |
| |
|------------------|
0x4280 -->| li r27,0 | \ fixup entry
| b 0x4024 | /
0x4288 -->| li r3,0 |
| b 0x40b0 |
|------------------|
0x4290 -->| insn=0xfffffd90 | \ extable entry
| fixup=0xffffffec | /
0x4298 -->| insn=0xfffffe14 |
| fixup=0xffffffec |
+------------------+
(Addresses shown here are chosen random, not real)
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* Used JITing code after refactoring.
* Replaced 'xor reg,reg,reg' with 'li reg,0' where appropriate.
* Avoided unnecessary init during declaration.
arch/powerpc/net/bpf_jit.h | 5 ++-
arch/powerpc/net/bpf_jit_comp.c | 25 ++++++++++----
arch/powerpc/net/bpf_jit_comp32.c | 2 +-
arch/powerpc/net/bpf_jit_comp64.c | 57 ++++++++++++++++++++++++++++++-
4 files changed, 80 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 0c8f885b8f48..6357c71c26eb 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -141,8 +141,11 @@ struct codegen_context {
unsigned int idx;
unsigned int stack_size;
int b2p[ARRAY_SIZE(b2p)];
+ unsigned int exentry_idx;
};
+#define BPF_FIXUP_LEN 8 /* Two instructions */
+
static inline void bpf_flush_icache(void *start, void *end)
{
smp_wmb(); /* smp write barrier */
@@ -166,7 +169,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs);
+ u32 *addrs, int pass);
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index c5c9e8ad1de7..e92bd79d3bac 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -101,6 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
struct bpf_prog *tmp_fp;
bool bpf_blinded = false;
bool extra_pass = false;
+ u32 extable_len;
+ u32 fixup_len;
if (!fp->jit_requested)
return org_fp;
@@ -131,7 +133,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
image = jit_data->image;
bpf_hdr = jit_data->header;
proglen = jit_data->proglen;
- alloclen = proglen + FUNCTION_DESCR_SIZE;
extra_pass = true;
goto skip_init_ctx;
}
@@ -149,7 +150,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
/* Scouting faux-generate pass 0 */
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
/* We hit something illegal or unsupported. */
fp = org_fp;
goto out_addrs;
@@ -162,7 +163,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
*/
if (cgctx.seen & SEEN_TAILCALL) {
cgctx.idx = 0;
- if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
+ if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
fp = org_fp;
goto out_addrs;
}
@@ -177,8 +178,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
bpf_jit_build_prologue(0, &cgctx);
bpf_jit_build_epilogue(0, &cgctx);
+ fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN;
+ extable_len = fp->aux->num_exentries * sizeof(struct exception_table_entry);
+
proglen = cgctx.idx * 4;
- alloclen = proglen + FUNCTION_DESCR_SIZE;
+ alloclen = proglen + FUNCTION_DESCR_SIZE + fixup_len + extable_len;
bpf_hdr = bpf_jit_binary_alloc(alloclen, &image, 4, bpf_jit_fill_ill_insns);
if (!bpf_hdr) {
@@ -186,6 +190,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
goto out_addrs;
}
+ if (extable_len) {
+ fp->aux->extable = (void *)image + FUNCTION_DESCR_SIZE +
+ proglen + fixup_len;
+ }
+
skip_init_ctx:
code_base = (u32 *)(image + FUNCTION_DESCR_SIZE);
@@ -210,7 +219,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
/* Now build the prologue, body code & epilogue for real. */
cgctx.idx = 0;
bpf_jit_build_prologue(code_base, &cgctx);
- bpf_jit_build_body(fp, code_base, &cgctx, addrs);
+ if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) {
+ bpf_jit_binary_free(bpf_hdr);
+ fp = org_fp;
+ goto out_addrs;
+ }
bpf_jit_build_epilogue(code_base, &cgctx);
if (bpf_jit_enable > 1)
@@ -234,7 +247,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
fp->bpf_func = (void *)image;
fp->jited = 1;
- fp->jited_len = alloclen;
+ fp->jited_len = proglen + FUNCTION_DESCR_SIZE;
bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + (bpf_hdr->pages * PAGE_SIZE));
bpf_jit_binary_lock_ro(bpf_hdr);
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index c8ae14c316e3..94641b7be387 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs)
+ u32 *addrs, int pass)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 78b28f25555c..2fc10995f243 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -270,9 +270,54 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* out: */
}
+/*
+ * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
+ * this function, as this only applies to BPF_PROBE_MEM, for now.
+ */
+static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
+ struct codegen_context *ctx, int dst_reg)
+{
+ off_t offset;
+ unsigned long pc;
+ struct exception_table_entry *ex;
+ u32 *fixup;
+
+ /* Populate extable entries only in the last pass */
+ if (pass != 2)
+ return 0;
+
+ if (!fp->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
+ return -EINVAL;
+
+ pc = (unsigned long)&image[ctx->idx - 1];
+
+ fixup = (void *)fp->aux->extable -
+ (fp->aux->num_exentries * BPF_FIXUP_LEN) +
+ (ctx->exentry_idx * BPF_FIXUP_LEN);
+
+ fixup[0] = PPC_RAW_LI(dst_reg, 0);
+ fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
+
+ ex = &fp->aux->extable[ctx->exentry_idx];
+
+ offset = pc - (long)&ex->insn;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->insn = offset;
+
+ offset = (long)fixup - (long)&ex->fixup;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->fixup = offset;
+
+ ctx->exentry_idx++;
+ return 0;
+}
+
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
- u32 *addrs)
+ u32 *addrs, int pass)
{
const struct bpf_insn *insn = fp->insnsi;
int flen = fp->len;
@@ -714,12 +759,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_H:
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_W:
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
@@ -737,6 +786,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
addrs[++i] = ctx->idx * 4;
+
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
+ if (ret)
+ return ret;
+ }
break;
/*
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 6/8] bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev,
Ravi Bangoria, Hari Bathini
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
On PPC64 with KUAP enabled, any kernel code which wants to
access userspace needs to be surrounded by disable-enable KUAP.
But that is not happening for BPF_PROBE_MEM load instruction.
So, when BPF program tries to access invalid userspace address,
page-fault handler considers it as bad KUAP fault:
Kernel attempted to read user page (d0000000) - exploit attempt? (uid: 0)
Considering the fact that PTR_TO_BTF_ID (which uses BPF_PROBE_MEM
mode) could either be a valid kernel pointer or NULL but should
never be a pointer to userspace address, execute BPF_PROBE_MEM load
only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
This will catch NULL, valid or invalid userspace pointers. Only bad
kernel pointer will be handled by BPF exception table.
[Alexei suggested for x86]
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* Refactored the code based on Christophe's comments.
arch/powerpc/net/bpf_jit_comp64.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 2fc10995f243..eb28dbc67151 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -769,6 +769,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
+ /*
+ * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
+ * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
+ * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
+ */
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ unsigned int adjusted_idx;
+
+ /*
+ * Check if 'off' is word aligned because PPC_BPF_LL()
+ * (BPF_DW case) generates two instructions if 'off' is not
+ * word-aligned and one instruction otherwise.
+ */
+ adjusted_idx = ((BPF_SIZE(code) == BPF_DW) && (off & 3)) ? 1 : 0;
+
+ EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], src_reg, off));
+ PPC_LI64(b2p[TMP_REG_2], TASK_SIZE_MAX);
+ EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
+ PPC_BCC(COND_GT, (ctx->idx + 4) * 4);
+ EMIT(PPC_RAW_LI(dst_reg, 0));
+ PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
+ }
+
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 6/8] bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai, Hari Bathini
From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
On PPC64 with KUAP enabled, any kernel code which wants to
access userspace needs to be surrounded by disable-enable KUAP.
But that is not happening for BPF_PROBE_MEM load instruction.
So, when BPF program tries to access invalid userspace address,
page-fault handler considers it as bad KUAP fault:
Kernel attempted to read user page (d0000000) - exploit attempt? (uid: 0)
Considering the fact that PTR_TO_BTF_ID (which uses BPF_PROBE_MEM
mode) could either be a valid kernel pointer or NULL but should
never be a pointer to userspace address, execute BPF_PROBE_MEM load
only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
This will catch NULL, valid or invalid userspace pointers. Only bad
kernel pointer will be handled by BPF exception table.
[Alexei suggested for x86]
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* Refactored the code based on Christophe's comments.
arch/powerpc/net/bpf_jit_comp64.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 2fc10995f243..eb28dbc67151 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -769,6 +769,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
+ /*
+ * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
+ * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
+ * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
+ */
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ unsigned int adjusted_idx;
+
+ /*
+ * Check if 'off' is word aligned because PPC_BPF_LL()
+ * (BPF_DW case) generates two instructions if 'off' is not
+ * word-aligned and one instruction otherwise.
+ */
+ adjusted_idx = ((BPF_SIZE(code) == BPF_DW) && (off & 3)) ? 1 : 0;
+
+ EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], src_reg, off));
+ PPC_LI64(b2p[TMP_REG_2], TASK_SIZE_MAX);
+ EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
+ PPC_BCC(COND_GT, (ctx->idx + 4) * 4);
+ EMIT(PPC_RAW_LI(dst_reg, 0));
+ PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
+ }
+
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 7/8] bpf ppc32: Add BPF_PROBE_MEM support for JIT
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev, Hari Bathini
BPF load instruction with BPF_PROBE_MEM mode can cause a fault
inside kernel. Append exception table for such instructions
within BPF program.
Unlike other archs which uses extable 'fixup' field to pass dest_reg
and nip, BPF exception table on PowerPC follows the generic PowerPC
exception table design, where it populates both fixup and extable
sections within BPF program. fixup section contains 3 instructions,
first 2 instructions clear dest_reg (lower & higher 32-bit registers)
and last instruction jumps to next instruction in the BPF code.
extable 'insn' field contains relative offset of the instruction and
'fixup' field contains relative offset of the fixup entry. Example
layout of BPF program with extable present:
+------------------+
| |
| |
0x4020 -->| lwz r28,4(r4) |
| |
| |
0x40ac -->| lwz r3,0(r24) |
| lwz r4,4(r24) |
| |
| |
|------------------|
0x4278 -->| li r28,0 | \
| li r27,0 | | fixup entry
| b 0x4024 | /
0x4284 -->| li r4,0 |
| li r3,0 |
| b 0x40b4 |
|------------------|
0x4290 -->| insn=0xfffffd90 | \ extable entry
| fixup=0xffffffe4 | /
0x4298 -->| insn=0xfffffe14 |
| fixup=0xffffffe8 |
+------------------+
(Addresses shown here are chosen random, not real)
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to add BPF_PROBE_MEM support for PPC32.
arch/powerpc/net/bpf_jit.h | 7 +++++
arch/powerpc/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++
arch/powerpc/net/bpf_jit_comp32.c | 30 +++++++++++++++++++
arch/powerpc/net/bpf_jit_comp64.c | 48 ++---------------------------
4 files changed, 89 insertions(+), 46 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 6357c71c26eb..6a591ef88006 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -144,7 +144,11 @@ struct codegen_context {
unsigned int exentry_idx;
};
+#ifdef CONFIG_PPC32
+#define BPF_FIXUP_LEN 12 /* Three instructions */
+#else
#define BPF_FIXUP_LEN 8 /* Two instructions */
+#endif
static inline void bpf_flush_icache(void *start, void *end)
{
@@ -174,6 +178,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
+int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
+ int insn_idx, int jmp_off, int dst_reg);
+
#endif
#endif
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index e92bd79d3bac..a1753b8c78c8 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -271,3 +271,53 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
return fp;
}
+
+/*
+ * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
+ * this function, as this only applies to BPF_PROBE_MEM, for now.
+ */
+int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
+ int insn_idx, int jmp_off, int dst_reg)
+{
+ off_t offset;
+ unsigned long pc;
+ struct exception_table_entry *ex;
+ u32 *fixup;
+
+ /* Populate extable entries only in the last pass */
+ if (pass != 2)
+ return 0;
+
+ if (!fp->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
+ return -EINVAL;
+
+ pc = (unsigned long)&image[insn_idx];
+
+ fixup = (void *)fp->aux->extable -
+ (fp->aux->num_exentries * BPF_FIXUP_LEN) +
+ (ctx->exentry_idx * BPF_FIXUP_LEN);
+
+ fixup[0] = PPC_RAW_LI(dst_reg, 0);
+#ifdef CONFIG_PPC32
+ fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register too */
+ fixup[2] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[2]);
+#else
+ fixup[1] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[1]);
+#endif
+
+ ex = &fp->aux->extable[ctx->exentry_idx];
+
+ offset = pc - (long)&ex->insn;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->insn = offset;
+
+ offset = (long)fixup - (long)&ex->fixup;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->fixup = offset;
+
+ ctx->exentry_idx++;
+ return 0;
+}
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index 94641b7be387..c6262289dcc4 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -811,12 +811,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_H:
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_W:
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
@@ -835,6 +839,32 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if ((size != BPF_DW) && !fp->aux->verifier_zext)
EMIT(PPC_RAW_LI(dst_reg_h, 0));
+
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ int insn_idx = ctx->idx - 1;
+ int jmp_off = 4;
+
+ /*
+ * In case of BPF_DW, two lwz instructions are emitted, one
+ * for higher 32-bit and another for lower 32-bit. So, set
+ * ex->insn to the first of the two and jump over both
+ * instructions in fixup.
+ *
+ * Similarly, with !verifier_zext, two instructions are
+ * emitted for BPF_B/H/W case. So, set ex-insn to the
+ * instruction that could fault and skip over both
+ * instructions.
+ */
+ if ((size == BPF_DW) || !fp->aux->verifier_zext) {
+ insn_idx -= 1;
+ jmp_off += 4;
+ }
+
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, insn_idx,
+ jmp_off, dst_reg);
+ if (ret)
+ return ret;
+ }
break;
/*
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index eb28dbc67151..10cc9f04843c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -270,51 +270,6 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* out: */
}
-/*
- * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
- * this function, as this only applies to BPF_PROBE_MEM, for now.
- */
-static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
- struct codegen_context *ctx, int dst_reg)
-{
- off_t offset;
- unsigned long pc;
- struct exception_table_entry *ex;
- u32 *fixup;
-
- /* Populate extable entries only in the last pass */
- if (pass != 2)
- return 0;
-
- if (!fp->aux->extable ||
- WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
- return -EINVAL;
-
- pc = (unsigned long)&image[ctx->idx - 1];
-
- fixup = (void *)fp->aux->extable -
- (fp->aux->num_exentries * BPF_FIXUP_LEN) +
- (ctx->exentry_idx * BPF_FIXUP_LEN);
-
- fixup[0] = PPC_RAW_LI(dst_reg, 0);
- fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
-
- ex = &fp->aux->extable[ctx->exentry_idx];
-
- offset = pc - (long)&ex->insn;
- if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
- return -ERANGE;
- ex->insn = offset;
-
- offset = (long)fixup - (long)&ex->fixup;
- if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
- return -ERANGE;
- ex->fixup = offset;
-
- ctx->exentry_idx++;
- return 0;
-}
-
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
u32 *addrs, int pass)
@@ -811,7 +766,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
addrs[++i] = ctx->idx * 4;
if (BPF_MODE(code) == BPF_PROBE_MEM) {
- ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, ctx->idx - 1,
+ 4, dst_reg);
if (ret)
return ret;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 7/8] bpf ppc32: Add BPF_PROBE_MEM support for JIT
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai, Hari Bathini
BPF load instruction with BPF_PROBE_MEM mode can cause a fault
inside kernel. Append exception table for such instructions
within BPF program.
Unlike other archs which uses extable 'fixup' field to pass dest_reg
and nip, BPF exception table on PowerPC follows the generic PowerPC
exception table design, where it populates both fixup and extable
sections within BPF program. fixup section contains 3 instructions,
first 2 instructions clear dest_reg (lower & higher 32-bit registers)
and last instruction jumps to next instruction in the BPF code.
extable 'insn' field contains relative offset of the instruction and
'fixup' field contains relative offset of the fixup entry. Example
layout of BPF program with extable present:
+------------------+
| |
| |
0x4020 -->| lwz r28,4(r4) |
| |
| |
0x40ac -->| lwz r3,0(r24) |
| lwz r4,4(r24) |
| |
| |
|------------------|
0x4278 -->| li r28,0 | \
| li r27,0 | | fixup entry
| b 0x4024 | /
0x4284 -->| li r4,0 |
| li r3,0 |
| b 0x40b4 |
|------------------|
0x4290 -->| insn=0xfffffd90 | \ extable entry
| fixup=0xffffffe4 | /
0x4298 -->| insn=0xfffffe14 |
| fixup=0xffffffe8 |
+------------------+
(Addresses shown here are chosen random, not real)
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to add BPF_PROBE_MEM support for PPC32.
arch/powerpc/net/bpf_jit.h | 7 +++++
arch/powerpc/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++
arch/powerpc/net/bpf_jit_comp32.c | 30 +++++++++++++++++++
arch/powerpc/net/bpf_jit_comp64.c | 48 ++---------------------------
4 files changed, 89 insertions(+), 46 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 6357c71c26eb..6a591ef88006 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -144,7 +144,11 @@ struct codegen_context {
unsigned int exentry_idx;
};
+#ifdef CONFIG_PPC32
+#define BPF_FIXUP_LEN 12 /* Three instructions */
+#else
#define BPF_FIXUP_LEN 8 /* Two instructions */
+#endif
static inline void bpf_flush_icache(void *start, void *end)
{
@@ -174,6 +178,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
+int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
+ int insn_idx, int jmp_off, int dst_reg);
+
#endif
#endif
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index e92bd79d3bac..a1753b8c78c8 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -271,3 +271,53 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
return fp;
}
+
+/*
+ * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
+ * this function, as this only applies to BPF_PROBE_MEM, for now.
+ */
+int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
+ int insn_idx, int jmp_off, int dst_reg)
+{
+ off_t offset;
+ unsigned long pc;
+ struct exception_table_entry *ex;
+ u32 *fixup;
+
+ /* Populate extable entries only in the last pass */
+ if (pass != 2)
+ return 0;
+
+ if (!fp->aux->extable ||
+ WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
+ return -EINVAL;
+
+ pc = (unsigned long)&image[insn_idx];
+
+ fixup = (void *)fp->aux->extable -
+ (fp->aux->num_exentries * BPF_FIXUP_LEN) +
+ (ctx->exentry_idx * BPF_FIXUP_LEN);
+
+ fixup[0] = PPC_RAW_LI(dst_reg, 0);
+#ifdef CONFIG_PPC32
+ fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register too */
+ fixup[2] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[2]);
+#else
+ fixup[1] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[1]);
+#endif
+
+ ex = &fp->aux->extable[ctx->exentry_idx];
+
+ offset = pc - (long)&ex->insn;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->insn = offset;
+
+ offset = (long)fixup - (long)&ex->fixup;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+ ex->fixup = offset;
+
+ ctx->exentry_idx++;
+ return 0;
+}
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index 94641b7be387..c6262289dcc4 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -811,12 +811,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
*/
/* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_H:
/* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_W:
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
@@ -835,6 +839,32 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if ((size != BPF_DW) && !fp->aux->verifier_zext)
EMIT(PPC_RAW_LI(dst_reg_h, 0));
+
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ int insn_idx = ctx->idx - 1;
+ int jmp_off = 4;
+
+ /*
+ * In case of BPF_DW, two lwz instructions are emitted, one
+ * for higher 32-bit and another for lower 32-bit. So, set
+ * ex->insn to the first of the two and jump over both
+ * instructions in fixup.
+ *
+ * Similarly, with !verifier_zext, two instructions are
+ * emitted for BPF_B/H/W case. So, set ex-insn to the
+ * instruction that could fault and skip over both
+ * instructions.
+ */
+ if ((size == BPF_DW) || !fp->aux->verifier_zext) {
+ insn_idx -= 1;
+ jmp_off += 4;
+ }
+
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, insn_idx,
+ jmp_off, dst_reg);
+ if (ret)
+ return ret;
+ }
break;
/*
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index eb28dbc67151..10cc9f04843c 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -270,51 +270,6 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
/* out: */
}
-/*
- * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
- * this function, as this only applies to BPF_PROBE_MEM, for now.
- */
-static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
- struct codegen_context *ctx, int dst_reg)
-{
- off_t offset;
- unsigned long pc;
- struct exception_table_entry *ex;
- u32 *fixup;
-
- /* Populate extable entries only in the last pass */
- if (pass != 2)
- return 0;
-
- if (!fp->aux->extable ||
- WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
- return -EINVAL;
-
- pc = (unsigned long)&image[ctx->idx - 1];
-
- fixup = (void *)fp->aux->extable -
- (fp->aux->num_exentries * BPF_FIXUP_LEN) +
- (ctx->exentry_idx * BPF_FIXUP_LEN);
-
- fixup[0] = PPC_RAW_LI(dst_reg, 0);
- fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
-
- ex = &fp->aux->extable[ctx->exentry_idx];
-
- offset = pc - (long)&ex->insn;
- if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
- return -ERANGE;
- ex->insn = offset;
-
- offset = (long)fixup - (long)&ex->fixup;
- if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
- return -ERANGE;
- ex->fixup = offset;
-
- ctx->exentry_idx++;
- return 0;
-}
-
/* Assemble the body code between the prologue & epilogue */
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
u32 *addrs, int pass)
@@ -811,7 +766,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
addrs[++i] = ctx->idx * 4;
if (BPF_MODE(code) == BPF_PROBE_MEM) {
- ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
+ ret = bpf_add_extable_entry(fp, image, pass, ctx, ctx->idx - 1,
+ 4, dst_reg);
if (ret)
return ret;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 8/8] bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 15:30 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: christophe.leroy, paulus, andrii, kafai, songliubraving, yhs,
john.fastabend, kpsingh, netdev, bpf, linuxppc-dev, Hari Bathini
With KUAP enabled, any kernel code which wants to access userspace
needs to be surrounded by disable-enable KUAP. But that is not
happening for BPF_PROBE_MEM load instruction. Though PPC32 does not
support read protection, considering the fact that PTR_TO_BTF_ID
(which uses BPF_PROBE_MEM mode) could either be a valid kernel pointer
or NULL but should never be a pointer to userspace address, execute
BPF_PROBE_MEM load only if addr > TASK_SIZE_MAX, otherwise set
dst_reg=0 and move on.
This will catch NULL, valid or invalid userspace pointers. Only bad
kernel pointer will be handled by BPF exception table.
[Alexei suggested for x86]
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to handle bad userspace pointers on PPC32.
arch/powerpc/net/bpf_jit_comp32.c | 39 +++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index c6262289dcc4..faa8047fcf4a 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -821,6 +821,45 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
+ /*
+ * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
+ * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
+ * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
+ */
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ bool extra_insn_needed = false;
+ unsigned int adjusted_idx;
+
+ /*
+ * For BPF_DW case, "li reg_h,0" would be needed when
+ * !fp->aux->verifier_zext. Adjust conditional branch'ing
+ * address accordingly.
+ */
+ if ((size == BPF_DW) && !fp->aux->verifier_zext)
+ extra_insn_needed = true;
+
+ /*
+ * Need to jump two instructions instead of one for BPF_DW case
+ * as there are two load instructions for dst_reg_h & dst_reg
+ * respectively.
+ */
+ adjusted_idx = (size == BPF_DW) ? 1 : 0;
+
+ EMIT(PPC_RAW_ADDI(b2p[TMP_REG], src_reg, off));
+ PPC_LI32(_R0, TASK_SIZE_MAX);
+ EMIT(PPC_RAW_CMPLW(b2p[TMP_REG], _R0));
+ PPC_BCC(COND_GT, (ctx->idx + 4 + (extra_insn_needed ? 1 : 0)) * 4);
+ EMIT(PPC_RAW_LI(dst_reg, 0));
+ /*
+ * Note that "li reg_h,0" is emitted for BPF_B/H/W case,
+ * if necessary. So, jump there insted of emitting an
+ * additional "li reg_h,0" instruction.
+ */
+ if (extra_insn_needed)
+ EMIT(PPC_RAW_LI(dst_reg_h, 0));
+ PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
+ }
+
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH v2 8/8] bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
@ 2021-09-17 15:30 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-17 15:30 UTC (permalink / raw)
To: naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai, Hari Bathini
With KUAP enabled, any kernel code which wants to access userspace
needs to be surrounded by disable-enable KUAP. But that is not
happening for BPF_PROBE_MEM load instruction. Though PPC32 does not
support read protection, considering the fact that PTR_TO_BTF_ID
(which uses BPF_PROBE_MEM mode) could either be a valid kernel pointer
or NULL but should never be a pointer to userspace address, execute
BPF_PROBE_MEM load only if addr > TASK_SIZE_MAX, otherwise set
dst_reg=0 and move on.
This will catch NULL, valid or invalid userspace pointers. Only bad
kernel pointer will be handled by BPF exception table.
[Alexei suggested for x86]
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
---
Changes in v2:
* New patch to handle bad userspace pointers on PPC32.
arch/powerpc/net/bpf_jit_comp32.c | 39 +++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index c6262289dcc4..faa8047fcf4a 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -821,6 +821,45 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
+ /*
+ * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
+ * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
+ * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
+ */
+ if (BPF_MODE(code) == BPF_PROBE_MEM) {
+ bool extra_insn_needed = false;
+ unsigned int adjusted_idx;
+
+ /*
+ * For BPF_DW case, "li reg_h,0" would be needed when
+ * !fp->aux->verifier_zext. Adjust conditional branch'ing
+ * address accordingly.
+ */
+ if ((size == BPF_DW) && !fp->aux->verifier_zext)
+ extra_insn_needed = true;
+
+ /*
+ * Need to jump two instructions instead of one for BPF_DW case
+ * as there are two load instructions for dst_reg_h & dst_reg
+ * respectively.
+ */
+ adjusted_idx = (size == BPF_DW) ? 1 : 0;
+
+ EMIT(PPC_RAW_ADDI(b2p[TMP_REG], src_reg, off));
+ PPC_LI32(_R0, TASK_SIZE_MAX);
+ EMIT(PPC_RAW_CMPLW(b2p[TMP_REG], _R0));
+ PPC_BCC(COND_GT, (ctx->idx + 4 + (extra_insn_needed ? 1 : 0)) * 4);
+ EMIT(PPC_RAW_LI(dst_reg, 0));
+ /*
+ * Note that "li reg_h,0" is emitted for BPF_B/H/W case,
+ * if necessary. So, jump there insted of emitting an
+ * additional "li reg_h,0" instruction.
+ */
+ if (extra_insn_needed)
+ EMIT(PPC_RAW_LI(dst_reg_h, 0));
+ PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
+ }
+
switch (size) {
case BPF_B:
EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
--
2.31.1
^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: [PATCH v2 1/8] bpf powerpc: Remove unused SEEN_STACK
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:02 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:02 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev, Ravi Bangoria
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> SEEN_STACK is unused on PowerPC. Remove it. Also, have
> SEEN_TAILCALL use 0x40000000.
>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Reviewed-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
>
> * No changes in v2.
>
>
> arch/powerpc/net/bpf_jit.h | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 99fad093f43e..d6267e93027a 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -116,8 +116,7 @@ static inline bool is_nearbranch(int offset)
> #define COND_LE (CR0_GT | COND_CMP_FALSE)
>
> #define SEEN_FUNC 0x20000000 /* might call external helpers */
> -#define SEEN_STACK 0x40000000 /* uses BPF stack */
> -#define SEEN_TAILCALL 0x80000000 /* uses tail calls */
> +#define SEEN_TAILCALL 0x40000000 /* uses tail calls */
>
> #define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */
> #define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 1/8] bpf powerpc: Remove unused SEEN_STACK
@ 2021-09-17 16:02 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:02 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> SEEN_STACK is unused on PowerPC. Remove it. Also, have
> SEEN_TAILCALL use 0x40000000.
>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Reviewed-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
>
> * No changes in v2.
>
>
> arch/powerpc/net/bpf_jit.h | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 99fad093f43e..d6267e93027a 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -116,8 +116,7 @@ static inline bool is_nearbranch(int offset)
> #define COND_LE (CR0_GT | COND_CMP_FALSE)
>
> #define SEEN_FUNC 0x20000000 /* might call external helpers */
> -#define SEEN_STACK 0x40000000 /* uses BPF stack */
> -#define SEEN_TAILCALL 0x80000000 /* uses tail calls */
> +#define SEEN_TAILCALL 0x40000000 /* uses tail calls */
>
> #define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */
> #define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:10 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:10 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
Could you describe a bit more what you are refactoring exactly ?
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to refactor a bit of JITing code.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
> 2 files changed, 68 insertions(+), 58 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index b60b59426a24..c8ae14c316e3 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
> - u32 dst_reg_h = dst_reg - 1;
> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
> - u32 src_reg_h = src_reg - 1;
> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
> + u32 true_cond, code = insn[i].code;
> + u32 dst_reg_h = dst_reg - 1;
> + u32 src_reg_h = src_reg - 1;
All changes above seems unneeded and not linked to the current patch.
Please leave cosmetic changes outside and focus on necessary changes.
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> u64 func_addr;
> - u32 true_cond;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /*
> * BPF_LDX
> */
> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> + /* dst = *(u8 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_B:
> + /* dst = *(u16 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_H:
> + /* dst = *(u32 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_W:
> + /* dst = *(u64 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_DW:
Why changing the location of the comments ? I found it more readable before.
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> + break;
> + }
BPF_B, BPF_H, ... are not part of an enum. Are you sure GCC is happy to
have no default ?
> +
> + if ((size != BPF_DW) && !fp->aux->verifier_zext)
You don't need () around size != BPF_DW
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> break;
> - case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> - break;
>
> /*
> * Doubleword load
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2a87da50d9a4..78b28f25555c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = b2p[insn[i].dst_reg];
> u32 src_reg = b2p[insn[i].src_reg];
> + u32 tmp_idx, code = insn[i].code;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> - u64 func_addr;
> - u64 imm64;
> + u64 func_addr, imm64;
> u32 true_cond;
> - u32 tmp_idx;
All changes other than the addition of 'size' seems unneeded and not
linked to the current patch.
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
> if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> + if ((size == BPF_B) || (size == BPF_H))
> + EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> + else /* size == BPF_W || size == BPF_DW */
> + PPC_LI32(b2p[TMP_REG_1], imm);
I think you can use PPC_LI32() for all cases, it should generate the
same code.
> src_reg = b2p[TMP_REG_1];
> }
> - PPC_BPF_STL(src_reg, dst_reg, off);
> +
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_STL(src_reg, dst_reg, off);
> + break;
> + }
> break;
>
> /*
> @@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> - PPC_BPF_LL(dst_reg, src_reg, off);
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_LL(dst_reg, src_reg, off);
> + break;
> + }
> +
> + if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> + addrs[++i] = ctx->idx * 4;
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
@ 2021-09-17 16:10 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:10 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
Could you describe a bit more what you are refactoring exactly ?
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to refactor a bit of JITing code.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
> 2 files changed, 68 insertions(+), 58 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index b60b59426a24..c8ae14c316e3 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
> - u32 dst_reg_h = dst_reg - 1;
> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
> - u32 src_reg_h = src_reg - 1;
> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
> + u32 true_cond, code = insn[i].code;
> + u32 dst_reg_h = dst_reg - 1;
> + u32 src_reg_h = src_reg - 1;
All changes above seems unneeded and not linked to the current patch.
Please leave cosmetic changes outside and focus on necessary changes.
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> u64 func_addr;
> - u32 true_cond;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /*
> * BPF_LDX
> */
> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> + /* dst = *(u8 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_B:
> + /* dst = *(u16 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_H:
> + /* dst = *(u32 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_W:
> + /* dst = *(u64 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_DW:
Why changing the location of the comments ? I found it more readable before.
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> + break;
> + }
BPF_B, BPF_H, ... are not part of an enum. Are you sure GCC is happy to
have no default ?
> +
> + if ((size != BPF_DW) && !fp->aux->verifier_zext)
You don't need () around size != BPF_DW
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> break;
> - case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> - break;
>
> /*
> * Doubleword load
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2a87da50d9a4..78b28f25555c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = b2p[insn[i].dst_reg];
> u32 src_reg = b2p[insn[i].src_reg];
> + u32 tmp_idx, code = insn[i].code;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> - u64 func_addr;
> - u64 imm64;
> + u64 func_addr, imm64;
> u32 true_cond;
> - u32 tmp_idx;
All changes other than the addition of 'size' seems unneeded and not
linked to the current patch.
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
> if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> + if ((size == BPF_B) || (size == BPF_H))
> + EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> + else /* size == BPF_W || size == BPF_DW */
> + PPC_LI32(b2p[TMP_REG_1], imm);
I think you can use PPC_LI32() for all cases, it should generate the
same code.
> src_reg = b2p[TMP_REG_1];
> }
> - PPC_BPF_STL(src_reg, dst_reg, off);
> +
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_STL(src_reg, dst_reg, off);
> + break;
> + }
> break;
>
> /*
> @@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> - PPC_BPF_LL(dst_reg, src_reg, off);
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_LL(dst_reg, src_reg, off);
> + break;
> + }
> +
> + if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> + addrs[++i] = ctx->idx * 4;
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 4/8] powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:14 ` LEROY Christophe
-1 siblings, 0 replies; 36+ messages in thread
From: LEROY Christophe @ 2021-09-17 16:14 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Define and use PPC_RAW_BRANCH() macro instead of open coding it. This
> macro is used while adding BPF_PROBE_MEM support.
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Reviewed-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
>
> Changes in v2:
> * New patch to introduce PPC_RAW_BRANCH() macro.
>
>
> arch/powerpc/include/asm/ppc-opcode.h | 2 ++
> arch/powerpc/net/bpf_jit.h | 4 ++--
> 2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
> index baea657bc868..f50213e2a3e0 100644
> --- a/arch/powerpc/include/asm/ppc-opcode.h
> +++ b/arch/powerpc/include/asm/ppc-opcode.h
> @@ -566,6 +566,8 @@
> #define PPC_RAW_MTSPR(spr, d) (0x7c0003a6 | ___PPC_RS(d) | __PPC_SPR(spr))
> #define PPC_RAW_EIEIO() (0x7c0006ac)
>
> +#define PPC_RAW_BRANCH(addr) (PPC_INST_BRANCH | ((addr) & 0x03fffffc))
> +
> /* Deal with instructions that older assemblers aren't aware of */
> #define PPC_BCCTR_FLUSH stringify_in_c(.long PPC_INST_BCCTR_FLUSH)
> #define PPC_CP_ABORT stringify_in_c(.long PPC_RAW_CP_ABORT)
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 411c63d945c7..0c8f885b8f48 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -24,8 +24,8 @@
> #define EMIT(instr) PLANT_INSTR(image, ctx->idx, instr)
>
> /* Long jump; (unconditional 'branch') */
> -#define PPC_JMP(dest) EMIT(PPC_INST_BRANCH | \
> - (((dest) - (ctx->idx * 4)) & 0x03fffffc))
> +#define PPC_JMP(dest) EMIT(PPC_RAW_BRANCH((dest) - (ctx->idx * 4)))
> +
> /* blr; (unconditional 'branch' with link) to absolute address */
> #define PPC_BL_ABS(dest) EMIT(PPC_INST_BL | \
> (((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 4/8] powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro
@ 2021-09-17 16:14 ` LEROY Christophe
0 siblings, 0 replies; 36+ messages in thread
From: LEROY Christophe @ 2021-09-17 16:14 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Define and use PPC_RAW_BRANCH() macro instead of open coding it. This
> macro is used while adding BPF_PROBE_MEM support.
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Reviewed-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
>
> Changes in v2:
> * New patch to introduce PPC_RAW_BRANCH() macro.
>
>
> arch/powerpc/include/asm/ppc-opcode.h | 2 ++
> arch/powerpc/net/bpf_jit.h | 4 ++--
> 2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
> index baea657bc868..f50213e2a3e0 100644
> --- a/arch/powerpc/include/asm/ppc-opcode.h
> +++ b/arch/powerpc/include/asm/ppc-opcode.h
> @@ -566,6 +566,8 @@
> #define PPC_RAW_MTSPR(spr, d) (0x7c0003a6 | ___PPC_RS(d) | __PPC_SPR(spr))
> #define PPC_RAW_EIEIO() (0x7c0006ac)
>
> +#define PPC_RAW_BRANCH(addr) (PPC_INST_BRANCH | ((addr) & 0x03fffffc))
> +
> /* Deal with instructions that older assemblers aren't aware of */
> #define PPC_BCCTR_FLUSH stringify_in_c(.long PPC_INST_BCCTR_FLUSH)
> #define PPC_CP_ABORT stringify_in_c(.long PPC_RAW_CP_ABORT)
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 411c63d945c7..0c8f885b8f48 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -24,8 +24,8 @@
> #define EMIT(instr) PLANT_INSTR(image, ctx->idx, instr)
>
> /* Long jump; (unconditional 'branch') */
> -#define PPC_JMP(dest) EMIT(PPC_INST_BRANCH | \
> - (((dest) - (ctx->idx * 4)) & 0x03fffffc))
> +#define PPC_JMP(dest) EMIT(PPC_RAW_BRANCH((dest) - (ctx->idx * 4)))
> +
> /* blr; (unconditional 'branch' with link) to absolute address */
> #define PPC_BL_ABS(dest) EMIT(PPC_INST_BL | \
> (((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 5/8] bpf ppc64: Add BPF_PROBE_MEM support for JIT
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:20 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:20 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev, Ravi Bangoria
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> BPF load instruction with BPF_PROBE_MEM mode can cause a fault
> inside kernel. Append exception table for such instructions
> within BPF program.
>
> Unlike other archs which uses extable 'fixup' field to pass dest_reg
> and nip, BPF exception table on PowerPC follows the generic PowerPC
> exception table design, where it populates both fixup and extable
> sections within BPF program. fixup section contains two instructions,
> first instruction clears dest_reg and 2nd jumps to next instruction
> in the BPF code. extable 'insn' field contains relative offset of
> the instruction and 'fixup' field contains relative offset of the
> fixup entry. Example layout of BPF program with extable present:
>
> +------------------+
> | |
> | |
> 0x4020 -->| ld r27,4(r3) |
> | |
> | |
> 0x40ac -->| lwz r3,0(r4) |
> | |
> | |
> |------------------|
> 0x4280 -->| li r27,0 | \ fixup entry
> | b 0x4024 | /
> 0x4288 -->| li r3,0 |
> | b 0x40b0 |
> |------------------|
> 0x4290 -->| insn=0xfffffd90 | \ extable entry
> | fixup=0xffffffec | /
> 0x4298 -->| insn=0xfffffe14 |
> | fixup=0xffffffec |
> +------------------+
>
> (Addresses shown here are chosen random, not real)
>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * Used JITing code after refactoring.
> * Replaced 'xor reg,reg,reg' with 'li reg,0' where appropriate.
> * Avoided unnecessary init during declaration.
>
>
> arch/powerpc/net/bpf_jit.h | 5 ++-
> arch/powerpc/net/bpf_jit_comp.c | 25 ++++++++++----
> arch/powerpc/net/bpf_jit_comp32.c | 2 +-
> arch/powerpc/net/bpf_jit_comp64.c | 57 ++++++++++++++++++++++++++++++-
> 4 files changed, 80 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 0c8f885b8f48..6357c71c26eb 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -141,8 +141,11 @@ struct codegen_context {
> unsigned int idx;
> unsigned int stack_size;
> int b2p[ARRAY_SIZE(b2p)];
> + unsigned int exentry_idx;
> };
>
> +#define BPF_FIXUP_LEN 8 /* Two instructions */
> +
> static inline void bpf_flush_icache(void *start, void *end)
> {
> smp_wmb(); /* smp write barrier */
> @@ -166,7 +169,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
>
> void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs);
> + u32 *addrs, int pass);
> void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_realloc_regs(struct codegen_context *ctx);
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index c5c9e8ad1de7..e92bd79d3bac 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -101,6 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> struct bpf_prog *tmp_fp;
> bool bpf_blinded = false;
> bool extra_pass = false;
> + u32 extable_len;
> + u32 fixup_len;
>
> if (!fp->jit_requested)
> return org_fp;
> @@ -131,7 +133,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> image = jit_data->image;
> bpf_hdr = jit_data->header;
> proglen = jit_data->proglen;
> - alloclen = proglen + FUNCTION_DESCR_SIZE;
> extra_pass = true;
> goto skip_init_ctx;
> }
> @@ -149,7 +150,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
>
> /* Scouting faux-generate pass 0 */
> - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
> + if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
> /* We hit something illegal or unsupported. */
> fp = org_fp;
> goto out_addrs;
> @@ -162,7 +163,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> */
> if (cgctx.seen & SEEN_TAILCALL) {
> cgctx.idx = 0;
> - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
> + if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
> fp = org_fp;
> goto out_addrs;
> }
> @@ -177,8 +178,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> bpf_jit_build_prologue(0, &cgctx);
> bpf_jit_build_epilogue(0, &cgctx);
>
> + fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN;
> + extable_len = fp->aux->num_exentries * sizeof(struct exception_table_entry);
> +
> proglen = cgctx.idx * 4;
> - alloclen = proglen + FUNCTION_DESCR_SIZE;
> + alloclen = proglen + FUNCTION_DESCR_SIZE + fixup_len + extable_len;
>
> bpf_hdr = bpf_jit_binary_alloc(alloclen, &image, 4, bpf_jit_fill_ill_insns);
> if (!bpf_hdr) {
> @@ -186,6 +190,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> goto out_addrs;
> }
>
> + if (extable_len) {
> + fp->aux->extable = (void *)image + FUNCTION_DESCR_SIZE +
> + proglen + fixup_len;
> + }
No { } for single lines statements (See kernel coding style)
> +
> skip_init_ctx:
> code_base = (u32 *)(image + FUNCTION_DESCR_SIZE);
>
> @@ -210,7 +219,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> /* Now build the prologue, body code & epilogue for real. */
> cgctx.idx = 0;
> bpf_jit_build_prologue(code_base, &cgctx);
> - bpf_jit_build_body(fp, code_base, &cgctx, addrs);
> + if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) {
> + bpf_jit_binary_free(bpf_hdr);
> + fp = org_fp;
> + goto out_addrs;
> + }
> bpf_jit_build_epilogue(code_base, &cgctx);
>
> if (bpf_jit_enable > 1)
> @@ -234,7 +247,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>
> fp->bpf_func = (void *)image;
> fp->jited = 1;
> - fp->jited_len = alloclen;
> + fp->jited_len = proglen + FUNCTION_DESCR_SIZE;
>
> bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + (bpf_hdr->pages * PAGE_SIZE));
> bpf_jit_binary_lock_ro(bpf_hdr);
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index c8ae14c316e3..94641b7be387 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
>
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs)
> + u32 *addrs, int pass)
> {
> const struct bpf_insn *insn = fp->insnsi;
> int flen = fp->len;
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 78b28f25555c..2fc10995f243 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -270,9 +270,54 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
> /* out: */
> }
>
> +/*
> + * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> + * this function, as this only applies to BPF_PROBE_MEM, for now.
> + */
> +static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
> + struct codegen_context *ctx, int dst_reg)
Patch 7 will reuse this function so put it in
arch/powerpc/net/bpf_jit_comp.c now instead of moving it later.
> +{
> + off_t offset;
> + unsigned long pc;
> + struct exception_table_entry *ex;
> + u32 *fixup;
> +
> + /* Populate extable entries only in the last pass */
> + if (pass != 2)
> + return 0;
> +
> + if (!fp->aux->extable ||
> + WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> + return -EINVAL;
> +
> + pc = (unsigned long)&image[ctx->idx - 1];
> +
> + fixup = (void *)fp->aux->extable -
> + (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> + (ctx->exentry_idx * BPF_FIXUP_LEN);
> +
> + fixup[0] = PPC_RAW_LI(dst_reg, 0);
> + fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
> +
> + ex = &fp->aux->extable[ctx->exentry_idx];
> +
> + offset = pc - (long)&ex->insn;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->insn = offset;
> +
> + offset = (long)fixup - (long)&ex->fixup;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->fixup = offset;
> +
> + ctx->exentry_idx++;
> + return 0;
> +}
> +
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs)
> + u32 *addrs, int pass)
> {
> const struct bpf_insn *insn = fp->insnsi;
> int flen = fp->len;
> @@ -714,12 +759,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_B:
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_H:
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_W:
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> @@ -737,6 +786,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
>
> if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> addrs[++i] = ctx->idx * 4;
> +
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
> + if (ret)
> + return ret;
> + }
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 5/8] bpf ppc64: Add BPF_PROBE_MEM support for JIT
@ 2021-09-17 16:20 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:20 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> BPF load instruction with BPF_PROBE_MEM mode can cause a fault
> inside kernel. Append exception table for such instructions
> within BPF program.
>
> Unlike other archs which uses extable 'fixup' field to pass dest_reg
> and nip, BPF exception table on PowerPC follows the generic PowerPC
> exception table design, where it populates both fixup and extable
> sections within BPF program. fixup section contains two instructions,
> first instruction clears dest_reg and 2nd jumps to next instruction
> in the BPF code. extable 'insn' field contains relative offset of
> the instruction and 'fixup' field contains relative offset of the
> fixup entry. Example layout of BPF program with extable present:
>
> +------------------+
> | |
> | |
> 0x4020 -->| ld r27,4(r3) |
> | |
> | |
> 0x40ac -->| lwz r3,0(r4) |
> | |
> | |
> |------------------|
> 0x4280 -->| li r27,0 | \ fixup entry
> | b 0x4024 | /
> 0x4288 -->| li r3,0 |
> | b 0x40b0 |
> |------------------|
> 0x4290 -->| insn=0xfffffd90 | \ extable entry
> | fixup=0xffffffec | /
> 0x4298 -->| insn=0xfffffe14 |
> | fixup=0xffffffec |
> +------------------+
>
> (Addresses shown here are chosen random, not real)
>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * Used JITing code after refactoring.
> * Replaced 'xor reg,reg,reg' with 'li reg,0' where appropriate.
> * Avoided unnecessary init during declaration.
>
>
> arch/powerpc/net/bpf_jit.h | 5 ++-
> arch/powerpc/net/bpf_jit_comp.c | 25 ++++++++++----
> arch/powerpc/net/bpf_jit_comp32.c | 2 +-
> arch/powerpc/net/bpf_jit_comp64.c | 57 ++++++++++++++++++++++++++++++-
> 4 files changed, 80 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 0c8f885b8f48..6357c71c26eb 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -141,8 +141,11 @@ struct codegen_context {
> unsigned int idx;
> unsigned int stack_size;
> int b2p[ARRAY_SIZE(b2p)];
> + unsigned int exentry_idx;
> };
>
> +#define BPF_FIXUP_LEN 8 /* Two instructions */
> +
> static inline void bpf_flush_icache(void *start, void *end)
> {
> smp_wmb(); /* smp write barrier */
> @@ -166,7 +169,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
>
> void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs);
> + u32 *addrs, int pass);
> void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_realloc_regs(struct codegen_context *ctx);
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index c5c9e8ad1de7..e92bd79d3bac 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -101,6 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> struct bpf_prog *tmp_fp;
> bool bpf_blinded = false;
> bool extra_pass = false;
> + u32 extable_len;
> + u32 fixup_len;
>
> if (!fp->jit_requested)
> return org_fp;
> @@ -131,7 +133,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> image = jit_data->image;
> bpf_hdr = jit_data->header;
> proglen = jit_data->proglen;
> - alloclen = proglen + FUNCTION_DESCR_SIZE;
> extra_pass = true;
> goto skip_init_ctx;
> }
> @@ -149,7 +150,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
>
> /* Scouting faux-generate pass 0 */
> - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
> + if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
> /* We hit something illegal or unsupported. */
> fp = org_fp;
> goto out_addrs;
> @@ -162,7 +163,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> */
> if (cgctx.seen & SEEN_TAILCALL) {
> cgctx.idx = 0;
> - if (bpf_jit_build_body(fp, 0, &cgctx, addrs)) {
> + if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
> fp = org_fp;
> goto out_addrs;
> }
> @@ -177,8 +178,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> bpf_jit_build_prologue(0, &cgctx);
> bpf_jit_build_epilogue(0, &cgctx);
>
> + fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN;
> + extable_len = fp->aux->num_exentries * sizeof(struct exception_table_entry);
> +
> proglen = cgctx.idx * 4;
> - alloclen = proglen + FUNCTION_DESCR_SIZE;
> + alloclen = proglen + FUNCTION_DESCR_SIZE + fixup_len + extable_len;
>
> bpf_hdr = bpf_jit_binary_alloc(alloclen, &image, 4, bpf_jit_fill_ill_insns);
> if (!bpf_hdr) {
> @@ -186,6 +190,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> goto out_addrs;
> }
>
> + if (extable_len) {
> + fp->aux->extable = (void *)image + FUNCTION_DESCR_SIZE +
> + proglen + fixup_len;
> + }
No { } for single lines statements (See kernel coding style)
> +
> skip_init_ctx:
> code_base = (u32 *)(image + FUNCTION_DESCR_SIZE);
>
> @@ -210,7 +219,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> /* Now build the prologue, body code & epilogue for real. */
> cgctx.idx = 0;
> bpf_jit_build_prologue(code_base, &cgctx);
> - bpf_jit_build_body(fp, code_base, &cgctx, addrs);
> + if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) {
> + bpf_jit_binary_free(bpf_hdr);
> + fp = org_fp;
> + goto out_addrs;
> + }
> bpf_jit_build_epilogue(code_base, &cgctx);
>
> if (bpf_jit_enable > 1)
> @@ -234,7 +247,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>
> fp->bpf_func = (void *)image;
> fp->jited = 1;
> - fp->jited_len = alloclen;
> + fp->jited_len = proglen + FUNCTION_DESCR_SIZE;
>
> bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + (bpf_hdr->pages * PAGE_SIZE));
> bpf_jit_binary_lock_ro(bpf_hdr);
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index c8ae14c316e3..94641b7be387 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -266,7 +266,7 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
>
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs)
> + u32 *addrs, int pass)
> {
> const struct bpf_insn *insn = fp->insnsi;
> int flen = fp->len;
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 78b28f25555c..2fc10995f243 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -270,9 +270,54 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
> /* out: */
> }
>
> +/*
> + * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> + * this function, as this only applies to BPF_PROBE_MEM, for now.
> + */
> +static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
> + struct codegen_context *ctx, int dst_reg)
Patch 7 will reuse this function so put it in
arch/powerpc/net/bpf_jit_comp.c now instead of moving it later.
> +{
> + off_t offset;
> + unsigned long pc;
> + struct exception_table_entry *ex;
> + u32 *fixup;
> +
> + /* Populate extable entries only in the last pass */
> + if (pass != 2)
> + return 0;
> +
> + if (!fp->aux->extable ||
> + WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> + return -EINVAL;
> +
> + pc = (unsigned long)&image[ctx->idx - 1];
> +
> + fixup = (void *)fp->aux->extable -
> + (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> + (ctx->exentry_idx * BPF_FIXUP_LEN);
> +
> + fixup[0] = PPC_RAW_LI(dst_reg, 0);
> + fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
> +
> + ex = &fp->aux->extable[ctx->exentry_idx];
> +
> + offset = pc - (long)&ex->insn;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->insn = offset;
> +
> + offset = (long)fixup - (long)&ex->fixup;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->fixup = offset;
> +
> + ctx->exentry_idx++;
> + return 0;
> +}
> +
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> - u32 *addrs)
> + u32 *addrs, int pass)
> {
> const struct bpf_insn *insn = fp->insnsi;
> int flen = fp->len;
> @@ -714,12 +759,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_B:
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_H:
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_W:
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> @@ -737,6 +786,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
>
> if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> addrs[++i] = ctx->idx * 4;
> +
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
> + if (ret)
> + return ret;
> + }
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:22 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:22 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to refactor a bit of JITing code.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
> 2 files changed, 68 insertions(+), 58 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index b60b59426a24..c8ae14c316e3 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
> - u32 dst_reg_h = dst_reg - 1;
> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
> - u32 src_reg_h = src_reg - 1;
> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
> + u32 true_cond, code = insn[i].code;
> + u32 dst_reg_h = dst_reg - 1;
> + u32 src_reg_h = src_reg - 1;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> u64 func_addr;
> - u32 true_cond;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /*
> * BPF_LDX
> */
> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> + /* dst = *(u8 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_B:
> + /* dst = *(u16 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_H:
> + /* dst = *(u32 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_W:
> + /* dst = *(u64 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_DW:
You have to add the 'fallthrough;' keyword to avoid build failure with
later versions of GCC.
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> + break;
> + }
> +
> + if ((size != BPF_DW) && !fp->aux->verifier_zext)
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> break;
> - case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> - break;
>
> /*
> * Doubleword load
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2a87da50d9a4..78b28f25555c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = b2p[insn[i].dst_reg];
> u32 src_reg = b2p[insn[i].src_reg];
> + u32 tmp_idx, code = insn[i].code;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> - u64 func_addr;
> - u64 imm64;
> + u64 func_addr, imm64;
> u32 true_cond;
> - u32 tmp_idx;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> - break;
You have to add the 'fallthrough;' keyword to avoid build failure with
later versions of GCC.
> case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
> if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> + if ((size == BPF_B) || (size == BPF_H))
> + EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> + else /* size == BPF_W || size == BPF_DW */
> + PPC_LI32(b2p[TMP_REG_1], imm);
> src_reg = b2p[TMP_REG_1];
> }
> - PPC_BPF_STL(src_reg, dst_reg, off);
> +
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_STL(src_reg, dst_reg, off);
> + break;
> + }
> break;
>
> /*
> @@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> - PPC_BPF_LL(dst_reg, src_reg, off);
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_LL(dst_reg, src_reg, off);
> + break;
> + }
> +
> + if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> + addrs[++i] = ctx->idx * 4;
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
@ 2021-09-17 16:22 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:22 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to refactor a bit of JITing code.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
> 2 files changed, 68 insertions(+), 58 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index b60b59426a24..c8ae14c316e3 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
> - u32 dst_reg_h = dst_reg - 1;
> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
> - u32 src_reg_h = src_reg - 1;
> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
> + u32 true_cond, code = insn[i].code;
> + u32 dst_reg_h = dst_reg - 1;
> + u32 src_reg_h = src_reg - 1;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> u64 func_addr;
> - u32 true_cond;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /*
> * BPF_LDX
> */
> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
> - break;
> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (!fp->aux->verifier_zext)
> + /* dst = *(u8 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_B:
> + /* dst = *(u16 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_H:
> + /* dst = *(u32 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_W:
> + /* dst = *(u64 *)(ul) (src + off) */
> + case BPF_LDX | BPF_MEM | BPF_DW:
You have to add the 'fallthrough;' keyword to avoid build failure with
later versions of GCC.
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> + break;
> + }
> +
> + if ((size != BPF_DW) && !fp->aux->verifier_zext)
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> break;
> - case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
> - EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
> - break;
>
> /*
> * Doubleword load
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2a87da50d9a4..78b28f25555c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -282,16 +282,15 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> u32 exit_addr = addrs[flen];
>
> for (i = 0; i < flen; i++) {
> - u32 code = insn[i].code;
> u32 dst_reg = b2p[insn[i].dst_reg];
> u32 src_reg = b2p[insn[i].src_reg];
> + u32 tmp_idx, code = insn[i].code;
> + u32 size = BPF_SIZE(code);
> s16 off = insn[i].off;
> s32 imm = insn[i].imm;
> bool func_addr_fixed;
> - u64 func_addr;
> - u64 imm64;
> + u64 func_addr, imm64;
> u32 true_cond;
> - u32 tmp_idx;
>
> /*
> * addrs[] maps a BPF bytecode address into a real offset from
> @@ -638,35 +637,34 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> - break;
> case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
> - if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> - src_reg = b2p[TMP_REG_1];
> - }
> - EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> - break;
You have to add the 'fallthrough;' keyword to avoid build failure with
later versions of GCC.
> case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
> case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
> if (BPF_CLASS(code) == BPF_ST) {
> - PPC_LI32(b2p[TMP_REG_1], imm);
> + if ((size == BPF_B) || (size == BPF_H))
> + EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
> + else /* size == BPF_W || size == BPF_DW */
> + PPC_LI32(b2p[TMP_REG_1], imm);
> src_reg = b2p[TMP_REG_1];
> }
> - PPC_BPF_STL(src_reg, dst_reg, off);
> +
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_STL(src_reg, dst_reg, off);
> + break;
> + }
> break;
>
> /*
> @@ -716,25 +714,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> - if (insn_is_zext(&insn[i + 1]))
> - addrs[++i] = ctx->idx * 4;
> - break;
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> - PPC_BPF_LL(dst_reg, src_reg, off);
> + switch (size) {
> + case BPF_B:
> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> + break;
> + case BPF_H:
> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
> + break;
> + case BPF_W:
> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
> + break;
> + case BPF_DW:
> + PPC_BPF_LL(dst_reg, src_reg, off);
> + break;
> + }
> +
> + if ((size != BPF_DW) && insn_is_zext(&insn[i + 1]))
> + addrs[++i] = ctx->idx * 4;
> break;
>
> /*
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 7/8] bpf ppc32: Add BPF_PROBE_MEM support for JIT
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:39 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:39 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> BPF load instruction with BPF_PROBE_MEM mode can cause a fault
> inside kernel. Append exception table for such instructions
> within BPF program.
>
> Unlike other archs which uses extable 'fixup' field to pass dest_reg
> and nip, BPF exception table on PowerPC follows the generic PowerPC
> exception table design, where it populates both fixup and extable
> sections within BPF program. fixup section contains 3 instructions,
> first 2 instructions clear dest_reg (lower & higher 32-bit registers)
> and last instruction jumps to next instruction in the BPF code.
> extable 'insn' field contains relative offset of the instruction and
> 'fixup' field contains relative offset of the fixup entry. Example
> layout of BPF program with extable present:
>
> +------------------+
> | |
> | |
> 0x4020 -->| lwz r28,4(r4) |
> | |
> | |
> 0x40ac -->| lwz r3,0(r24) |
> | lwz r4,4(r24) |
> | |
> | |
> |------------------|
> 0x4278 -->| li r28,0 | \
> | li r27,0 | | fixup entry
> | b 0x4024 | /
> 0x4284 -->| li r4,0 |
> | li r3,0 |
> | b 0x40b4 |
> |------------------|
> 0x4290 -->| insn=0xfffffd90 | \ extable entry
> | fixup=0xffffffe4 | /
> 0x4298 -->| insn=0xfffffe14 |
> | fixup=0xffffffe8 |
> +------------------+
>
> (Addresses shown here are chosen random, not real)
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to add BPF_PROBE_MEM support for PPC32.
>
>
> arch/powerpc/net/bpf_jit.h | 7 +++++
> arch/powerpc/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++
> arch/powerpc/net/bpf_jit_comp32.c | 30 +++++++++++++++++++
> arch/powerpc/net/bpf_jit_comp64.c | 48 ++---------------------------
> 4 files changed, 89 insertions(+), 46 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 6357c71c26eb..6a591ef88006 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -144,7 +144,11 @@ struct codegen_context {
> unsigned int exentry_idx;
> };
>
> +#ifdef CONFIG_PPC32
> +#define BPF_FIXUP_LEN 12 /* Three instructions */
Could use 3 and 2 instead of 12 and 8, see later why.
> +#else
> #define BPF_FIXUP_LEN 8 /* Two instructions */
> +#endif
>
> static inline void bpf_flush_icache(void *start, void *end)
> {
> @@ -174,6 +178,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_realloc_regs(struct codegen_context *ctx);
>
> +int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
> + int insn_idx, int jmp_off, int dst_reg);
> +
> #endif
>
> #endif
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index e92bd79d3bac..a1753b8c78c8 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -271,3 +271,53 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>
> return fp;
> }
> +
> +/*
> + * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> + * this function, as this only applies to BPF_PROBE_MEM, for now.
> + */
> +int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
> + int insn_idx, int jmp_off, int dst_reg)
> +{
Modify patch 5 to get that function already in
arch/powerpc/net/bpf_jit_comp.c, so that only changes/additions to the
function appear here.
And you can have the prototype ready for the final version in patch 5
instead of adding new arguments here and having to change ppc64 call site.
And in fact you can use them already in patch 5, like jmp_off.
> + off_t offset;
> + unsigned long pc;
> + struct exception_table_entry *ex;
> + u32 *fixup;
> +
> + /* Populate extable entries only in the last pass */
> + if (pass != 2)
> + return 0;
> +
> + if (!fp->aux->extable ||
> + WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> + return -EINVAL;
> +
> + pc = (unsigned long)&image[insn_idx];
> +
> + fixup = (void *)fp->aux->extable -
> + (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> + (ctx->exentry_idx * BPF_FIXUP_LEN);
Use 2 or 3 for BPF_FIXUP_LEN and multiply by 4 here.
> +
> + fixup[0] = PPC_RAW_LI(dst_reg, 0);
> +#ifdef CONFIG_PPC32
You should use 'if (IS_ENABLED(CONFIG_PPC32)' instead of a #ifdef here.
> + fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register too */
> + fixup[2] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[2]);
> +#else
> + fixup[1] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[1]);
> +#endif
Use 2 or 3 for BPF_FIXUP_LEN and you can do
if (IS_ENABLED(CONFIG_PPC32)
fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register
too */
fixup[BPF_FIXUP_LEN - 1] = PPC_RAW_BRANCH((long)(pc + jmp_off) -
(long)&fixup[BPF_FIXUP_LEN - 1]);
> +
> + ex = &fp->aux->extable[ctx->exentry_idx];
> +
> + offset = pc - (long)&ex->insn;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->insn = offset;
> +
> + offset = (long)fixup - (long)&ex->fixup;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->fixup = offset;
> +
> + ctx->exentry_idx++;
> + return 0;
> +}
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index 94641b7be387..c6262289dcc4 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -811,12 +811,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_B:
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_H:
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_W:
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> @@ -835,6 +839,32 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
>
> if ((size != BPF_DW) && !fp->aux->verifier_zext)
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> +
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + int insn_idx = ctx->idx - 1;
> + int jmp_off = 4;
> +
> + /*
> + * In case of BPF_DW, two lwz instructions are emitted, one
> + * for higher 32-bit and another for lower 32-bit. So, set
> + * ex->insn to the first of the two and jump over both
> + * instructions in fixup.
> + *
> + * Similarly, with !verifier_zext, two instructions are
> + * emitted for BPF_B/H/W case. So, set ex-insn to the
> + * instruction that could fault and skip over both
> + * instructions.
> + */
> + if ((size == BPF_DW) || !fp->aux->verifier_zext) {
No need of ( ) around 'size == BPF_DW'
> + insn_idx -= 1;
> + jmp_off += 4;
> + }
> +
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, insn_idx,
> + jmp_off, dst_reg);
> + if (ret)
> + return ret;
> + }
> break;
>
> /*
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index eb28dbc67151..10cc9f04843c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -270,51 +270,6 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
> /* out: */
> }
>
> -/*
> - * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> - * this function, as this only applies to BPF_PROBE_MEM, for now.
> - */
> -static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
> - struct codegen_context *ctx, int dst_reg)
> -{
> - off_t offset;
> - unsigned long pc;
> - struct exception_table_entry *ex;
> - u32 *fixup;
> -
> - /* Populate extable entries only in the last pass */
> - if (pass != 2)
> - return 0;
> -
> - if (!fp->aux->extable ||
> - WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> - return -EINVAL;
> -
> - pc = (unsigned long)&image[ctx->idx - 1];
> -
> - fixup = (void *)fp->aux->extable -
> - (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> - (ctx->exentry_idx * BPF_FIXUP_LEN);
> -
> - fixup[0] = PPC_RAW_LI(dst_reg, 0);
> - fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
> -
> - ex = &fp->aux->extable[ctx->exentry_idx];
> -
> - offset = pc - (long)&ex->insn;
> - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> - return -ERANGE;
> - ex->insn = offset;
> -
> - offset = (long)fixup - (long)&ex->fixup;
> - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> - return -ERANGE;
> - ex->fixup = offset;
> -
> - ctx->exentry_idx++;
> - return 0;
> -}
> -
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> u32 *addrs, int pass)
> @@ -811,7 +766,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> addrs[++i] = ctx->idx * 4;
>
> if (BPF_MODE(code) == BPF_PROBE_MEM) {
> - ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, ctx->idx - 1,
> + 4, dst_reg);
Make that ready in patch 5 so that you don't need to change here in patch 7.
> if (ret)
> return ret;
> }
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 7/8] bpf ppc32: Add BPF_PROBE_MEM support for JIT
@ 2021-09-17 16:39 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:39 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> BPF load instruction with BPF_PROBE_MEM mode can cause a fault
> inside kernel. Append exception table for such instructions
> within BPF program.
>
> Unlike other archs which uses extable 'fixup' field to pass dest_reg
> and nip, BPF exception table on PowerPC follows the generic PowerPC
> exception table design, where it populates both fixup and extable
> sections within BPF program. fixup section contains 3 instructions,
> first 2 instructions clear dest_reg (lower & higher 32-bit registers)
> and last instruction jumps to next instruction in the BPF code.
> extable 'insn' field contains relative offset of the instruction and
> 'fixup' field contains relative offset of the fixup entry. Example
> layout of BPF program with extable present:
>
> +------------------+
> | |
> | |
> 0x4020 -->| lwz r28,4(r4) |
> | |
> | |
> 0x40ac -->| lwz r3,0(r24) |
> | lwz r4,4(r24) |
> | |
> | |
> |------------------|
> 0x4278 -->| li r28,0 | \
> | li r27,0 | | fixup entry
> | b 0x4024 | /
> 0x4284 -->| li r4,0 |
> | li r3,0 |
> | b 0x40b4 |
> |------------------|
> 0x4290 -->| insn=0xfffffd90 | \ extable entry
> | fixup=0xffffffe4 | /
> 0x4298 -->| insn=0xfffffe14 |
> | fixup=0xffffffe8 |
> +------------------+
>
> (Addresses shown here are chosen random, not real)
>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to add BPF_PROBE_MEM support for PPC32.
>
>
> arch/powerpc/net/bpf_jit.h | 7 +++++
> arch/powerpc/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++
> arch/powerpc/net/bpf_jit_comp32.c | 30 +++++++++++++++++++
> arch/powerpc/net/bpf_jit_comp64.c | 48 ++---------------------------
> 4 files changed, 89 insertions(+), 46 deletions(-)
>
> diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
> index 6357c71c26eb..6a591ef88006 100644
> --- a/arch/powerpc/net/bpf_jit.h
> +++ b/arch/powerpc/net/bpf_jit.h
> @@ -144,7 +144,11 @@ struct codegen_context {
> unsigned int exentry_idx;
> };
>
> +#ifdef CONFIG_PPC32
> +#define BPF_FIXUP_LEN 12 /* Three instructions */
Could use 3 and 2 instead of 12 and 8, see later why.
> +#else
> #define BPF_FIXUP_LEN 8 /* Two instructions */
> +#endif
>
> static inline void bpf_flush_icache(void *start, void *end)
> {
> @@ -174,6 +178,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
> void bpf_jit_realloc_regs(struct codegen_context *ctx);
>
> +int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
> + int insn_idx, int jmp_off, int dst_reg);
> +
> #endif
>
> #endif
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index e92bd79d3bac..a1753b8c78c8 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -271,3 +271,53 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>
> return fp;
> }
> +
> +/*
> + * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> + * this function, as this only applies to BPF_PROBE_MEM, for now.
> + */
> +int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
> + int insn_idx, int jmp_off, int dst_reg)
> +{
Modify patch 5 to get that function already in
arch/powerpc/net/bpf_jit_comp.c, so that only changes/additions to the
function appear here.
And you can have the prototype ready for the final version in patch 5
instead of adding new arguments here and having to change ppc64 call site.
And in fact you can use them already in patch 5, like jmp_off.
> + off_t offset;
> + unsigned long pc;
> + struct exception_table_entry *ex;
> + u32 *fixup;
> +
> + /* Populate extable entries only in the last pass */
> + if (pass != 2)
> + return 0;
> +
> + if (!fp->aux->extable ||
> + WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> + return -EINVAL;
> +
> + pc = (unsigned long)&image[insn_idx];
> +
> + fixup = (void *)fp->aux->extable -
> + (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> + (ctx->exentry_idx * BPF_FIXUP_LEN);
Use 2 or 3 for BPF_FIXUP_LEN and multiply by 4 here.
> +
> + fixup[0] = PPC_RAW_LI(dst_reg, 0);
> +#ifdef CONFIG_PPC32
You should use 'if (IS_ENABLED(CONFIG_PPC32)' instead of a #ifdef here.
> + fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register too */
> + fixup[2] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[2]);
> +#else
> + fixup[1] = PPC_RAW_BRANCH((long)(pc + jmp_off) - (long)&fixup[1]);
> +#endif
Use 2 or 3 for BPF_FIXUP_LEN and you can do
if (IS_ENABLED(CONFIG_PPC32)
fixup[1] = PPC_RAW_LI(dst_reg - 1, 0); /* clear higher 32-bit register
too */
fixup[BPF_FIXUP_LEN - 1] = PPC_RAW_BRANCH((long)(pc + jmp_off) -
(long)&fixup[BPF_FIXUP_LEN - 1]);
> +
> + ex = &fp->aux->extable[ctx->exentry_idx];
> +
> + offset = pc - (long)&ex->insn;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->insn = offset;
> +
> + offset = (long)fixup - (long)&ex->fixup;
> + if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> + return -ERANGE;
> + ex->fixup = offset;
> +
> + ctx->exentry_idx++;
> + return 0;
> +}
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index 94641b7be387..c6262289dcc4 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -811,12 +811,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> */
> /* dst = *(u8 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_B:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_B:
> /* dst = *(u16 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_H:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_H:
> /* dst = *(u32 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_W:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_W:
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> + case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
> @@ -835,6 +839,32 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
>
> if ((size != BPF_DW) && !fp->aux->verifier_zext)
> EMIT(PPC_RAW_LI(dst_reg_h, 0));
> +
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + int insn_idx = ctx->idx - 1;
> + int jmp_off = 4;
> +
> + /*
> + * In case of BPF_DW, two lwz instructions are emitted, one
> + * for higher 32-bit and another for lower 32-bit. So, set
> + * ex->insn to the first of the two and jump over both
> + * instructions in fixup.
> + *
> + * Similarly, with !verifier_zext, two instructions are
> + * emitted for BPF_B/H/W case. So, set ex-insn to the
> + * instruction that could fault and skip over both
> + * instructions.
> + */
> + if ((size == BPF_DW) || !fp->aux->verifier_zext) {
No need of ( ) around 'size == BPF_DW'
> + insn_idx -= 1;
> + jmp_off += 4;
> + }
> +
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, insn_idx,
> + jmp_off, dst_reg);
> + if (ret)
> + return ret;
> + }
> break;
>
> /*
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index eb28dbc67151..10cc9f04843c 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -270,51 +270,6 @@ static void bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32
> /* out: */
> }
>
> -/*
> - * The caller should check for (BPF_MODE(code) == BPF_PROBE_MEM) before calling
> - * this function, as this only applies to BPF_PROBE_MEM, for now.
> - */
> -static int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass,
> - struct codegen_context *ctx, int dst_reg)
> -{
> - off_t offset;
> - unsigned long pc;
> - struct exception_table_entry *ex;
> - u32 *fixup;
> -
> - /* Populate extable entries only in the last pass */
> - if (pass != 2)
> - return 0;
> -
> - if (!fp->aux->extable ||
> - WARN_ON_ONCE(ctx->exentry_idx >= fp->aux->num_exentries))
> - return -EINVAL;
> -
> - pc = (unsigned long)&image[ctx->idx - 1];
> -
> - fixup = (void *)fp->aux->extable -
> - (fp->aux->num_exentries * BPF_FIXUP_LEN) +
> - (ctx->exentry_idx * BPF_FIXUP_LEN);
> -
> - fixup[0] = PPC_RAW_LI(dst_reg, 0);
> - fixup[1] = PPC_RAW_BRANCH((long)(pc + 4) - (long)&fixup[1]);
> -
> - ex = &fp->aux->extable[ctx->exentry_idx];
> -
> - offset = pc - (long)&ex->insn;
> - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> - return -ERANGE;
> - ex->insn = offset;
> -
> - offset = (long)fixup - (long)&ex->fixup;
> - if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
> - return -ERANGE;
> - ex->fixup = offset;
> -
> - ctx->exentry_idx++;
> - return 0;
> -}
> -
> /* Assemble the body code between the prologue & epilogue */
> int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
> u32 *addrs, int pass)
> @@ -811,7 +766,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> addrs[++i] = ctx->idx * 4;
>
> if (BPF_MODE(code) == BPF_PROBE_MEM) {
> - ret = bpf_add_extable_entry(fp, image, pass, ctx, dst_reg);
> + ret = bpf_add_extable_entry(fp, image, pass, ctx, ctx->idx - 1,
> + 4, dst_reg);
Make that ready in patch 5 so that you don't need to change here in patch 7.
> if (ret)
> return ret;
> }
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 6/8] bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:50 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:50 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev, Ravi Bangoria
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> On PPC64 with KUAP enabled, any kernel code which wants to
> access userspace needs to be surrounded by disable-enable KUAP.
> But that is not happening for BPF_PROBE_MEM load instruction.
> So, when BPF program tries to access invalid userspace address,
> page-fault handler considers it as bad KUAP fault:
>
> Kernel attempted to read user page (d0000000) - exploit attempt? (uid: 0)
>
> Considering the fact that PTR_TO_BTF_ID (which uses BPF_PROBE_MEM
> mode) could either be a valid kernel pointer or NULL but should
> never be a pointer to userspace address, execute BPF_PROBE_MEM load
> only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
You should do like copy_from_kernel_nofault_allowed() and use the same
criterias as is_kernel_addr() instead of using TASK_SIZE_MAX.
>
> This will catch NULL, valid or invalid userspace pointers. Only bad
> kernel pointer will be handled by BPF exception table.
>
> [Alexei suggested for x86]
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * Refactored the code based on Christophe's comments.
>
>
> arch/powerpc/net/bpf_jit_comp64.c | 23 +++++++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2fc10995f243..eb28dbc67151 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -769,6 +769,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> + /*
> + * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
> + * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
> + * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
> + */
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + unsigned int adjusted_idx;
> +
> + /*
> + * Check if 'off' is word aligned because PPC_BPF_LL()
> + * (BPF_DW case) generates two instructions if 'off' is not
> + * word-aligned and one instruction otherwise.
> + */
> + adjusted_idx = ((BPF_SIZE(code) == BPF_DW) && (off & 3)) ? 1 : 0;
No need of ( ) around 'BPF_SIZE(code) == BPF_DW'
> +
> + EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], src_reg, off));
> + PPC_LI64(b2p[TMP_REG_2], TASK_SIZE_MAX);
> + EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
> + PPC_BCC(COND_GT, (ctx->idx + 4) * 4);
> + EMIT(PPC_RAW_LI(dst_reg, 0));
> + PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
I think it would be more explicit if you drop adjusted_idx and do :
if (BPF_SIZE(code) == BPF_DW) && (off & 3)
PPC_JMP((ctx->idx + 3) * 4);
else
PPC_JMP((ctx->idx + 2) * 4);
> + }
> +
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 6/8] bpf ppc64: Add addr > TASK_SIZE_MAX explicit check
@ 2021-09-17 16:50 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:50 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: Ravi Bangoria, songliubraving, netdev, john.fastabend, andrii,
kpsingh, paulus, yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> From: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
>
> On PPC64 with KUAP enabled, any kernel code which wants to
> access userspace needs to be surrounded by disable-enable KUAP.
> But that is not happening for BPF_PROBE_MEM load instruction.
> So, when BPF program tries to access invalid userspace address,
> page-fault handler considers it as bad KUAP fault:
>
> Kernel attempted to read user page (d0000000) - exploit attempt? (uid: 0)
>
> Considering the fact that PTR_TO_BTF_ID (which uses BPF_PROBE_MEM
> mode) could either be a valid kernel pointer or NULL but should
> never be a pointer to userspace address, execute BPF_PROBE_MEM load
> only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
You should do like copy_from_kernel_nofault_allowed() and use the same
criterias as is_kernel_addr() instead of using TASK_SIZE_MAX.
>
> This will catch NULL, valid or invalid userspace pointers. Only bad
> kernel pointer will be handled by BPF exception table.
>
> [Alexei suggested for x86]
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * Refactored the code based on Christophe's comments.
>
>
> arch/powerpc/net/bpf_jit_comp64.c | 23 +++++++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
> index 2fc10995f243..eb28dbc67151 100644
> --- a/arch/powerpc/net/bpf_jit_comp64.c
> +++ b/arch/powerpc/net/bpf_jit_comp64.c
> @@ -769,6 +769,29 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> + /*
> + * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
> + * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
> + * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
> + */
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + unsigned int adjusted_idx;
> +
> + /*
> + * Check if 'off' is word aligned because PPC_BPF_LL()
> + * (BPF_DW case) generates two instructions if 'off' is not
> + * word-aligned and one instruction otherwise.
> + */
> + adjusted_idx = ((BPF_SIZE(code) == BPF_DW) && (off & 3)) ? 1 : 0;
No need of ( ) around 'BPF_SIZE(code) == BPF_DW'
> +
> + EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], src_reg, off));
> + PPC_LI64(b2p[TMP_REG_2], TASK_SIZE_MAX);
> + EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
> + PPC_BCC(COND_GT, (ctx->idx + 4) * 4);
> + EMIT(PPC_RAW_LI(dst_reg, 0));
> + PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
I think it would be more explicit if you drop adjusted_idx and do :
if (BPF_SIZE(code) == BPF_DW) && (off & 3)
PPC_JMP((ctx->idx + 3) * 4);
else
PPC_JMP((ctx->idx + 2) * 4);
> + }
> +
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 8/8] bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
2021-09-17 15:30 ` Hari Bathini
@ 2021-09-17 16:57 ` Christophe Leroy
-1 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:57 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> With KUAP enabled, any kernel code which wants to access userspace
> needs to be surrounded by disable-enable KUAP. But that is not
> happening for BPF_PROBE_MEM load instruction. Though PPC32 does not
> support read protection, considering the fact that PTR_TO_BTF_ID
> (which uses BPF_PROBE_MEM mode) could either be a valid kernel pointer
> or NULL but should never be a pointer to userspace address, execute
> BPF_PROBE_MEM load only if addr > TASK_SIZE_MAX, otherwise set
> dst_reg=0 and move on.
Same comment as patch 6.
>
> This will catch NULL, valid or invalid userspace pointers. Only bad
> kernel pointer will be handled by BPF exception table.
>
> [Alexei suggested for x86]
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to handle bad userspace pointers on PPC32.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 39 +++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index c6262289dcc4..faa8047fcf4a 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -821,6 +821,45 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> + /*
> + * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
> + * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
> + * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
> + */
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + bool extra_insn_needed = false;
> + unsigned int adjusted_idx;
> +
> + /*
> + * For BPF_DW case, "li reg_h,0" would be needed when
> + * !fp->aux->verifier_zext. Adjust conditional branch'ing
> + * address accordingly.
> + */
> + if ((size == BPF_DW) && !fp->aux->verifier_zext)
> + extra_insn_needed = true;
Don't make it too complicated. That's a fallback that should never
happen, no need to optimise. You can put that instruction all the time
(or put a NOP) and keep the jumps always the same.
> +
> + /*
> + * Need to jump two instructions instead of one for BPF_DW case
> + * as there are two load instructions for dst_reg_h & dst_reg
> + * respectively.
> + */
> + adjusted_idx = (size == BPF_DW) ? 1 : 0;
Same comment as patch 6, drop adjusted_idx and do an if/else directly
for the PPC_JMP.
> +
> + EMIT(PPC_RAW_ADDI(b2p[TMP_REG], src_reg, off));
> + PPC_LI32(_R0, TASK_SIZE_MAX);
> + EMIT(PPC_RAW_CMPLW(b2p[TMP_REG], _R0));
> + PPC_BCC(COND_GT, (ctx->idx + 4 + (extra_insn_needed ? 1 : 0)) * 4);
> + EMIT(PPC_RAW_LI(dst_reg, 0));
> + /*
> + * Note that "li reg_h,0" is emitted for BPF_B/H/W case,
> + * if necessary. So, jump there insted of emitting an
> + * additional "li reg_h,0" instruction.
> + */
> + if (extra_insn_needed)
> + EMIT(PPC_RAW_LI(dst_reg_h, 0));
> + PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
> + }
> +
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 8/8] bpf ppc32: Add addr > TASK_SIZE_MAX explicit check
@ 2021-09-17 16:57 ` Christophe Leroy
0 siblings, 0 replies; 36+ messages in thread
From: Christophe Leroy @ 2021-09-17 16:57 UTC (permalink / raw)
To: Hari Bathini, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Le 17/09/2021 à 17:30, Hari Bathini a écrit :
> With KUAP enabled, any kernel code which wants to access userspace
> needs to be surrounded by disable-enable KUAP. But that is not
> happening for BPF_PROBE_MEM load instruction. Though PPC32 does not
> support read protection, considering the fact that PTR_TO_BTF_ID
> (which uses BPF_PROBE_MEM mode) could either be a valid kernel pointer
> or NULL but should never be a pointer to userspace address, execute
> BPF_PROBE_MEM load only if addr > TASK_SIZE_MAX, otherwise set
> dst_reg=0 and move on.
Same comment as patch 6.
>
> This will catch NULL, valid or invalid userspace pointers. Only bad
> kernel pointer will be handled by BPF exception table.
>
> [Alexei suggested for x86]
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
> ---
>
> Changes in v2:
> * New patch to handle bad userspace pointers on PPC32.
>
>
> arch/powerpc/net/bpf_jit_comp32.c | 39 +++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
> index c6262289dcc4..faa8047fcf4a 100644
> --- a/arch/powerpc/net/bpf_jit_comp32.c
> +++ b/arch/powerpc/net/bpf_jit_comp32.c
> @@ -821,6 +821,45 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
> /* dst = *(u64 *)(ul) (src + off) */
> case BPF_LDX | BPF_MEM | BPF_DW:
> case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
> + /*
> + * As PTR_TO_BTF_ID that uses BPF_PROBE_MEM mode could either be a valid
> + * kernel pointer or NULL but not a userspace address, execute BPF_PROBE_MEM
> + * load only if addr > TASK_SIZE_MAX, otherwise set dst_reg=0 and move on.
> + */
> + if (BPF_MODE(code) == BPF_PROBE_MEM) {
> + bool extra_insn_needed = false;
> + unsigned int adjusted_idx;
> +
> + /*
> + * For BPF_DW case, "li reg_h,0" would be needed when
> + * !fp->aux->verifier_zext. Adjust conditional branch'ing
> + * address accordingly.
> + */
> + if ((size == BPF_DW) && !fp->aux->verifier_zext)
> + extra_insn_needed = true;
Don't make it too complicated. That's a fallback that should never
happen, no need to optimise. You can put that instruction all the time
(or put a NOP) and keep the jumps always the same.
> +
> + /*
> + * Need to jump two instructions instead of one for BPF_DW case
> + * as there are two load instructions for dst_reg_h & dst_reg
> + * respectively.
> + */
> + adjusted_idx = (size == BPF_DW) ? 1 : 0;
Same comment as patch 6, drop adjusted_idx and do an if/else directly
for the PPC_JMP.
> +
> + EMIT(PPC_RAW_ADDI(b2p[TMP_REG], src_reg, off));
> + PPC_LI32(_R0, TASK_SIZE_MAX);
> + EMIT(PPC_RAW_CMPLW(b2p[TMP_REG], _R0));
> + PPC_BCC(COND_GT, (ctx->idx + 4 + (extra_insn_needed ? 1 : 0)) * 4);
> + EMIT(PPC_RAW_LI(dst_reg, 0));
> + /*
> + * Note that "li reg_h,0" is emitted for BPF_B/H/W case,
> + * if necessary. So, jump there insted of emitting an
> + * additional "li reg_h,0" instruction.
> + */
> + if (extra_insn_needed)
> + EMIT(PPC_RAW_LI(dst_reg_h, 0));
> + PPC_JMP((ctx->idx + 2 + adjusted_idx) * 4);
> + }
> +
> switch (size) {
> case BPF_B:
> EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
2021-09-17 16:10 ` Christophe Leroy
@ 2021-09-20 13:28 ` Hari Bathini
-1 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-20 13:28 UTC (permalink / raw)
To: Christophe Leroy, naveen.n.rao, mpe, ast, daniel
Cc: paulus, andrii, kafai, songliubraving, yhs, john.fastabend,
kpsingh, netdev, bpf, linuxppc-dev
Hi Christophe,
Thanks for reviewing the series.
On 17/09/21 9:40 pm, Christophe Leroy wrote:
>
>
> Le 17/09/2021 à 17:30, Hari Bathini a écrit :
>> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
>
> Could you describe a bit more what you are refactoring exactly ?
I am trying to do more than BPF_PROBE_MEM needs. Will keep the changes
minimal (BPF_PROBE_MEM specific) and update the changelog..
>
>
>>
>> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
>> ---
>>
>> Changes in v2:
>> * New patch to refactor a bit of JITing code.
>>
>>
>> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
>> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
>> 2 files changed, 68 insertions(+), 58 deletions(-)
>>
>> diff --git a/arch/powerpc/net/bpf_jit_comp32.c
>> b/arch/powerpc/net/bpf_jit_comp32.c
>> index b60b59426a24..c8ae14c316e3 100644
>> --- a/arch/powerpc/net/bpf_jit_comp32.c
>> +++ b/arch/powerpc/net/bpf_jit_comp32.c
>> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32
>> *image, struct codegen_context *
>> u32 exit_addr = addrs[flen];
>> for (i = 0; i < flen; i++) {
>> - u32 code = insn[i].code;
>> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
>> - u32 dst_reg_h = dst_reg - 1;
>> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
>> - u32 src_reg_h = src_reg - 1;
>> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
>> + u32 true_cond, code = insn[i].code;
>> + u32 dst_reg_h = dst_reg - 1;
>> + u32 src_reg_h = src_reg - 1;
>
> All changes above seems unneeded and not linked to the current patch.
> Please leave cosmetic changes outside and focus on necessary changes.
>
>> + u32 size = BPF_SIZE(code);
>> s16 off = insn[i].off;
>> s32 imm = insn[i].imm;
>> bool func_addr_fixed;
>> u64 func_addr;
>> - u32 true_cond;
>> /*
>> * addrs[] maps a BPF bytecode address into a real offset from
>> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32
>> *image, struct codegen_context *
>> /*
>> * BPF_LDX
>> */
>> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
>> - break;
>> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
>> - break;
>> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> + /* dst = *(u8 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_B:
>> + /* dst = *(u16 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_H:
>> + /* dst = *(u32 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_W:
>> + /* dst = *(u64 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_DW:
> Why changing the location of the comments ? I found it more readable
> before.
Sure. I will revert that change.
>> + switch (size) {
>> + case BPF_B:
>> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_H:
>> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_W:
>> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_DW:
>> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
>> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
>> + break;
>> + }
>
> BPF_B, BPF_H, ... are not part of an enum. Are you sure GCC is happy to
> have no default ?
I used gcc 10.3 for ppc32 & gcc 8.3 for ppc64. No warnings.
Though, no harm adding the below, I guess..
default:
break;
Thanks
Hari
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code
@ 2021-09-20 13:28 ` Hari Bathini
0 siblings, 0 replies; 36+ messages in thread
From: Hari Bathini @ 2021-09-20 13:28 UTC (permalink / raw)
To: Christophe Leroy, naveen.n.rao, mpe, ast, daniel
Cc: songliubraving, netdev, john.fastabend, andrii, kpsingh, paulus,
yhs, bpf, linuxppc-dev, kafai
Hi Christophe,
Thanks for reviewing the series.
On 17/09/21 9:40 pm, Christophe Leroy wrote:
>
>
> Le 17/09/2021 à 17:30, Hari Bathini a écrit :
>> Refactor powerpc JITing. This simplifies adding BPF_PROBE_MEM support.
>
> Could you describe a bit more what you are refactoring exactly ?
I am trying to do more than BPF_PROBE_MEM needs. Will keep the changes
minimal (BPF_PROBE_MEM specific) and update the changelog..
>
>
>>
>> Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
>> ---
>>
>> Changes in v2:
>> * New patch to refactor a bit of JITing code.
>>
>>
>> arch/powerpc/net/bpf_jit_comp32.c | 50 +++++++++++---------
>> arch/powerpc/net/bpf_jit_comp64.c | 76 ++++++++++++++++---------------
>> 2 files changed, 68 insertions(+), 58 deletions(-)
>>
>> diff --git a/arch/powerpc/net/bpf_jit_comp32.c
>> b/arch/powerpc/net/bpf_jit_comp32.c
>> index b60b59426a24..c8ae14c316e3 100644
>> --- a/arch/powerpc/net/bpf_jit_comp32.c
>> +++ b/arch/powerpc/net/bpf_jit_comp32.c
>> @@ -276,17 +276,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32
>> *image, struct codegen_context *
>> u32 exit_addr = addrs[flen];
>> for (i = 0; i < flen; i++) {
>> - u32 code = insn[i].code;
>> u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
>> - u32 dst_reg_h = dst_reg - 1;
>> u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
>> - u32 src_reg_h = src_reg - 1;
>> u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
>> + u32 true_cond, code = insn[i].code;
>> + u32 dst_reg_h = dst_reg - 1;
>> + u32 src_reg_h = src_reg - 1;
>
> All changes above seems unneeded and not linked to the current patch.
> Please leave cosmetic changes outside and focus on necessary changes.
>
>> + u32 size = BPF_SIZE(code);
>> s16 off = insn[i].off;
>> s32 imm = insn[i].imm;
>> bool func_addr_fixed;
>> u64 func_addr;
>> - u32 true_cond;
>> /*
>> * addrs[] maps a BPF bytecode address into a real offset from
>> @@ -809,25 +809,33 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32
>> *image, struct codegen_context *
>> /*
>> * BPF_LDX
>> */
>> - case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
>> - break;
>> - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> - EMIT(PPC_RAW_LI(dst_reg_h, 0));
>> - break;
>> - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src +
>> off) */
>> - EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
>> - if (!fp->aux->verifier_zext)
>> + /* dst = *(u8 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_B:
>> + /* dst = *(u16 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_H:
>> + /* dst = *(u32 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_W:
>> + /* dst = *(u64 *)(ul) (src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_DW:
> Why changing the location of the comments ? I found it more readable
> before.
Sure. I will revert that change.
>> + switch (size) {
>> + case BPF_B:
>> + EMIT(PPC_RAW_LBZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_H:
>> + EMIT(PPC_RAW_LHZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_W:
>> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
>> + break;
>> + case BPF_DW:
>> + EMIT(PPC_RAW_LWZ(dst_reg_h, src_reg, off));
>> + EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off + 4));
>> + break;
>> + }
>
> BPF_B, BPF_H, ... are not part of an enum. Are you sure GCC is happy to
> have no default ?
I used gcc 10.3 for ppc32 & gcc 8.3 for ppc64. No warnings.
Though, no harm adding the below, I guess..
default:
break;
Thanks
Hari
^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2021-09-20 13:29 UTC | newest]
Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-17 15:30 [PATCH v2 0/8] bpf powerpc: Add BPF_PROBE_MEM support in powerpc JIT compiler Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 15:30 ` [PATCH v2 1/8] bpf powerpc: Remove unused SEEN_STACK Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:02 ` Christophe Leroy
2021-09-17 16:02 ` Christophe Leroy
2021-09-17 15:30 ` [PATCH v2 2/8] bpf powerpc: Remove extra_pass from bpf_jit_build_body() Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 15:30 ` [PATCH v2 3/8] bpf powerpc: refactor JIT compiler code Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:10 ` Christophe Leroy
2021-09-17 16:10 ` Christophe Leroy
2021-09-20 13:28 ` Hari Bathini
2021-09-20 13:28 ` Hari Bathini
2021-09-17 16:22 ` Christophe Leroy
2021-09-17 16:22 ` Christophe Leroy
2021-09-17 15:30 ` [PATCH v2 4/8] powerpc/ppc-opcode: introduce PPC_RAW_BRANCH() macro Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:14 ` LEROY Christophe
2021-09-17 16:14 ` LEROY Christophe
2021-09-17 15:30 ` [PATCH v2 5/8] bpf ppc64: Add BPF_PROBE_MEM support for JIT Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:20 ` Christophe Leroy
2021-09-17 16:20 ` Christophe Leroy
2021-09-17 15:30 ` [PATCH v2 6/8] bpf ppc64: Add addr > TASK_SIZE_MAX explicit check Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:50 ` Christophe Leroy
2021-09-17 16:50 ` Christophe Leroy
2021-09-17 15:30 ` [PATCH v2 7/8] bpf ppc32: Add BPF_PROBE_MEM support for JIT Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:39 ` Christophe Leroy
2021-09-17 16:39 ` Christophe Leroy
2021-09-17 15:30 ` [PATCH v2 8/8] bpf ppc32: Add addr > TASK_SIZE_MAX explicit check Hari Bathini
2021-09-17 15:30 ` Hari Bathini
2021-09-17 16:57 ` Christophe Leroy
2021-09-17 16:57 ` Christophe Leroy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.