* [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program. @ 2022-04-12 16:55 Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links Kui-Feng Lee ` (4 more replies) 0 siblings, 5 replies; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee Allow users to attach a 64-bits cookie to a bpf_link of fentry, fexit, or fmod_ret. This patchset includes several major changes. - Define struct bpf_tramp_links to replace bpf_tramp_prog. struct bpf_tramp_links collects bpf_links of a trampoline - Generate a trampoline to call bpf_progs of given bpf_links. - Trampolines always set/reset bpf_run_ctx before/after calling/leaving a tracing program. - Attach a cookie to a bpf_link of fentry/fexit/fmod_ret. The value will be available when running the associated bpf_prog. The major differences from v4: - Remove condition checking in __bpf_prog_{enter,exit) The major differences from v3: - rebase v3 to latest tip. The major differences from v2: - Move the allocations of run_ctx (struct bpf_tramp_run_ctx) out of invoke_bpf_prog(). - Move hlist_node out of bpf_link and introduce struct bpf_tramp_link to own hlist_node. - Store cookies at struct bpf_tracing_link. - Use SIB byte to reduce the number of instructions to set cookie values. (Use RSP directly) v1: https://lore.kernel.org/all/20220126214809.3868787-1-kuifeng@fb.com/ v2: https://lore.kernel.org/bpf/20220316004231.1103318-1-kuifeng@fb.com/ v3: https://lore.kernel.org/bpf/20220407192552.2343076-1-kuifeng@fb.com/ v4: https://lore.kernel.org/bpf/20220411173429.4139609-1-kuifeng@fb.com/ Kui-Feng Lee (5): bpf, x86: Generate trampolines from bpf_tramp_links bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack bpf, x86: Attach a cookie to fentry/fexit/fmod_ret. lib/bpf: Assign cookies to links in libbpf. selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret. arch/x86/net/bpf_jit_comp.c | 98 +++++++++++++++---- include/linux/bpf.h | 56 ++++++++--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 9 ++ kernel/bpf/bpf_struct_ops.c | 69 ++++++++----- kernel/bpf/syscall.c | 57 +++++++---- kernel/bpf/trampoline.c | 93 +++++++++++------- kernel/trace/bpf_trace.c | 17 ++++ net/bpf/bpf_dummy_struct_ops.c | 35 ++++++- tools/bpf/bpftool/link.c | 1 + tools/include/uapi/linux/bpf.h | 8 ++ tools/lib/bpf/bpf.c | 7 ++ tools/lib/bpf/bpf.h | 3 + tools/lib/bpf/libbpf.c | 33 +++++++ tools/lib/bpf/libbpf.h | 12 +++ tools/lib/bpf/libbpf.map | 1 + .../selftests/bpf/prog_tests/bpf_cookie.c | 52 ++++++++++ .../selftests/bpf/progs/test_bpf_cookie.c | 24 +++++ 18 files changed, 459 insertions(+), 117 deletions(-) -- 2.30.2 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee @ 2022-04-12 16:55 ` Kui-Feng Lee 2022-04-13 2:43 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack Kui-Feng Lee ` (3 subsequent siblings) 4 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee Replace struct bpf_tramp_progs with struct bpf_tramp_links to collect struct bpf_tramp_link(s) for a trampoline. struct bpf_tramp_link extends bpf_link to act as a linked list node. arch_prepare_bpf_trampoline() accepts a struct bpf_tramp_links to collects all bpf_tramp_link(s) that a trampoline should call. Change BPF trampoline and bpf_struct_ops to pass bpf_tramp_links instead of bpf_tramp_progs. Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> --- arch/x86/net/bpf_jit_comp.c | 36 +++++++++-------- include/linux/bpf.h | 38 ++++++++++++------ include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 69 ++++++++++++++++++++++---------- kernel/bpf/syscall.c | 23 ++++------- kernel/bpf/trampoline.c | 73 +++++++++++++++++++--------------- net/bpf/bpf_dummy_struct_ops.c | 35 +++++++++++++--- tools/bpf/bpftool/link.c | 1 + tools/include/uapi/linux/bpf.h | 1 + 10 files changed, 175 insertions(+), 103 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 8fe35ed11fd6..4dcc0b1ac770 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1761,10 +1761,12 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_args, } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, - struct bpf_prog *p, int stack_size, bool save_ret) + struct bpf_tramp_link *l, int stack_size, + bool save_ret) { u8 *prog = *pprog; u8 *jmp_insn; + struct bpf_prog *p = l->link.prog; /* arg1: mov rdi, progs[i] */ emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); @@ -1849,14 +1851,14 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) } static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, - struct bpf_tramp_progs *tp, int stack_size, + struct bpf_tramp_links *tl, int stack_size, bool save_ret) { int i; u8 *prog = *pprog; - for (i = 0; i < tp->nr_progs; i++) { - if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, + for (i = 0; i < tl->nr_links; i++) { + if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, save_ret)) return -EINVAL; } @@ -1865,7 +1867,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, } static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, - struct bpf_tramp_progs *tp, int stack_size, + struct bpf_tramp_links *tl, int stack_size, u8 **branches) { u8 *prog = *pprog; @@ -1876,8 +1878,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, */ emit_mov_imm32(&prog, false, BPF_REG_0, 0); emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); - for (i = 0; i < tp->nr_progs; i++) { - if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, true)) + for (i = 0; i < tl->nr_links; i++) { + if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, true)) return -EINVAL; /* mod_ret prog stored return value into [rbp - 8]. Emit: @@ -1979,14 +1981,14 @@ static bool is_valid_bpf_tramp_flags(unsigned int flags) */ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, const struct btf_func_model *m, u32 flags, - struct bpf_tramp_progs *tprogs, + struct bpf_tramp_links *tlinks, void *orig_call) { int ret, i, nr_args = m->nr_args; int regs_off, ip_off, args_off, stack_size = nr_args * 8; - struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY]; - struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT]; - struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN]; + struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; + struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; + struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; u8 **branches = NULL; u8 *prog; bool save_ret; @@ -2077,13 +2079,13 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } } - if (fentry->nr_progs) + if (fentry->nr_links) if (invoke_bpf(m, &prog, fentry, regs_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) return -EINVAL; - if (fmod_ret->nr_progs) { - branches = kcalloc(fmod_ret->nr_progs, sizeof(u8 *), + if (fmod_ret->nr_links) { + branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *), GFP_KERNEL); if (!branches) return -ENOMEM; @@ -2110,7 +2112,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i prog += X86_PATCH_SIZE; } - if (fmod_ret->nr_progs) { + if (fmod_ret->nr_links) { /* From Intel 64 and IA-32 Architectures Optimization * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler * Coding Rule 11: All branch targets should be 16-byte @@ -2120,12 +2122,12 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i /* Update the branches saved in invoke_bpf_mod_ret with the * aligned address of do_fexit. */ - for (i = 0; i < fmod_ret->nr_progs; i++) + for (i = 0; i < fmod_ret->nr_links; i++) emit_cond_near_jump(&branches[i], prog, branches[i], X86_JNE); } - if (fexit->nr_progs) + if (fexit->nr_links) if (invoke_bpf(m, &prog, fexit, regs_off, false)) { ret = -EINVAL; goto cleanup; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index bdb5298735ce..2f7909eb51ec 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -674,11 +674,11 @@ struct btf_func_model { /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. Pick a number to fit into BPF_IMAGE_SIZE / 2 */ -#define BPF_MAX_TRAMP_PROGS 38 +#define BPF_MAX_TRAMP_LINKS 38 -struct bpf_tramp_progs { - struct bpf_prog *progs[BPF_MAX_TRAMP_PROGS]; - int nr_progs; +struct bpf_tramp_links { + struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS]; + int nr_links; }; /* Different use cases for BPF trampoline: @@ -704,7 +704,7 @@ struct bpf_tramp_progs { struct bpf_tramp_image; int arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void *image, void *image_end, const struct btf_func_model *m, u32 flags, - struct bpf_tramp_progs *tprogs, + struct bpf_tramp_links *tlinks, void *orig_call); /* these two functions are called from generated trampoline */ u64 notrace __bpf_prog_enter(struct bpf_prog *prog); @@ -803,9 +803,12 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( { return bpf_func(ctx, insnsi); } + +struct bpf_link; + #ifdef CONFIG_BPF_JIT -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); @@ -856,12 +859,12 @@ int bpf_jit_charge_modmem(u32 size); void bpf_jit_uncharge_modmem(u32 size); bool bpf_prog_has_trampoline(const struct bpf_prog *prog); #else -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) { return -ENOTSUPP; } -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) { return -ENOTSUPP; @@ -960,7 +963,6 @@ struct bpf_prog_aux { bool tail_call_reachable; bool xdp_has_frags; bool use_bpf_prog_pack; - struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ @@ -1047,6 +1049,18 @@ struct bpf_link_ops { struct bpf_link_info *info); }; +struct bpf_tramp_link { + struct bpf_link link; + struct hlist_node tramp_hlist; +}; + +struct bpf_tracing_link { + struct bpf_tramp_link link; + enum bpf_attach_type attach_type; + struct bpf_trampoline *trampoline; + struct bpf_prog *tgt_prog; +}; + struct bpf_link_primer { struct bpf_link *link; struct file *file; @@ -1084,8 +1098,8 @@ bool bpf_struct_ops_get(const void *kdata); void bpf_struct_ops_put(const void *kdata); int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, void *value); -int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_progs *tprogs, - struct bpf_prog *prog, +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, + struct bpf_tramp_link *link, const struct btf_func_model *model, void *image, void *image_end); static inline bool bpf_try_module_get(const void *data, struct module *owner) diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 3e24ad0c4b3c..2b9112b80171 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -141,3 +141,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_XDP, xdp) BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) #endif BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) +BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index d14b10b85e51..a4f557338af7 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1013,6 +1013,7 @@ enum bpf_link_type { BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, BPF_LINK_TYPE_KPROBE_MULTI = 8, + BPF_LINK_TYPE_STRUCT_OPS = 9, MAX_BPF_LINK_TYPE, }; diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 21069dbe9138..6aaaec69ec54 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -32,15 +32,15 @@ struct bpf_struct_ops_map { const struct bpf_struct_ops *st_ops; /* protect map_update */ struct mutex lock; - /* progs has all the bpf_prog that is populated + /* link has all the bpf_links that is populated * to the func ptr of the kernel's struct * (in kvalue.data). */ - struct bpf_prog **progs; + struct bpf_link **links; /* image is a page that has all the trampolines * that stores the func args before calling the bpf_prog. * A PAGE_SIZE "image" is enough to store all trampoline for - * "progs[]". + * "links[]". */ void *image; /* uvalue->data stores the kernel struct @@ -282,9 +282,9 @@ static void bpf_struct_ops_map_put_progs(struct bpf_struct_ops_map *st_map) u32 i; for (i = 0; i < btf_type_vlen(t); i++) { - if (st_map->progs[i]) { - bpf_prog_put(st_map->progs[i]); - st_map->progs[i] = NULL; + if (st_map->links[i]) { + bpf_link_put(st_map->links[i]); + st_map->links[i] = NULL; } } } @@ -315,18 +315,32 @@ static int check_zero_holes(const struct btf_type *t, void *data) return 0; } -int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_progs *tprogs, - struct bpf_prog *prog, +static void bpf_struct_ops_link_release(struct bpf_link *link) +{ +} + +static void bpf_struct_ops_link_dealloc(struct bpf_link *link) +{ + kfree(link); +} + +static const struct bpf_link_ops bpf_struct_ops_link_lops = { + .release = bpf_struct_ops_link_release, + .dealloc = bpf_struct_ops_link_dealloc, +}; + +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, + struct bpf_tramp_link *link, const struct btf_func_model *model, void *image, void *image_end) { u32 flags; - tprogs[BPF_TRAMP_FENTRY].progs[0] = prog; - tprogs[BPF_TRAMP_FENTRY].nr_progs = 1; + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; flags = model->ret_size > 0 ? BPF_TRAMP_F_RET_FENTRY_RET : 0; return arch_prepare_bpf_trampoline(NULL, image, image_end, - model, flags, tprogs, NULL); + model, flags, tlinks, NULL); } static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, @@ -337,7 +351,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, struct bpf_struct_ops_value *uvalue, *kvalue; const struct btf_member *member; const struct btf_type *t = st_ops->type; - struct bpf_tramp_progs *tprogs = NULL; + struct bpf_tramp_links *tlinks = NULL; void *udata, *kdata; int prog_fd, err = 0; void *image, *image_end; @@ -361,8 +375,8 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (uvalue->state || refcount_read(&uvalue->refcnt)) return -EINVAL; - tprogs = kcalloc(BPF_TRAMP_MAX, sizeof(*tprogs), GFP_KERNEL); - if (!tprogs) + tlinks = kcalloc(BPF_TRAMP_MAX, sizeof(*tlinks), GFP_KERNEL); + if (!tlinks) return -ENOMEM; uvalue = (struct bpf_struct_ops_value *)st_map->uvalue; @@ -385,6 +399,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; + struct bpf_tramp_link *link; u32 moff; moff = __btf_member_bit_offset(t, member) / 8; @@ -438,16 +453,26 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, err = PTR_ERR(prog); goto reset_unlock; } - st_map->progs[i] = prog; if (prog->type != BPF_PROG_TYPE_STRUCT_OPS || prog->aux->attach_btf_id != st_ops->type_id || prog->expected_attach_type != i) { + bpf_prog_put(prog); err = -EINVAL; goto reset_unlock; } - err = bpf_struct_ops_prepare_trampoline(tprogs, prog, + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + bpf_prog_put(prog); + err = -ENOMEM; + goto reset_unlock; + } + bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, + &bpf_struct_ops_link_lops, prog); + st_map->links[i] = &link->link; + + err = bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i], image, image_end); if (err < 0) @@ -490,7 +515,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, memset(uvalue, 0, map->value_size); memset(kvalue, 0, map->value_size); unlock: - kfree(tprogs); + kfree(tlinks); mutex_unlock(&st_map->lock); return err; } @@ -545,9 +570,9 @@ static void bpf_struct_ops_map_free(struct bpf_map *map) { struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map; - if (st_map->progs) + if (st_map->links) bpf_struct_ops_map_put_progs(st_map); - bpf_map_area_free(st_map->progs); + bpf_map_area_free(st_map->links); bpf_jit_free_exec(st_map->image); bpf_map_area_free(st_map->uvalue); bpf_map_area_free(st_map); @@ -596,11 +621,11 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) map = &st_map->map; st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); - st_map->progs = - bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_prog *), + st_map->links = + bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_links *), NUMA_NO_NODE); st_map->image = bpf_jit_alloc_exec(PAGE_SIZE); - if (!st_map->uvalue || !st_map->progs || !st_map->image) { + if (!st_map->uvalue || !st_map->links || !st_map->image) { bpf_struct_ops_map_free(map); return ERR_PTR(-ENOMEM); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index cdaa1152436a..3078c0c9317f 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2640,19 +2640,12 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) } EXPORT_SYMBOL(bpf_link_get_from_fd); -struct bpf_tracing_link { - struct bpf_link link; - enum bpf_attach_type attach_type; - struct bpf_trampoline *trampoline; - struct bpf_prog *tgt_prog; -}; - static void bpf_tracing_link_release(struct bpf_link *link) { struct bpf_tracing_link *tr_link = - container_of(link, struct bpf_tracing_link, link); + container_of(link, struct bpf_tracing_link, link.link); - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link, tr_link->trampoline)); bpf_trampoline_put(tr_link->trampoline); @@ -2665,7 +2658,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) static void bpf_tracing_link_dealloc(struct bpf_link *link) { struct bpf_tracing_link *tr_link = - container_of(link, struct bpf_tracing_link, link); + container_of(link, struct bpf_tracing_link, link.link); kfree(tr_link); } @@ -2674,7 +2667,7 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, struct seq_file *seq) { struct bpf_tracing_link *tr_link = - container_of(link, struct bpf_tracing_link, link); + container_of(link, struct bpf_tracing_link, link.link); seq_printf(seq, "attach_type:\t%d\n", @@ -2685,7 +2678,7 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, struct bpf_link_info *info) { struct bpf_tracing_link *tr_link = - container_of(link, struct bpf_tracing_link, link); + container_of(link, struct bpf_tracing_link, link.link); info->tracing.attach_type = tr_link->attach_type; bpf_trampoline_unpack_key(tr_link->trampoline->key, @@ -2766,7 +2759,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, err = -ENOMEM; goto out_put_prog; } - bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING, + bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, &bpf_tracing_link_lops, prog); link->attach_type = prog->expected_attach_type; @@ -2836,11 +2829,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, tgt_prog = prog->aux->dst_prog; } - err = bpf_link_prime(&link->link, &link_primer); + err = bpf_link_prime(&link->link.link, &link_primer); if (err) goto out_unlock; - err = bpf_trampoline_link_prog(prog, tr); + err = bpf_trampoline_link_prog(&link->link, tr); if (err) { bpf_link_cleanup(&link_primer); link = NULL; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index ada97751ae1b..d5e6bc5517cb 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -168,30 +168,30 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) return ret; } -static struct bpf_tramp_progs * +static struct bpf_tramp_links * bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) { - const struct bpf_prog_aux *aux; - struct bpf_tramp_progs *tprogs; - struct bpf_prog **progs; + struct bpf_tramp_link *link; + struct bpf_tramp_links *tlinks; + struct bpf_tramp_link **links; int kind; *total = 0; - tprogs = kcalloc(BPF_TRAMP_MAX, sizeof(*tprogs), GFP_KERNEL); - if (!tprogs) + tlinks = kcalloc(BPF_TRAMP_MAX, sizeof(*tlinks), GFP_KERNEL); + if (!tlinks) return ERR_PTR(-ENOMEM); for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { - tprogs[kind].nr_progs = tr->progs_cnt[kind]; + tlinks[kind].nr_links = tr->progs_cnt[kind]; *total += tr->progs_cnt[kind]; - progs = tprogs[kind].progs; + links = tlinks[kind].links; - hlist_for_each_entry(aux, &tr->progs_hlist[kind], tramp_hlist) { - *ip_arg |= aux->prog->call_get_func_ip; - *progs++ = aux->prog; + hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { + *ip_arg |= link->link.prog->call_get_func_ip; + *links++ = link; } } - return tprogs; + return tlinks; } static void __bpf_tramp_image_put_deferred(struct work_struct *work) @@ -330,14 +330,14 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx) static int bpf_trampoline_update(struct bpf_trampoline *tr) { struct bpf_tramp_image *im; - struct bpf_tramp_progs *tprogs; + struct bpf_tramp_links *tlinks; u32 flags = BPF_TRAMP_F_RESTORE_REGS; bool ip_arg = false; int err, total; - tprogs = bpf_trampoline_get_progs(tr, &total, &ip_arg); - if (IS_ERR(tprogs)) - return PTR_ERR(tprogs); + tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg); + if (IS_ERR(tlinks)) + return PTR_ERR(tlinks); if (total == 0) { err = unregister_fentry(tr, tr->cur_image->image); @@ -353,15 +353,15 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) goto out; } - if (tprogs[BPF_TRAMP_FEXIT].nr_progs || - tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) + if (tlinks[BPF_TRAMP_FEXIT].nr_links || + tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME; if (ip_arg) flags |= BPF_TRAMP_F_IP_ARG; err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE, - &tr->func.model, flags, tprogs, + &tr->func.model, flags, tlinks, tr->func.addr); if (err < 0) goto out; @@ -381,7 +381,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) tr->cur_image = im; tr->selector++; out: - kfree(tprogs); + kfree(tlinks); return err; } @@ -407,13 +407,14 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; + struct bpf_tramp_link *link_exiting; int err = 0; int cnt; - kind = bpf_attach_type_to_tramp(prog); + kind = bpf_attach_type_to_tramp(link->link.prog); mutex_lock(&tr->mutex); if (tr->extension_prog) { /* cannot attach fentry/fexit if extension prog is attached. @@ -429,25 +430,33 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) err = -EBUSY; goto out; } - tr->extension_prog = prog; + tr->extension_prog = link->link.prog; err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL, - prog->bpf_func); + link->link.prog->bpf_func); goto out; } - if (cnt >= BPF_MAX_TRAMP_PROGS) { + if (cnt >= BPF_MAX_TRAMP_LINKS) { err = -E2BIG; goto out; } - if (!hlist_unhashed(&prog->aux->tramp_hlist)) { + if (!hlist_unhashed(&link->tramp_hlist)) { /* prog already linked */ err = -EBUSY; goto out; } - hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]); + hlist_for_each_entry(link_exiting, &tr->progs_hlist[kind], tramp_hlist) { + if (link_exiting->link.prog != link->link.prog) + continue; + /* prog already linked */ + err = -EBUSY; + goto out; + } + + hlist_add_head(&link->tramp_hlist, &tr->progs_hlist[kind]); tr->progs_cnt[kind]++; err = bpf_trampoline_update(tr); if (err) { - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&link->tramp_hlist); tr->progs_cnt[kind]--; } out: @@ -456,12 +465,12 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; int err; - kind = bpf_attach_type_to_tramp(prog); + kind = bpf_attach_type_to_tramp(link->link.prog); mutex_lock(&tr->mutex); if (kind == BPF_TRAMP_REPLACE) { WARN_ON_ONCE(!tr->extension_prog); @@ -470,7 +479,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) tr->extension_prog = NULL; goto out; } - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&link->tramp_hlist); tr->progs_cnt[kind]--; err = bpf_trampoline_update(tr); out: @@ -635,7 +644,7 @@ void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr) int __weak arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void *image, void *image_end, const struct btf_func_model *m, u32 flags, - struct bpf_tramp_progs *tprogs, + struct bpf_tramp_links *tlinks, void *orig_call) { return -ENOTSUPP; diff --git a/net/bpf/bpf_dummy_struct_ops.c b/net/bpf/bpf_dummy_struct_ops.c index d0e54e30658a..b3717f3ab44b 100644 --- a/net/bpf/bpf_dummy_struct_ops.c +++ b/net/bpf/bpf_dummy_struct_ops.c @@ -72,13 +72,28 @@ static int dummy_ops_call_op(void *image, struct bpf_dummy_ops_test_args *args) args->args[3], args->args[4]); } +static void bpf_struct_ops_link_release(struct bpf_link *link) +{ +} + +static void bpf_struct_ops_link_dealloc(struct bpf_link *link) +{ + kfree(link); +} + +static const struct bpf_link_ops bpf_struct_ops_link_lops = { + .release = bpf_struct_ops_link_release, + .dealloc = bpf_struct_ops_link_dealloc, +}; + int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr) { const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops; const struct btf_type *func_proto; struct bpf_dummy_ops_test_args *args; - struct bpf_tramp_progs *tprogs; + struct bpf_tramp_links *tlinks; + struct bpf_tramp_link *link = NULL; void *image = NULL; unsigned int op_idx; int prog_ret; @@ -92,8 +107,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, if (IS_ERR(args)) return PTR_ERR(args); - tprogs = kcalloc(BPF_TRAMP_MAX, sizeof(*tprogs), GFP_KERNEL); - if (!tprogs) { + tlinks = kcalloc(BPF_TRAMP_MAX, sizeof(*tlinks), GFP_KERNEL); + if (!tlinks) { err = -ENOMEM; goto out; } @@ -105,10 +120,18 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, } set_vm_flush_reset_perms(image); + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err = -ENOMEM; + goto out; + } + bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog); + op_idx = prog->expected_attach_type; - err = bpf_struct_ops_prepare_trampoline(tprogs, prog, + err = bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[op_idx], image, image + PAGE_SIZE); + if (err < 0) goto out; @@ -124,7 +147,9 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, out: kfree(args); bpf_jit_free_exec(image); - kfree(tprogs); + if (link) + bpf_link_put(&link->link); + kfree(tlinks); return err; } diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index 8fb0116f9136..6353a789322b 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -23,6 +23,7 @@ static const char * const link_type_name[] = { [BPF_LINK_TYPE_XDP] = "xdp", [BPF_LINK_TYPE_PERF_EVENT] = "perf_event", [BPF_LINK_TYPE_KPROBE_MULTI] = "kprobe_multi", + [BPF_LINK_TYPE_STRUCT_OPS] = "struct_ops", }; static struct hashmap *link_table; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index d14b10b85e51..a4f557338af7 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1013,6 +1013,7 @@ enum bpf_link_type { BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, BPF_LINK_TYPE_KPROBE_MULTI = 8, + BPF_LINK_TYPE_STRUCT_OPS = 9, MAX_BPF_LINK_TYPE, }; -- 2.30.2 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links 2022-04-12 16:55 ` [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links Kui-Feng Lee @ 2022-04-13 2:43 ` Andrii Nakryiko 2022-04-13 17:53 ` Kui-Feng Lee 0 siblings, 1 reply; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 2:43 UTC (permalink / raw) To: Kui-Feng Lee Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > Replace struct bpf_tramp_progs with struct bpf_tramp_links to collect > struct bpf_tramp_link(s) for a trampoline. struct bpf_tramp_link > extends bpf_link to act as a linked list node. > > arch_prepare_bpf_trampoline() accepts a struct bpf_tramp_links to > collects all bpf_tramp_link(s) that a trampoline should call. > > Change BPF trampoline and bpf_struct_ops to pass bpf_tramp_links > instead of bpf_tramp_progs. > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > --- Looks good, see two comments below. Acked-by: Andrii Nakryiko <andrii@kernel.org> > arch/x86/net/bpf_jit_comp.c | 36 +++++++++-------- > include/linux/bpf.h | 38 ++++++++++++------ > include/linux/bpf_types.h | 1 + > include/uapi/linux/bpf.h | 1 + > kernel/bpf/bpf_struct_ops.c | 69 ++++++++++++++++++++++---------- > kernel/bpf/syscall.c | 23 ++++------- > kernel/bpf/trampoline.c | 73 +++++++++++++++++++--------------- > net/bpf/bpf_dummy_struct_ops.c | 35 +++++++++++++--- > tools/bpf/bpftool/link.c | 1 + > tools/include/uapi/linux/bpf.h | 1 + > 10 files changed, 175 insertions(+), 103 deletions(-) > [...] > /* Different use cases for BPF trampoline: > @@ -704,7 +704,7 @@ struct bpf_tramp_progs { > struct bpf_tramp_image; > int arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void *image, void *image_end, > const struct btf_func_model *m, u32 flags, > - struct bpf_tramp_progs *tprogs, > + struct bpf_tramp_links *tlinks, > void *orig_call); > /* these two functions are called from generated trampoline */ > u64 notrace __bpf_prog_enter(struct bpf_prog *prog); > @@ -803,9 +803,12 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( > { > return bpf_func(ctx, insnsi); > } > + > +struct bpf_link; > + is this forward declaration still needed? was it supposed to be a struct bpf_tramp_link instead? and also probably higher above, before bpf_tramp_links? > #ifdef CONFIG_BPF_JIT > -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); > -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); > +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); > +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); > struct bpf_trampoline *bpf_trampoline_get(u64 key, > struct bpf_attach_target_info *tgt_info); > void bpf_trampoline_put(struct bpf_trampoline *tr); > @@ -856,12 +859,12 @@ int bpf_jit_charge_modmem(u32 size); > void bpf_jit_uncharge_modmem(u32 size); > bool bpf_prog_has_trampoline(const struct bpf_prog *prog); > #else > -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, > +static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, > struct bpf_trampoline *tr) > { > return -ENOTSUPP; > } > -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, > +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, > struct bpf_trampoline *tr) > { > return -ENOTSUPP; > @@ -960,7 +963,6 @@ struct bpf_prog_aux { > bool tail_call_reachable; > bool xdp_has_frags; > bool use_bpf_prog_pack; > - struct hlist_node tramp_hlist; > /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ > const struct btf_type *attach_func_proto; > /* function name for valid attach_btf_id */ > @@ -1047,6 +1049,18 @@ struct bpf_link_ops { > struct bpf_link_info *info); > }; > > +struct bpf_tramp_link { > + struct bpf_link link; > + struct hlist_node tramp_hlist; > +}; > + > +struct bpf_tracing_link { > + struct bpf_tramp_link link; > + enum bpf_attach_type attach_type; > + struct bpf_trampoline *trampoline; > + struct bpf_prog *tgt_prog; > +}; struct bpf_tracing_link can stay in syscall.c, no? don't see anyone needing it outside of syscall.c > + > struct bpf_link_primer { > struct bpf_link *link; > struct file *file; > @@ -1084,8 +1098,8 @@ bool bpf_struct_ops_get(const void *kdata); > void bpf_struct_ops_put(const void *kdata); > int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, > void *value); > -int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_progs *tprogs, > - struct bpf_prog *prog, > +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, > + struct bpf_tramp_link *link, > const struct btf_func_model *model, > void *image, void *image_end); [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links 2022-04-13 2:43 ` Andrii Nakryiko @ 2022-04-13 17:53 ` Kui-Feng Lee 2022-04-13 19:08 ` Andrii Nakryiko 0 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-13 17:53 UTC (permalink / raw) To: andrii.nakryiko; +Cc: daniel, Kernel Team, ast, andrii, bpf On Tue, 2022-04-12 at 19:43 -0700, Andrii Nakryiko wrote: > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > Replace struct bpf_tramp_progs with struct bpf_tramp_links to > > collect > > struct bpf_tramp_link(s) for a trampoline. struct bpf_tramp_link > > extends bpf_link to act as a linked list node. > > > > arch_prepare_bpf_trampoline() accepts a struct bpf_tramp_links to > > collects all bpf_tramp_link(s) that a trampoline should call. > > > > Change BPF trampoline and bpf_struct_ops to pass bpf_tramp_links > > instead of bpf_tramp_progs. > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > --- > > Looks good, see two comments below. > > Acked-by: Andrii Nakryiko <andrii@kernel.org> > > > arch/x86/net/bpf_jit_comp.c | 36 +++++++++-------- > > include/linux/bpf.h | 38 ++++++++++++------ > > include/linux/bpf_types.h | 1 + > > include/uapi/linux/bpf.h | 1 + > > kernel/bpf/bpf_struct_ops.c | 69 ++++++++++++++++++++++-------- > > -- > > kernel/bpf/syscall.c | 23 ++++------- > > kernel/bpf/trampoline.c | 73 +++++++++++++++++++----------- > > ---- > > net/bpf/bpf_dummy_struct_ops.c | 35 +++++++++++++--- > > tools/bpf/bpftool/link.c | 1 + > > tools/include/uapi/linux/bpf.h | 1 + > > 10 files changed, 175 insertions(+), 103 deletions(-) > > > > [...] > > > /* Different use cases for BPF trampoline: > > @@ -704,7 +704,7 @@ struct bpf_tramp_progs { > > struct bpf_tramp_image; > > int arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void > > *image, void *image_end, > > const struct btf_func_model *m, u32 > > flags, > > - struct bpf_tramp_progs *tprogs, > > + struct bpf_tramp_links *tlinks, > > void *orig_call); > > /* these two functions are called from generated trampoline */ > > u64 notrace __bpf_prog_enter(struct bpf_prog *prog); > > @@ -803,9 +803,12 @@ static __always_inline __nocfi unsigned int > > bpf_dispatcher_nop_func( > > { > > return bpf_func(ctx, insnsi); > > } > > + > > +struct bpf_link; > > + > > is this forward declaration still needed? was it supposed to be a > struct bpf_tramp_link instead? and also probably higher above, before > bpf_tramp_links? You are right, I should remvoe it. > > > #ifdef CONFIG_BPF_JIT > > -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct > > bpf_trampoline *tr); > > -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct > > bpf_trampoline *tr); > > +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct > > bpf_trampoline *tr); > > +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct > > bpf_trampoline *tr); > > struct bpf_trampoline *bpf_trampoline_get(u64 key, > > struct > > bpf_attach_target_info *tgt_info); > > void bpf_trampoline_put(struct bpf_trampoline *tr); > > @@ -856,12 +859,12 @@ int bpf_jit_charge_modmem(u32 size); > > void bpf_jit_uncharge_modmem(u32 size); > > bool bpf_prog_has_trampoline(const struct bpf_prog *prog); > > #else > > -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, > > +static inline int bpf_trampoline_link_prog(struct bpf_tramp_link > > *link, > > struct bpf_trampoline > > *tr) > > { > > return -ENOTSUPP; > > } > > -static inline int bpf_trampoline_unlink_prog(struct bpf_prog > > *prog, > > +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link > > *link, > > struct bpf_trampoline > > *tr) > > { > > return -ENOTSUPP; > > @@ -960,7 +963,6 @@ struct bpf_prog_aux { > > bool tail_call_reachable; > > bool xdp_has_frags; > > bool use_bpf_prog_pack; > > - struct hlist_node tramp_hlist; > > /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ > > const struct btf_type *attach_func_proto; > > /* function name for valid attach_btf_id */ > > @@ -1047,6 +1049,18 @@ struct bpf_link_ops { > > struct bpf_link_info *info); > > }; > > > > +struct bpf_tramp_link { > > + struct bpf_link link; > > + struct hlist_node tramp_hlist; > > +}; > > + > > https://docs.google.com/document/d/1FYYdi1Hw4pjulbkvI1VmG-kGqTJRCg7bh1vAF4bwjMc/edit?usp=sharing > > +struct bpf_tracing_link { > > + struct bpf_tramp_link link; > > + enum bpf_attach_type attach_type; > > + struct bpf_trampoline *trampoline; > > + struct bpf_prog *tgt_prog; > > +}; > > struct bpf_tracing_link can stay in syscall.c, no? don't see anyone > needing it outside of syscall.c It will be used by invoke_bpf_prog() of bpf_jit_comp.c in the 3rd patch to get the cookie value. > > > + > > struct bpf_link_primer { > > struct bpf_link *link; > > struct file *file; > > @@ -1084,8 +1098,8 @@ bool bpf_struct_ops_get(const void *kdata); > > void bpf_struct_ops_put(const void *kdata); > > int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void > > *key, > > void *value); > > -int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_progs > > *tprogs, > > - struct bpf_prog *prog, > > +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links > > *tlinks, > > + struct bpf_tramp_link *link, > > const struct btf_func_model > > *model, > > void *image, void *image_end); > > [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links 2022-04-13 17:53 ` Kui-Feng Lee @ 2022-04-13 19:08 ` Andrii Nakryiko 0 siblings, 0 replies; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 19:08 UTC (permalink / raw) To: Kui-Feng Lee; +Cc: daniel, Kernel Team, ast, andrii, bpf On Wed, Apr 13, 2022 at 10:53 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > On Tue, 2022-04-12 at 19:43 -0700, Andrii Nakryiko wrote: > > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > > > Replace struct bpf_tramp_progs with struct bpf_tramp_links to > > > collect > > > struct bpf_tramp_link(s) for a trampoline. struct bpf_tramp_link > > > extends bpf_link to act as a linked list node. > > > > > > arch_prepare_bpf_trampoline() accepts a struct bpf_tramp_links to > > > collects all bpf_tramp_link(s) that a trampoline should call. > > > > > > Change BPF trampoline and bpf_struct_ops to pass bpf_tramp_links > > > instead of bpf_tramp_progs. > > > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > > --- > > > > Looks good, see two comments below. > > > > Acked-by: Andrii Nakryiko <andrii@kernel.org> > > > > > arch/x86/net/bpf_jit_comp.c | 36 +++++++++-------- > > > include/linux/bpf.h | 38 ++++++++++++------ > > > include/linux/bpf_types.h | 1 + > > > include/uapi/linux/bpf.h | 1 + > > > kernel/bpf/bpf_struct_ops.c | 69 ++++++++++++++++++++++-------- > > > -- > > > kernel/bpf/syscall.c | 23 ++++------- > > > kernel/bpf/trampoline.c | 73 +++++++++++++++++++----------- > > > ---- > > > net/bpf/bpf_dummy_struct_ops.c | 35 +++++++++++++--- > > > tools/bpf/bpftool/link.c | 1 + > > > tools/include/uapi/linux/bpf.h | 1 + > > > 10 files changed, 175 insertions(+), 103 deletions(-) > > > > > > > [...] > > > > > /* Different use cases for BPF trampoline: > > > @@ -704,7 +704,7 @@ struct bpf_tramp_progs { > > > struct bpf_tramp_image; > > > int arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void > > > *image, void *image_end, > > > const struct btf_func_model *m, u32 > > > flags, > > > - struct bpf_tramp_progs *tprogs, > > > + struct bpf_tramp_links *tlinks, > > > void *orig_call); > > > /* these two functions are called from generated trampoline */ > > > u64 notrace __bpf_prog_enter(struct bpf_prog *prog); > > > @@ -803,9 +803,12 @@ static __always_inline __nocfi unsigned int > > > bpf_dispatcher_nop_func( > > > { > > > return bpf_func(ctx, insnsi); > > > } > > > + > > > +struct bpf_link; > > > + > > > > is this forward declaration still needed? was it supposed to be a > > struct bpf_tramp_link instead? and also probably higher above, before > > bpf_tramp_links? > > You are right, I should remvoe it. > > > > > > #ifdef CONFIG_BPF_JIT > > > -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct > > > bpf_trampoline *tr); > > > -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct > > > bpf_trampoline *tr); > > > +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct > > > bpf_trampoline *tr); > > > +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct > > > bpf_trampoline *tr); > > > struct bpf_trampoline *bpf_trampoline_get(u64 key, > > > struct > > > bpf_attach_target_info *tgt_info); > > > void bpf_trampoline_put(struct bpf_trampoline *tr); > > > @@ -856,12 +859,12 @@ int bpf_jit_charge_modmem(u32 size); > > > void bpf_jit_uncharge_modmem(u32 size); > > > bool bpf_prog_has_trampoline(const struct bpf_prog *prog); > > > #else > > > -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, > > > +static inline int bpf_trampoline_link_prog(struct bpf_tramp_link > > > *link, > > > struct bpf_trampoline > > > *tr) > > > { > > > return -ENOTSUPP; > > > } > > > -static inline int bpf_trampoline_unlink_prog(struct bpf_prog > > > *prog, > > > +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link > > > *link, > > > struct bpf_trampoline > > > *tr) > > > { > > > return -ENOTSUPP; > > > @@ -960,7 +963,6 @@ struct bpf_prog_aux { > > > bool tail_call_reachable; > > > bool xdp_has_frags; > > > bool use_bpf_prog_pack; > > > - struct hlist_node tramp_hlist; > > > /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ > > > const struct btf_type *attach_func_proto; > > > /* function name for valid attach_btf_id */ > > > @@ -1047,6 +1049,18 @@ struct bpf_link_ops { > > > struct bpf_link_info *info); > > > }; > > > > > > +struct bpf_tramp_link { > > > + struct bpf_link link; > > > + struct hlist_node tramp_hlist; > > > +}; > > > + > > > https://docs.google.com/document/d/1FYYdi1Hw4pjulbkvI1VmG-kGqTJRCg7bh1vAF4bwjMc/edit?usp=sharing > > > +struct bpf_tracing_link { > > > + struct bpf_tramp_link link; > > > + enum bpf_attach_type attach_type; > > > + struct bpf_trampoline *trampoline; > > > + struct bpf_prog *tgt_prog; > > > +}; > > > > struct bpf_tracing_link can stay in syscall.c, no? don't see anyone > > needing it outside of syscall.c > > It will be used by invoke_bpf_prog() of bpf_jit_comp.c in the 3rd patch > to get the cookie value. > Yep, saw it when I got to it, forgot to mention it here. Ignore this suggestion. > > > > > + > > > struct bpf_link_primer { > > > struct bpf_link *link; > > > struct file *file; > > > @@ -1084,8 +1098,8 @@ bool bpf_struct_ops_get(const void *kdata); > > > void bpf_struct_ops_put(const void *kdata); > > > int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void > > > *key, > > > void *value); > > > -int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_progs > > > *tprogs, > > > - struct bpf_prog *prog, > > > +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links > > > *tlinks, > > > + struct bpf_tramp_link *link, > > > const struct btf_func_model > > > *model, > > > void *image, void *image_end); > > > > [...] > ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links Kui-Feng Lee @ 2022-04-12 16:55 ` Kui-Feng Lee 2022-04-13 2:55 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret Kui-Feng Lee ` (2 subsequent siblings) 4 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee BPF trampolines will create a bpf_tramp_run_ctx, a bpf_run_ctx, on stacks and set/reset the current bpf_run_ctx before/after calling a bpf_prog. Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> --- arch/x86/net/bpf_jit_comp.c | 55 +++++++++++++++++++++++++++++++++++++ include/linux/bpf.h | 17 +++++++++--- kernel/bpf/syscall.c | 7 +++-- kernel/bpf/trampoline.c | 20 +++++++++++--- 4 files changed, 89 insertions(+), 10 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 4dcc0b1ac770..0f521be68f7b 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1766,10 +1766,26 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, { u8 *prog = *pprog; u8 *jmp_insn; + int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog *p = l->link.prog; + /* mov rdi, 0 */ + emit_mov_imm64(&prog, BPF_REG_1, 0, 0); + + /* Prepare struct bpf_tramp_run_ctx. + * + * bpf_tramp_run_ctx is already preserved by + * arch_prepare_bpf_trampoline(). + * + * mov QWORD PTR [rsp + ctx_cookie_off], rdi + */ + EMIT4(0x48, 0x89, 0x7C, 0x24); EMIT1(ctx_cookie_off); + /* arg1: mov rdi, progs[i] */ emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); + /* arg2: mov rsi, rsp (struct bpf_run_ctx *) */ + EMIT3(0x48, 0x89, 0xE6); + if (emit_call(&prog, p->aux->sleepable ? __bpf_prog_enter_sleepable : __bpf_prog_enter, prog)) @@ -1815,6 +1831,8 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); /* arg2: mov rsi, rbx <- start time in nsec */ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); + /* arg3: mov rdx, rsp (struct bpf_run_ctx *) */ + EMIT3(0x48, 0x89, 0xE2); if (emit_call(&prog, p->aux->sleepable ? __bpf_prog_exit_sleepable : __bpf_prog_exit, prog)) @@ -2079,6 +2097,16 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } } + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) + EMIT1(0x52); /* push rdx */ + + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) { + /* Prepare struct bpf_tramp_run_ctx. + * sub rsp, sizeof(struct bpf_tramp_run_ctx) + */ + EMIT4(0x48, 0x83, 0xEC, sizeof(struct bpf_tramp_run_ctx)); + } + if (fentry->nr_links) if (invoke_bpf(m, &prog, fentry, regs_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) @@ -2098,6 +2126,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } if (flags & BPF_TRAMP_F_CALL_ORIG) { + /* pop struct bpf_tramp_run_ctx + * add rsp, sizeof(struct bpf_tramp_run_ctx) + */ + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) + EMIT4(0x48, 0x83, 0xC4, sizeof(struct bpf_tramp_run_ctx)); + + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) + EMIT1(0x5A); /* pop rdx */ + restore_regs(m, &prog, nr_args, regs_off); /* call original function */ @@ -2110,6 +2147,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i im->ip_after_call = prog; memcpy(prog, x86_nops[5], X86_PATCH_SIZE); prog += X86_PATCH_SIZE; + + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) + EMIT1(0x52); /* push rdx */ + + /* Prepare struct bpf_tramp_run_ctx. + * sub rsp, sizeof(struct bpf_tramp_run_ctx) + */ + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) + EMIT4(0x48, 0x83, 0xEC, sizeof(struct bpf_tramp_run_ctx)); } if (fmod_ret->nr_links) { @@ -2133,6 +2179,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i goto cleanup; } + /* pop struct bpf_tramp_run_ctx + * add rsp, sizeof(struct bpf_tramp_run_ctx) + */ + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) + EMIT4(0x48, 0x83, 0xC4, sizeof(struct bpf_tramp_run_ctx)); + + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) + EMIT1(0x5A); /* pop rdx */ + if (flags & BPF_TRAMP_F_RESTORE_REGS) restore_regs(m, &prog, nr_args, regs_off); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2f7909eb51ec..d87df049e6b1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -681,6 +681,8 @@ struct bpf_tramp_links { int nr_links; }; +struct bpf_tramp_run_ctx; + /* Different use cases for BPF trampoline: * 1. replace nop at the function entry (kprobe equivalent) * flags = BPF_TRAMP_F_RESTORE_REGS @@ -707,10 +709,11 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *tr, void *image, void *i struct bpf_tramp_links *tlinks, void *orig_call); /* these two functions are called from generated trampoline */ -u64 notrace __bpf_prog_enter(struct bpf_prog *prog); -void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start); -u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog); -void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start); +u64 notrace __bpf_prog_enter(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx); +void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start, struct bpf_tramp_run_ctx *run_ctx); +u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx); +void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx); void notrace __bpf_tramp_enter(struct bpf_tramp_image *tr); void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr); @@ -1304,6 +1307,12 @@ struct bpf_trace_run_ctx { u64 bpf_cookie; }; +struct bpf_tramp_run_ctx { + struct bpf_run_ctx run_ctx; + u64 bpf_cookie; + struct bpf_run_ctx *saved_run_ctx; +}; + static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx) { struct bpf_run_ctx *old_ctx = NULL; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 3078c0c9317f..56e69a582b21 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4775,6 +4775,7 @@ static bool syscall_prog_is_valid_access(int off, int size, BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) { struct bpf_prog * __maybe_unused prog; + struct bpf_tramp_run_ctx __maybe_unused run_ctx; switch (cmd) { case BPF_MAP_CREATE: @@ -4802,13 +4803,15 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) return -EINVAL; } - if (!__bpf_prog_enter_sleepable(prog)) { + run_ctx.bpf_cookie = 0; + run_ctx.saved_run_ctx = NULL; + if (!__bpf_prog_enter_sleepable(prog, &run_ctx)) { /* recursion detected */ bpf_prog_put(prog); return -EBUSY; } attr->test.retval = bpf_prog_run(prog, (void *) (long) attr->test.ctx_in); - __bpf_prog_exit_sleepable(prog, 0 /* bpf_prog_run does runtime stats */); + __bpf_prog_exit_sleepable(prog, 0 /* bpf_prog_run does runtime stats */, &run_ctx); bpf_prog_put(prog); return 0; #endif diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d5e6bc5517cb..baf1b65d523e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -568,11 +568,14 @@ static void notrace inc_misses_counter(struct bpf_prog *prog) * [2..MAX_U64] - execute bpf prog and record execution time. * This is start time. */ -u64 notrace __bpf_prog_enter(struct bpf_prog *prog) +u64 notrace __bpf_prog_enter(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx) __acquires(RCU) { rcu_read_lock(); migrate_disable(); + + run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx); + if (unlikely(__this_cpu_inc_return(*(prog->active)) != 1)) { inc_misses_counter(prog); return 0; @@ -602,29 +605,38 @@ static void notrace update_prog_stats(struct bpf_prog *prog, } } -void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start) +void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start, struct bpf_tramp_run_ctx *run_ctx) __releases(RCU) { + bpf_reset_run_ctx(run_ctx->saved_run_ctx); + update_prog_stats(prog, start); __this_cpu_dec(*(prog->active)); migrate_enable(); rcu_read_unlock(); } -u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog) +u64 notrace __bpf_prog_enter_sleepable(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx) { rcu_read_lock_trace(); migrate_disable(); might_fault(); + if (unlikely(__this_cpu_inc_return(*(prog->active)) != 1)) { inc_misses_counter(prog); return 0; } + + run_ctx->saved_run_ctx = bpf_set_run_ctx(&run_ctx->run_ctx); + return bpf_prog_start_time(); } -void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start) +void notrace __bpf_prog_exit_sleepable(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx) { + bpf_reset_run_ctx(run_ctx->saved_run_ctx); + update_prog_stats(prog, start); __this_cpu_dec(*(prog->active)); migrate_enable(); -- 2.30.2 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack 2022-04-12 16:55 ` [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack Kui-Feng Lee @ 2022-04-13 2:55 ` Andrii Nakryiko 2022-04-13 18:06 ` Kui-Feng Lee 2022-04-13 20:14 ` Kui-Feng Lee 0 siblings, 2 replies; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 2:55 UTC (permalink / raw) To: Kui-Feng Lee Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > BPF trampolines will create a bpf_tramp_run_ctx, a bpf_run_ctx, on > stacks and set/reset the current bpf_run_ctx before/after calling a > bpf_prog. > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > --- > arch/x86/net/bpf_jit_comp.c | 55 +++++++++++++++++++++++++++++++++++++ > include/linux/bpf.h | 17 +++++++++--- > kernel/bpf/syscall.c | 7 +++-- > kernel/bpf/trampoline.c | 20 +++++++++++--- > 4 files changed, 89 insertions(+), 10 deletions(-) > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c > index 4dcc0b1ac770..0f521be68f7b 100644 > --- a/arch/x86/net/bpf_jit_comp.c > +++ b/arch/x86/net/bpf_jit_comp.c > @@ -1766,10 +1766,26 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, > { > u8 *prog = *pprog; > u8 *jmp_insn; > + int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); > struct bpf_prog *p = l->link.prog; > > + /* mov rdi, 0 */ > + emit_mov_imm64(&prog, BPF_REG_1, 0, 0); > + > + /* Prepare struct bpf_tramp_run_ctx. > + * > + * bpf_tramp_run_ctx is already preserved by > + * arch_prepare_bpf_trampoline(). > + * > + * mov QWORD PTR [rsp + ctx_cookie_off], rdi > + */ > + EMIT4(0x48, 0x89, 0x7C, 0x24); EMIT1(ctx_cookie_off); > + > /* arg1: mov rdi, progs[i] */ > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); > + /* arg2: mov rsi, rsp (struct bpf_run_ctx *) */ > + EMIT3(0x48, 0x89, 0xE6); > + > if (emit_call(&prog, > p->aux->sleepable ? __bpf_prog_enter_sleepable : > __bpf_prog_enter, prog)) > @@ -1815,6 +1831,8 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); > /* arg2: mov rsi, rbx <- start time in nsec */ > emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); > + /* arg3: mov rdx, rsp (struct bpf_run_ctx *) */ > + EMIT3(0x48, 0x89, 0xE2); > if (emit_call(&prog, > p->aux->sleepable ? __bpf_prog_exit_sleepable : > __bpf_prog_exit, prog)) > @@ -2079,6 +2097,16 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i > } > } > > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) > + EMIT1(0x52); /* push rdx */ this nr_args < 3 condition is new, maybe leave a comment on why we need this? Also instead of repeating this whole (fentry->nr_links || ... || ...) check, why not move if (nr_args < 3) inside the if right below? > + > + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) { if (nr_args > 3) here? > + /* Prepare struct bpf_tramp_run_ctx. > + * sub rsp, sizeof(struct bpf_tramp_run_ctx) > + */ > + EMIT4(0x48, 0x83, 0xEC, sizeof(struct bpf_tramp_run_ctx)); > + } > + > if (fentry->nr_links) > if (invoke_bpf(m, &prog, fentry, regs_off, > flags & BPF_TRAMP_F_RET_FENTRY_RET)) [...] > if (fmod_ret->nr_links) { > @@ -2133,6 +2179,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i > goto cleanup; > } > > + /* pop struct bpf_tramp_run_ctx > + * add rsp, sizeof(struct bpf_tramp_run_ctx) > + */ > + if (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links) well, actually, can it ever be that this condition doesn't hold? That would mean we are generating empty trampoline for some reason, no? Do we do that? Checking bpf_trampoline_update() and bpf_struct_ops_prepare_trampoline() doesn't seem like we ever do this. So seems like all these checks can be dropped? > + EMIT4(0x48, 0x83, 0xC4, sizeof(struct bpf_tramp_run_ctx)); > + > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || fmod_ret->nr_links)) > + EMIT1(0x5A); /* pop rdx */ same, move it inside if above? > + > if (flags & BPF_TRAMP_F_RESTORE_REGS) > restore_regs(m, &prog, nr_args, regs_off); > [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack 2022-04-13 2:55 ` Andrii Nakryiko @ 2022-04-13 18:06 ` Kui-Feng Lee 2022-04-13 20:14 ` Kui-Feng Lee 1 sibling, 0 replies; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-13 18:06 UTC (permalink / raw) To: andrii.nakryiko; +Cc: daniel, Kernel Team, ast, andrii, bpf On Tue, 2022-04-12 at 19:55 -0700, Andrii Nakryiko wrote: > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > BPF trampolines will create a bpf_tramp_run_ctx, a bpf_run_ctx, on > > stacks and set/reset the current bpf_run_ctx before/after calling a > > bpf_prog. > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > --- > > arch/x86/net/bpf_jit_comp.c | 55 > > +++++++++++++++++++++++++++++++++++++ > > include/linux/bpf.h | 17 +++++++++--- > > kernel/bpf/syscall.c | 7 +++-- > > kernel/bpf/trampoline.c | 20 +++++++++++--- > > 4 files changed, 89 insertions(+), 10 deletions(-) > > > > diff --git a/arch/x86/net/bpf_jit_comp.c > > b/arch/x86/net/bpf_jit_comp.c > > index 4dcc0b1ac770..0f521be68f7b 100644 > > --- a/arch/x86/net/bpf_jit_comp.c > > +++ b/arch/x86/net/bpf_jit_comp.c > > @@ -1766,10 +1766,26 @@ static int invoke_bpf_prog(const struct > > btf_func_model *m, u8 **pprog, > > { > > u8 *prog = *pprog; > > u8 *jmp_insn; > > + int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, > > bpf_cookie); > > struct bpf_prog *p = l->link.prog; > > > > + /* mov rdi, 0 */ > > + emit_mov_imm64(&prog, BPF_REG_1, 0, 0); > > + > > + /* Prepare struct bpf_tramp_run_ctx. > > + * > > + * bpf_tramp_run_ctx is already preserved by > > + * arch_prepare_bpf_trampoline(). > > + * > > + * mov QWORD PTR [rsp + ctx_cookie_off], rdi > > + */ > > + EMIT4(0x48, 0x89, 0x7C, 0x24); EMIT1(ctx_cookie_off); > > + > > /* arg1: mov rdi, progs[i] */ > > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) > > (long) p); > > + /* arg2: mov rsi, rsp (struct bpf_run_ctx *) */ > > + EMIT3(0x48, 0x89, 0xE6); > > + > > if (emit_call(&prog, > > p->aux->sleepable ? > > __bpf_prog_enter_sleepable : > > __bpf_prog_enter, prog)) > > @@ -1815,6 +1831,8 @@ static int invoke_bpf_prog(const struct > > btf_func_model *m, u8 **pprog, > > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) > > (long) p); > > /* arg2: mov rsi, rbx <- start time in nsec */ > > emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); > > + /* arg3: mov rdx, rsp (struct bpf_run_ctx *) */ > > + EMIT3(0x48, 0x89, 0xE2); > > if (emit_call(&prog, > > p->aux->sleepable ? __bpf_prog_exit_sleepable > > : > > __bpf_prog_exit, prog)) > > @@ -2079,6 +2097,16 @@ int arch_prepare_bpf_trampoline(struct > > bpf_tramp_image *im, void *image, void *i > > } > > } > > > > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || > > fmod_ret->nr_links)) > > + EMIT1(0x52); /* push rdx */ > > this nr_args < 3 condition is new, maybe leave a comment on why we > need this? Also instead of repeating this whole (fentry->nr_links || > ... || ...) check, why not move if (nr_args < 3) inside the if right > below? > > > + > > + if (fentry->nr_links || fexit->nr_links || fmod_ret- > > >nr_links) { > > if (nr_args > 3) here? > > > + /* Prepare struct bpf_tramp_run_ctx. > > + * sub rsp, sizeof(struct bpf_tramp_run_ctx) > > + */ > > + EMIT4(0x48, 0x83, 0xEC, sizeof(struct > > bpf_tramp_run_ctx)); > > + } > > + > > if (fentry->nr_links) > > if (invoke_bpf(m, &prog, fentry, regs_off, > > flags & BPF_TRAMP_F_RET_FENTRY_RET)) > > [...] > > > if (fmod_ret->nr_links) { > > @@ -2133,6 +2179,15 @@ int arch_prepare_bpf_trampoline(struct > > bpf_tramp_image *im, void *image, void *i > > goto cleanup; > > } > > > > + /* pop struct bpf_tramp_run_ctx > > + * add rsp, sizeof(struct bpf_tramp_run_ctx) > > + */ > > + if (fentry->nr_links || fexit->nr_links || fmod_ret- > > >nr_links) > > well, actually, can it ever be that this condition doesn't hold? That > would mean we are generating empty trampoline for some reason, no? Do > we do that? Checking bpf_trampoline_update() and > bpf_struct_ops_prepare_trampoline() doesn't seem like we ever do > this. > So seems like all these checks can be dropped? You are right. I had added this check for doing the following line only for some cases, and didn't aware the check is no more useful after changing the way of doing it. > > + EMIT4(0x48, 0x83, 0xC4, sizeof(struct > > bpf_tramp_run_ctx)); > > + > > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || > > fmod_ret->nr_links)) > > + EMIT1(0x5A); /* pop rdx */ > > same, move it inside if above? > > > + > > if (flags & BPF_TRAMP_F_RESTORE_REGS) > > restore_regs(m, &prog, nr_args, regs_off); > > > > [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack 2022-04-13 2:55 ` Andrii Nakryiko 2022-04-13 18:06 ` Kui-Feng Lee @ 2022-04-13 20:14 ` Kui-Feng Lee 1 sibling, 0 replies; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-13 20:14 UTC (permalink / raw) To: andrii.nakryiko; +Cc: daniel, Kernel Team, ast, andrii, bpf On Tue, 2022-04-12 at 19:55 -0700, Andrii Nakryiko wrote: > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > BPF trampolines will create a bpf_tramp_run_ctx, a bpf_run_ctx, on > > stacks and set/reset the current bpf_run_ctx before/after calling a > > bpf_prog. > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > --- > > arch/x86/net/bpf_jit_comp.c | 55 > > +++++++++++++++++++++++++++++++++++++ > > include/linux/bpf.h | 17 +++++++++--- > > kernel/bpf/syscall.c | 7 +++-- > > kernel/bpf/trampoline.c | 20 +++++++++++--- > > 4 files changed, 89 insertions(+), 10 deletions(-) > > > > diff --git a/arch/x86/net/bpf_jit_comp.c > > b/arch/x86/net/bpf_jit_comp.c > > index 4dcc0b1ac770..0f521be68f7b 100644 > > --- a/arch/x86/net/bpf_jit_comp.c > > +++ b/arch/x86/net/bpf_jit_comp.c > > @@ -1766,10 +1766,26 @@ static int invoke_bpf_prog(const struct > > btf_func_model *m, u8 **pprog, > > { > > u8 *prog = *pprog; > > u8 *jmp_insn; > > + int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, > > bpf_cookie); > > struct bpf_prog *p = l->link.prog; > > > > + /* mov rdi, 0 */ > > + emit_mov_imm64(&prog, BPF_REG_1, 0, 0); > > + > > + /* Prepare struct bpf_tramp_run_ctx. > > + * > > + * bpf_tramp_run_ctx is already preserved by > > + * arch_prepare_bpf_trampoline(). > > + * > > + * mov QWORD PTR [rsp + ctx_cookie_off], rdi > > + */ > > + EMIT4(0x48, 0x89, 0x7C, 0x24); EMIT1(ctx_cookie_off); > > + > > /* arg1: mov rdi, progs[i] */ > > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) > > (long) p); > > + /* arg2: mov rsi, rsp (struct bpf_run_ctx *) */ > > + EMIT3(0x48, 0x89, 0xE6); > > + > > if (emit_call(&prog, > > p->aux->sleepable ? > > __bpf_prog_enter_sleepable : > > __bpf_prog_enter, prog)) > > @@ -1815,6 +1831,8 @@ static int invoke_bpf_prog(const struct > > btf_func_model *m, u8 **pprog, > > emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) > > (long) p); > > /* arg2: mov rsi, rbx <- start time in nsec */ > > emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); > > + /* arg3: mov rdx, rsp (struct bpf_run_ctx *) */ > > + EMIT3(0x48, 0x89, 0xE2); > > if (emit_call(&prog, > > p->aux->sleepable ? __bpf_prog_exit_sleepable > > : > > __bpf_prog_exit, prog)) > > @@ -2079,6 +2097,16 @@ int arch_prepare_bpf_trampoline(struct > > bpf_tramp_image *im, void *image, void *i > > } > > } > > > > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || > > fmod_ret->nr_links)) > > + EMIT1(0x52); /* push rdx */ > > this nr_args < 3 condition is new, maybe leave a comment on why we > need this? Also instead of repeating this whole (fentry->nr_links || > ... || ...) check, why not move if (nr_args < 3) inside the if right > below? I thought rdx is a nonvolatile (callee-saved) register. Checking ABI again, I was wrong. I am removing this part. > > > + > > + if (fentry->nr_links || fexit->nr_links || fmod_ret- > > >nr_links) { > > if (nr_args > 3) here? > > > + /* Prepare struct bpf_tramp_run_ctx. > > + * sub rsp, sizeof(struct bpf_tramp_run_ctx) > > + */ > > + EMIT4(0x48, 0x83, 0xEC, sizeof(struct > > bpf_tramp_run_ctx)); > > + } > > + > > if (fentry->nr_links) > > if (invoke_bpf(m, &prog, fentry, regs_off, > > flags & BPF_TRAMP_F_RET_FENTRY_RET)) > > [...] > > > if (fmod_ret->nr_links) { > > @@ -2133,6 +2179,15 @@ int arch_prepare_bpf_trampoline(struct > > bpf_tramp_image *im, void *image, void *i > > goto cleanup; > > } > > > > + /* pop struct bpf_tramp_run_ctx > > + * add rsp, sizeof(struct bpf_tramp_run_ctx) > > + */ > > + if (fentry->nr_links || fexit->nr_links || fmod_ret- > > >nr_links) > > well, actually, can it ever be that this condition doesn't hold? That > would mean we are generating empty trampoline for some reason, no? Do > we do that? Checking bpf_trampoline_update() and > bpf_struct_ops_prepare_trampoline() doesn't seem like we ever do > this. > So seems like all these checks can be dropped? > > > + EMIT4(0x48, 0x83, 0xC4, sizeof(struct > > bpf_tramp_run_ctx)); > > + > > + if (nr_args < 3 && (fentry->nr_links || fexit->nr_links || > > fmod_ret->nr_links)) > > + EMIT1(0x5A); /* pop rdx */ > > same, move it inside if above? > > > + > > if (flags & BPF_TRAMP_F_RESTORE_REGS) > > restore_regs(m, &prog, nr_args, regs_off); > > > > [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret. 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack Kui-Feng Lee @ 2022-04-12 16:55 ` Kui-Feng Lee 2022-04-13 3:03 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret Kui-Feng Lee 4 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee Add a bpf_cookie field to struct bpf_tracing_link to attach a cookie to a link of a trace hook. A cookie of a bpf_tracing_link is available by calling bpf_get_attach_cookie when running the BPF program of the attached link. The value of a cookie will be set at bpf_tramp_run_ctx by the trampoline of the link. Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> --- arch/x86/net/bpf_jit_comp.c | 11 +++++++++-- include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 8 ++++++++ kernel/bpf/syscall.c | 27 +++++++++++++++++++++++---- kernel/trace/bpf_trace.c | 17 +++++++++++++++++ tools/include/uapi/linux/bpf.h | 7 +++++++ 6 files changed, 65 insertions(+), 6 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 0f521be68f7b..18d02dcd810a 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1764,13 +1764,20 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, struct bpf_tramp_link *l, int stack_size, bool save_ret) { + u64 cookie = 0; u8 *prog = *pprog; u8 *jmp_insn; int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog *p = l->link.prog; - /* mov rdi, 0 */ - emit_mov_imm64(&prog, BPF_REG_1, 0, 0); + if (l->link.type == BPF_LINK_TYPE_TRACING) { + struct bpf_tracing_link *tr_link = + container_of(l, struct bpf_tracing_link, link); + cookie = tr_link->cookie; + } + + /* mov rdi, cookie */ + emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); /* Prepare struct bpf_tramp_run_ctx. * diff --git a/include/linux/bpf.h b/include/linux/bpf.h index d87df049e6b1..49db9c065701 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1062,6 +1062,7 @@ struct bpf_tracing_link { enum bpf_attach_type attach_type; struct bpf_trampoline *trampoline; struct bpf_prog *tgt_prog; + u64 cookie; }; struct bpf_link_primer { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index a4f557338af7..5e901e6d17ea 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1436,6 +1436,7 @@ union bpf_attr { struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */ __u64 name; __u32 prog_fd; + __u64 bpf_cookie; } raw_tracepoint; struct { /* anonymous struct for BPF_BTF_LOAD */ @@ -1490,6 +1491,13 @@ union bpf_attr { __aligned_u64 addrs; __aligned_u64 cookies; } kprobe_multi; + struct { + /* black box user-provided value passed through + * to BPF program at the execution time and + * accessible through bpf_get_attach_cookie() BPF helper + */ + __u64 bpf_cookie; + } tracing; }; } link_create; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 56e69a582b21..53d4da5c76b5 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2695,9 +2695,10 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { .fill_link_info = bpf_tracing_link_fill_link_info, }; -static int bpf_tracing_prog_attach(struct bpf_prog *prog, - int tgt_prog_fd, - u32 btf_id) +static int bpf_tracing_prog_attach_cookie(struct bpf_prog *prog, + int tgt_prog_fd, + u32 btf_id, + u64 bpf_cookie) { struct bpf_link_primer link_primer; struct bpf_prog *tgt_prog = NULL; @@ -2762,6 +2763,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, &bpf_tracing_link_lops, prog); link->attach_type = prog->expected_attach_type; + link->cookie = bpf_cookie; mutex_lock(&prog->aux->dst_mutex); @@ -2871,6 +2873,13 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, return err; } +static int bpf_tracing_prog_attach(struct bpf_prog *prog, + int tgt_prog_fd, + u32 btf_id) +{ + return bpf_tracing_prog_attach_cookie(prog, tgt_prog_fd, btf_id, 0); +} + struct bpf_raw_tp_link { struct bpf_link link; struct bpf_raw_event_map *btp; @@ -3023,7 +3032,7 @@ static int bpf_perf_link_attach(const union bpf_attr *attr, struct bpf_prog *pro } #endif /* CONFIG_PERF_EVENTS */ -#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd +#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.bpf_cookie static int bpf_raw_tracepoint_open(const union bpf_attr *attr) { @@ -3187,6 +3196,10 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) return BPF_PROG_TYPE_SK_LOOKUP; case BPF_XDP: return BPF_PROG_TYPE_XDP; + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + case BPF_MODIFY_RETURN: + return BPF_PROG_TYPE_TRACING; default: return BPF_PROG_TYPE_UNSPEC; } @@ -4251,6 +4264,12 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr, return bpf_tracing_prog_attach(prog, attr->link_create.target_fd, attr->link_create.target_btf_id); + else if (prog->type == BPF_PROG_TYPE_TRACING) + return bpf_tracing_prog_attach_cookie(prog, + 0, + 0, + attr->link_create.tracing.bpf_cookie); + return -EINVAL; } diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index b26f3da943de..c5713d5fba45 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1088,6 +1088,21 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = { .arg1_type = ARG_PTR_TO_CTX, }; +BPF_CALL_1(bpf_get_attach_cookie_tracing, void *, ctx) +{ + struct bpf_trace_run_ctx *run_ctx; + + run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx); + return run_ctx->bpf_cookie; +} + +static const struct bpf_func_proto bpf_get_attach_cookie_proto_tracing = { + .func = bpf_get_attach_cookie_tracing, + .gpl_only = false, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, +}; + BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags) { #ifndef CONFIG_X86 @@ -1716,6 +1731,8 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return bpf_prog_has_trampoline(prog) ? &bpf_get_func_ret_proto : NULL; case BPF_FUNC_get_func_arg_cnt: return bpf_prog_has_trampoline(prog) ? &bpf_get_func_arg_cnt_proto : NULL; + case BPF_FUNC_get_attach_cookie: + return bpf_prog_has_trampoline(prog) ? &bpf_get_attach_cookie_proto_tracing : NULL; default: fn = raw_tp_prog_func_proto(func_id, prog); if (!fn && prog->expected_attach_type == BPF_TRACE_ITER) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index a4f557338af7..a5ee57f09e04 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1490,6 +1490,13 @@ union bpf_attr { __aligned_u64 addrs; __aligned_u64 cookies; } kprobe_multi; + struct { + /* black box user-provided value passed through + * to BPF program at the execution time and + * accessible through bpf_get_attach_cookie() BPF helper + */ + __u64 bpf_cookie; + } tracing; }; } link_create; -- 2.30.2 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret. 2022-04-12 16:55 ` [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret Kui-Feng Lee @ 2022-04-13 3:03 ` Andrii Nakryiko 2022-04-13 18:14 ` Kui-Feng Lee 0 siblings, 1 reply; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 3:03 UTC (permalink / raw) To: Kui-Feng Lee Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > Add a bpf_cookie field to struct bpf_tracing_link to attach a cookie > to a link of a trace hook. A cookie of a bpf_tracing_link is > available by calling bpf_get_attach_cookie when running the BPF > program of the attached link. > > The value of a cookie will be set at bpf_tramp_run_ctx by the > trampoline of the link. > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > --- > arch/x86/net/bpf_jit_comp.c | 11 +++++++++-- > include/linux/bpf.h | 1 + > include/uapi/linux/bpf.h | 8 ++++++++ > kernel/bpf/syscall.c | 27 +++++++++++++++++++++++---- > kernel/trace/bpf_trace.c | 17 +++++++++++++++++ > tools/include/uapi/linux/bpf.h | 7 +++++++ > 6 files changed, 65 insertions(+), 6 deletions(-) > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c > index 0f521be68f7b..18d02dcd810a 100644 > --- a/arch/x86/net/bpf_jit_comp.c > +++ b/arch/x86/net/bpf_jit_comp.c > @@ -1764,13 +1764,20 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, > struct bpf_tramp_link *l, int stack_size, > bool save_ret) > { > + u64 cookie = 0; > u8 *prog = *pprog; > u8 *jmp_insn; > int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); > struct bpf_prog *p = l->link.prog; > > - /* mov rdi, 0 */ > - emit_mov_imm64(&prog, BPF_REG_1, 0, 0); > + if (l->link.type == BPF_LINK_TYPE_TRACING) { > + struct bpf_tracing_link *tr_link = > + container_of(l, struct bpf_tracing_link, link); empty line between variable declaration block and statement > + cookie = tr_link->cookie; > + } > + > + /* mov rdi, cookie */ > + emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); > > /* Prepare struct bpf_tramp_run_ctx. > * > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > index d87df049e6b1..49db9c065701 100644 > --- a/include/linux/bpf.h > +++ b/include/linux/bpf.h > @@ -1062,6 +1062,7 @@ struct bpf_tracing_link { > enum bpf_attach_type attach_type; > struct bpf_trampoline *trampoline; > struct bpf_prog *tgt_prog; > + u64 cookie; > }; > > struct bpf_link_primer { > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index a4f557338af7..5e901e6d17ea 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -1436,6 +1436,7 @@ union bpf_attr { > struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */ > __u64 name; > __u32 prog_fd; > + __u64 bpf_cookie; I thought we are not adding bpf_cookie through RAW_TRACEPOINT_OPEN? > } raw_tracepoint; > > struct { /* anonymous struct for BPF_BTF_LOAD */ > @@ -1490,6 +1491,13 @@ union bpf_attr { > __aligned_u64 addrs; > __aligned_u64 cookies; > } kprobe_multi; > + struct { > + /* black box user-provided value passed through > + * to BPF program at the execution time and > + * accessible through bpf_get_attach_cookie() BPF helper > + */ > + __u64 bpf_cookie; > + } tracing; > }; > } link_create; > > diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c > index 56e69a582b21..53d4da5c76b5 100644 > --- a/kernel/bpf/syscall.c > +++ b/kernel/bpf/syscall.c > @@ -2695,9 +2695,10 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { > .fill_link_info = bpf_tracing_link_fill_link_info, > }; > > -static int bpf_tracing_prog_attach(struct bpf_prog *prog, > - int tgt_prog_fd, > - u32 btf_id) > +static int bpf_tracing_prog_attach_cookie(struct bpf_prog *prog, > + int tgt_prog_fd, > + u32 btf_id, > + u64 bpf_cookie) let's keep it as bpf_tracing_prog_attach and just update all the call sites > { > struct bpf_link_primer link_primer; > struct bpf_prog *tgt_prog = NULL; > @@ -2762,6 +2763,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, > bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, > &bpf_tracing_link_lops, prog); > link->attach_type = prog->expected_attach_type; > + link->cookie = bpf_cookie; > > mutex_lock(&prog->aux->dst_mutex); > > @@ -2871,6 +2873,13 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, > return err; > } > > +static int bpf_tracing_prog_attach(struct bpf_prog *prog, > + int tgt_prog_fd, > + u32 btf_id) > +{ > + return bpf_tracing_prog_attach_cookie(prog, tgt_prog_fd, btf_id, 0); > +} > + why this wrapper, just add that 0 for all existing bpf_tracing_prog_attach() invocations > struct bpf_raw_tp_link { > struct bpf_link link; > struct bpf_raw_event_map *btp; > @@ -3023,7 +3032,7 @@ static int bpf_perf_link_attach(const union bpf_attr *attr, struct bpf_prog *pro > } > #endif /* CONFIG_PERF_EVENTS */ > > -#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd > +#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.bpf_cookie > > static int bpf_raw_tracepoint_open(const union bpf_attr *attr) > { > @@ -3187,6 +3196,10 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) > return BPF_PROG_TYPE_SK_LOOKUP; > case BPF_XDP: > return BPF_PROG_TYPE_XDP; > + case BPF_TRACE_FENTRY: > + case BPF_TRACE_FEXIT: > + case BPF_MODIFY_RETURN: > + return BPF_PROG_TYPE_TRACING; can you please split the part that adds ability to create fentry/fexit/etc links through LINK_CREATE into a separate commit, separate from wiring the bpf_cookie? It's two logically distinct changes. > default: > return BPF_PROG_TYPE_UNSPEC; > } [...] > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index a4f557338af7..a5ee57f09e04 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h > @@ -1490,6 +1490,13 @@ union bpf_attr { > __aligned_u64 addrs; > __aligned_u64 cookies; > } kprobe_multi; > + struct { > + /* black box user-provided value passed through > + * to BPF program at the execution time and > + * accessible through bpf_get_attach_cookie() BPF helper > + */ > + __u64 bpf_cookie; for kprobe_multi we used "cookies", so let's do "cookie" here? We'll have to keep perf_event.bpf_cookie, of course. > + } tracing; > }; > } link_create; > > -- > 2.30.2 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret. 2022-04-13 3:03 ` Andrii Nakryiko @ 2022-04-13 18:14 ` Kui-Feng Lee 0 siblings, 0 replies; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-13 18:14 UTC (permalink / raw) To: andrii.nakryiko; +Cc: daniel, Kernel Team, ast, andrii, bpf On Tue, 2022-04-12 at 20:03 -0700, Andrii Nakryiko wrote: > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > Add a bpf_cookie field to struct bpf_tracing_link to attach a > > cookie > > to a link of a trace hook. A cookie of a bpf_tracing_link is > > available by calling bpf_get_attach_cookie when running the BPF > > program of the attached link. > > > > The value of a cookie will be set at bpf_tramp_run_ctx by the > > trampoline of the link. > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > --- > > arch/x86/net/bpf_jit_comp.c | 11 +++++++++-- > > include/linux/bpf.h | 1 + > > include/uapi/linux/bpf.h | 8 ++++++++ > > kernel/bpf/syscall.c | 27 +++++++++++++++++++++++---- > > kernel/trace/bpf_trace.c | 17 +++++++++++++++++ > > tools/include/uapi/linux/bpf.h | 7 +++++++ > > 6 files changed, 65 insertions(+), 6 deletions(-) > > > > diff --git a/arch/x86/net/bpf_jit_comp.c > > b/arch/x86/net/bpf_jit_comp.c > > index 0f521be68f7b..18d02dcd810a 100644 > > --- a/arch/x86/net/bpf_jit_comp.c > > +++ b/arch/x86/net/bpf_jit_comp.c > > @@ -1764,13 +1764,20 @@ static int invoke_bpf_prog(const struct > > btf_func_model *m, u8 **pprog, > > struct bpf_tramp_link *l, int > > stack_size, > > bool save_ret) > > { > > + u64 cookie = 0; > > u8 *prog = *pprog; > > u8 *jmp_insn; > > int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, > > bpf_cookie); > > struct bpf_prog *p = l->link.prog; > > > > - /* mov rdi, 0 */ > > - emit_mov_imm64(&prog, BPF_REG_1, 0, 0); > > + if (l->link.type == BPF_LINK_TYPE_TRACING) { > > + struct bpf_tracing_link *tr_link = > > + container_of(l, struct bpf_tracing_link, > > link); > > empty line between variable declaration block and statement Got it! > > > + cookie = tr_link->cookie; > > + } > > + > > + /* mov rdi, cookie */ > > + emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) > > (long) cookie); > > > > /* Prepare struct bpf_tramp_run_ctx. > > * > > diff --git a/include/linux/bpf.h b/include/linux/bpf.h > > index d87df049e6b1..49db9c065701 100644 > > --- a/include/linux/bpf.h > > +++ b/include/linux/bpf.h > > @@ -1062,6 +1062,7 @@ struct bpf_tracing_link { > > enum bpf_attach_type attach_type; > > struct bpf_trampoline *trampoline; > > struct bpf_prog *tgt_prog; > > + u64 cookie; > > }; > > > > struct bpf_link_primer { > > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > > index a4f557338af7..5e901e6d17ea 100644 > > --- a/include/uapi/linux/bpf.h > > +++ b/include/uapi/linux/bpf.h > > @@ -1436,6 +1436,7 @@ union bpf_attr { > > struct { /* anonymous struct used by > > BPF_RAW_TRACEPOINT_OPEN command */ > > __u64 name; > > __u32 prog_fd; > > + __u64 bpf_cookie; > > I thought we are not adding bpf_cookie through RAW_TRACEPOINT_OPEN? Yes, this line should be removed. > > > } raw_tracepoint; > > > > struct { /* anonymous struct for BPF_BTF_LOAD */ > > @@ -1490,6 +1491,13 @@ union bpf_attr { > > __aligned_u64 addrs; > > __aligned_u64 cookies; > > } kprobe_multi; > > + struct { > > + /* black box user-provided value > > passed through > > + * to BPF program at the execution > > time and > > + * accessible through > > bpf_get_attach_cookie() BPF helper > > + */ > > + __u64 bpf_cookie; > > + } tracing; > > }; > > } link_create; > > > > diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c > > index 56e69a582b21..53d4da5c76b5 100644 > > --- a/kernel/bpf/syscall.c > > +++ b/kernel/bpf/syscall.c > > @@ -2695,9 +2695,10 @@ static const struct bpf_link_ops > > bpf_tracing_link_lops = { > > .fill_link_info = bpf_tracing_link_fill_link_info, > > }; > > > > -static int bpf_tracing_prog_attach(struct bpf_prog *prog, > > - int tgt_prog_fd, > > - u32 btf_id) > > +static int bpf_tracing_prog_attach_cookie(struct bpf_prog *prog, > > + int tgt_prog_fd, > > + u32 btf_id, > > + u64 bpf_cookie) > > let's keep it as bpf_tracing_prog_attach and just update all the call > sites Roger > > > { > > struct bpf_link_primer link_primer; > > struct bpf_prog *tgt_prog = NULL; > > @@ -2762,6 +2763,7 @@ static int bpf_tracing_prog_attach(struct > > bpf_prog *prog, > > bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, > > &bpf_tracing_link_lops, prog); > > link->attach_type = prog->expected_attach_type; > > + link->cookie = bpf_cookie; > > > > mutex_lock(&prog->aux->dst_mutex); > > > > @@ -2871,6 +2873,13 @@ static int bpf_tracing_prog_attach(struct > > bpf_prog *prog, > > return err; > > } > > > > +static int bpf_tracing_prog_attach(struct bpf_prog *prog, > > + int tgt_prog_fd, > > + u32 btf_id) > > +{ > > + return bpf_tracing_prog_attach_cookie(prog, tgt_prog_fd, > > btf_id, 0); > > +} > > https://docs.google.com/document/d/1FYYdi1Hw4pjulbkvI1VmG-kGqTJRCg7bh1vAF4bwjMc/edit?usp=sharing > > + > > why this wrapper, just add that 0 for all existing > bpf_tracing_prog_attach() invocations Got it! > > > > struct bpf_raw_tp_link { > > struct bpf_link link; > > struct bpf_raw_event_map *btp; > > @@ -3023,7 +3032,7 @@ static int bpf_perf_link_attach(const union > > bpf_attr *attr, struct bpf_prog *pro > > } > > #endif /* CONFIG_PERF_EVENTS */ > > > > -#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD raw_tracepoint.prog_fd > > +#define BPF_RAW_TRACEPOINT_OPEN_LAST_FIELD > > raw_tracepoint.bpf_cookie > > > > static int bpf_raw_tracepoint_open(const union bpf_attr *attr) > > { > > @@ -3187,6 +3196,10 @@ attach_type_to_prog_type(enum > > bpf_attach_type attach_type) > > return BPF_PROG_TYPE_SK_LOOKUP; > > case BPF_XDP: > > return BPF_PROG_TYPE_XDP; > > + case BPF_TRACE_FENTRY: > > + case BPF_TRACE_FEXIT: > > + case BPF_MODIFY_RETURN: > > + return BPF_PROG_TYPE_TRACING; > > can you please split the part that adds ability to create > fentry/fexit/etc links through LINK_CREATE into a separate commit, > separate from wiring the bpf_cookie? It's two logically distinct > changes. No problem. > > > default: > > return BPF_PROG_TYPE_UNSPEC; > > } > > [...] > > > diff --git a/tools/include/uapi/linux/bpf.h > > b/tools/include/uapi/linux/bpf.h > > index a4f557338af7..a5ee57f09e04 100644 > > --- a/tools/include/uapi/linux/bpf.h > > +++ b/tools/include/uapi/linux/bpf.h > > @@ -1490,6 +1490,13 @@ union bpf_attr { > > __aligned_u64 addrs; > > __aligned_u64 cookies; > > } kprobe_multi; > > + struct { > > + /* black box user-provided value > > passed through > > + * to BPF program at the execution > > time and > > + * accessible through > > bpf_get_attach_cookie() BPF helper > > + */ > > + __u64 bpf_cookie; > > for kprobe_multi we used "cookies", so let's do "cookie" here? We'll > have to keep perf_event.bpf_cookie, of course. Got it! > > > + } tracing; > > }; > > } link_create; > > > > -- > > 2.30.2 > > ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf. 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee ` (2 preceding siblings ...) 2022-04-12 16:55 ` [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret Kui-Feng Lee @ 2022-04-12 16:55 ` Kui-Feng Lee 2022-04-13 3:14 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret Kui-Feng Lee 4 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee Add a cookie field to the attributes of bpf_link_create(). Add bpf_program__attach_trace_opts() to attach a cookie to a link. Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> --- tools/lib/bpf/bpf.c | 7 +++++++ tools/lib/bpf/bpf.h | 3 +++ tools/lib/bpf/libbpf.c | 33 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.h | 12 ++++++++++++ tools/lib/bpf/libbpf.map | 1 + 5 files changed, 56 insertions(+) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index cf27251adb92..c2454979c3c4 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -863,6 +863,13 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, kprobe_multi)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + case BPF_MODIFY_RETURN: + attr.link_create.tracing.bpf_cookie = OPTS_GET(opts, tracing.bpf_cookie, 0); + if (!OPTS_ZEROED(opts, tracing)) + return libbpf_err(-EINVAL); + break; default: if (!OPTS_ZEROED(opts, flags)) return libbpf_err(-EINVAL); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index f4b4afb6d4ba..4cdbabcccefc 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -410,6 +410,9 @@ struct bpf_link_create_opts { __u32 iter_info_len; __u32 target_btf_id; union { + struct { + __u64 bpf_cookie; + } tracing; struct { __u64 bpf_cookie; } perf_event; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index bf4f7ac54ebf..8586e1efd780 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -11262,6 +11262,39 @@ struct bpf_link *bpf_program__attach_trace(const struct bpf_program *prog) return bpf_program__attach_btf_id(prog); } +struct bpf_link *bpf_program__attach_trace_opts(const struct bpf_program *prog, + const struct bpf_trace_opts *opts) +{ + char errmsg[STRERR_BUFSIZE]; + struct bpf_link *link; + int prog_fd, pfd; + + LIBBPF_OPTS(bpf_link_create_opts, link_opts); + + prog_fd = bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return libbpf_err_ptr(-EINVAL); + } + + link = calloc(1, sizeof(*link)); + if (!link) + return libbpf_err_ptr(-ENOMEM); + link->detach = &bpf_link__detach_fd; + + link_opts.tracing.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0); + pfd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &link_opts); + if (pfd < 0) { + pfd = -errno; + free(link); + pr_warn("prog '%s': failed to attach: %s\n", + prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg))); + return libbpf_err_ptr(pfd); + } + link->fd = pfd; + return (struct bpf_link *)link; +} + struct bpf_link *bpf_program__attach_lsm(const struct bpf_program *prog) { return bpf_program__attach_btf_id(prog); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index 63d66f1adf1a..e0dd3f9a5aca 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -563,8 +563,20 @@ bpf_program__attach_tracepoint_opts(const struct bpf_program *prog, LIBBPF_API struct bpf_link * bpf_program__attach_raw_tracepoint(const struct bpf_program *prog, const char *tp_name); + +struct bpf_trace_opts { + /* size of this struct, for forward/backward compatibility */ + size_t sz; + /* custom user-provided value fetchable through bpf_get_attach_cookie() */ + __u64 bpf_cookie; +}; +#define bpf_trace_opts__last_field bpf_cookie + LIBBPF_API struct bpf_link * bpf_program__attach_trace(const struct bpf_program *prog); +LIBBPF_API struct bpf_link * +bpf_program__attach_trace_opts(const struct bpf_program *prog, const struct bpf_trace_opts *opts); + LIBBPF_API struct bpf_link * bpf_program__attach_lsm(const struct bpf_program *prog); LIBBPF_API struct bpf_link * diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 82f6d62176dd..9235da802e31 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -424,6 +424,7 @@ LIBBPF_0.6.0 { LIBBPF_0.7.0 { global: bpf_btf_load; + bpf_program__attach_trace_opts; bpf_program__expected_attach_type; bpf_program__log_buf; bpf_program__log_level; -- 2.30.2 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf. 2022-04-12 16:55 ` [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf Kui-Feng Lee @ 2022-04-13 3:14 ` Andrii Nakryiko 0 siblings, 0 replies; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 3:14 UTC (permalink / raw) To: Kui-Feng Lee Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > Add a cookie field to the attributes of bpf_link_create(). > Add bpf_program__attach_trace_opts() to attach a cookie to a link. > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > --- Please stick to "libbpf: " prefix for patch subject, we don't use "lib/bpf: " > tools/lib/bpf/bpf.c | 7 +++++++ > tools/lib/bpf/bpf.h | 3 +++ > tools/lib/bpf/libbpf.c | 33 +++++++++++++++++++++++++++++++++ > tools/lib/bpf/libbpf.h | 12 ++++++++++++ > tools/lib/bpf/libbpf.map | 1 + > 5 files changed, 56 insertions(+) > > diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c > index cf27251adb92..c2454979c3c4 100644 > --- a/tools/lib/bpf/bpf.c > +++ b/tools/lib/bpf/bpf.c > @@ -863,6 +863,13 @@ int bpf_link_create(int prog_fd, int target_fd, > if (!OPTS_ZEROED(opts, kprobe_multi)) > return libbpf_err(-EINVAL); > break; > + case BPF_TRACE_FENTRY: > + case BPF_TRACE_FEXIT: > + case BPF_MODIFY_RETURN: > + attr.link_create.tracing.bpf_cookie = OPTS_GET(opts, tracing.bpf_cookie, 0); > + if (!OPTS_ZEROED(opts, tracing)) > + return libbpf_err(-EINVAL); > + break; > default: > if (!OPTS_ZEROED(opts, flags)) > return libbpf_err(-EINVAL); > diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h > index f4b4afb6d4ba..4cdbabcccefc 100644 > --- a/tools/lib/bpf/bpf.h > +++ b/tools/lib/bpf/bpf.h > @@ -410,6 +410,9 @@ struct bpf_link_create_opts { > __u32 iter_info_len; > __u32 target_btf_id; > union { > + struct { > + __u64 bpf_cookie; > + } tracing; > struct { > __u64 bpf_cookie; > } perf_event; > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c > index bf4f7ac54ebf..8586e1efd780 100644 > --- a/tools/lib/bpf/libbpf.c > +++ b/tools/lib/bpf/libbpf.c > @@ -11262,6 +11262,39 @@ struct bpf_link *bpf_program__attach_trace(const struct bpf_program *prog) > return bpf_program__attach_btf_id(prog); > } > > +struct bpf_link *bpf_program__attach_trace_opts(const struct bpf_program *prog, > + const struct bpf_trace_opts *opts) > +{ > + char errmsg[STRERR_BUFSIZE]; > + struct bpf_link *link; > + int prog_fd, pfd; > + unnecessary empty line, LIBBPF_OPTS is a variable declaration, so keep it together with other variables > + LIBBPF_OPTS(bpf_link_create_opts, link_opts); > + > + prog_fd = bpf_program__fd(prog); > + if (prog_fd < 0) { > + pr_warn("prog '%s': can't attach before loaded\n", prog->name); > + return libbpf_err_ptr(-EINVAL); > + } > + > + link = calloc(1, sizeof(*link)); > + if (!link) > + return libbpf_err_ptr(-ENOMEM); > + link->detach = &bpf_link__detach_fd; > + > + link_opts.tracing.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0); > + pfd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &link_opts); so this won't work on old kernels that don't support creating fentry/fexit links through LINK_CREATE. Let's for now check if bpf_cookie is zero, then redirect to bpf_program__attach_trace(). Ideally we should perform feature detection of ability to use LINK_CREATE to attach fentry and have a unified single bpf_program__attach_btf_id() implementation that will handle all that internally > + if (pfd < 0) { > + pfd = -errno; > + free(link); > + pr_warn("prog '%s': failed to attach: %s\n", > + prog->name, libbpf_strerror_r(pfd, errmsg, sizeof(errmsg))); > + return libbpf_err_ptr(pfd); > + } > + link->fd = pfd; > + return (struct bpf_link *)link; > +} > + > struct bpf_link *bpf_program__attach_lsm(const struct bpf_program *prog) > { > return bpf_program__attach_btf_id(prog); > diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h > index 63d66f1adf1a..e0dd3f9a5aca 100644 > --- a/tools/lib/bpf/libbpf.h > +++ b/tools/lib/bpf/libbpf.h > @@ -563,8 +563,20 @@ bpf_program__attach_tracepoint_opts(const struct bpf_program *prog, > LIBBPF_API struct bpf_link * > bpf_program__attach_raw_tracepoint(const struct bpf_program *prog, > const char *tp_name); > + > +struct bpf_trace_opts { > + /* size of this struct, for forward/backward compatibility */ > + size_t sz; > + /* custom user-provided value fetchable through bpf_get_attach_cookie() */ > + __u64 bpf_cookie; > +}; > +#define bpf_trace_opts__last_field bpf_cookie > + > LIBBPF_API struct bpf_link * > bpf_program__attach_trace(const struct bpf_program *prog); > +LIBBPF_API struct bpf_link * > +bpf_program__attach_trace_opts(const struct bpf_program *prog, const struct bpf_trace_opts *opts); > + > LIBBPF_API struct bpf_link * > bpf_program__attach_lsm(const struct bpf_program *prog); > LIBBPF_API struct bpf_link * > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map > index 82f6d62176dd..9235da802e31 100644 > --- a/tools/lib/bpf/libbpf.map > +++ b/tools/lib/bpf/libbpf.map > @@ -424,6 +424,7 @@ LIBBPF_0.6.0 { > LIBBPF_0.7.0 { > global: > bpf_btf_load; > + bpf_program__attach_trace_opts; this should go into LIBBPF_0.8.0 > bpf_program__expected_attach_type; > bpf_program__log_buf; > bpf_program__log_level; > -- > 2.30.2 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret. 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee ` (3 preceding siblings ...) 2022-04-12 16:55 ` [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf Kui-Feng Lee @ 2022-04-12 16:55 ` Kui-Feng Lee 2022-04-13 3:17 ` Andrii Nakryiko 4 siblings, 1 reply; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-12 16:55 UTC (permalink / raw) To: bpf, ast, daniel, andrii, kernel-team; +Cc: Kui-Feng Lee Make sure BPF cookies are correct for fentry/fexit/fmod_ret. Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> --- .../selftests/bpf/prog_tests/bpf_cookie.c | 52 +++++++++++++++++++ .../selftests/bpf/progs/test_bpf_cookie.c | 24 +++++++++ 2 files changed, 76 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c index 923a6139b2d8..7f05056c66d4 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c @@ -410,6 +410,56 @@ static void pe_subtest(struct test_bpf_cookie *skel) bpf_link__destroy(link); } +static void tracing_subtest(struct test_bpf_cookie *skel) +{ + __u64 cookie; + int prog_fd; + int fentry_fd = -1, fexit_fd = -1, fmod_ret_fd = -1; + + LIBBPF_OPTS(bpf_test_run_opts, opts, .repeat = 1); + LIBBPF_OPTS(bpf_link_create_opts, link_opts); + + skel->bss->fentry_res = 0; + skel->bss->fexit_res = 0; + + cookie = 0x100000; + prog_fd = bpf_program__fd(skel->progs.fentry_test1); + link_opts.tracing.bpf_cookie = cookie; + fentry_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &link_opts); + + cookie = 0x200000; + prog_fd = bpf_program__fd(skel->progs.fexit_test1); + link_opts.tracing.bpf_cookie = cookie; + fexit_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &link_opts); + if (!ASSERT_GE(fexit_fd, 0, "fexit.open")) + goto cleanup; + + cookie = 0x300000; + prog_fd = bpf_program__fd(skel->progs.fmod_ret_test); + link_opts.tracing.bpf_cookie = cookie; + fmod_ret_fd = bpf_link_create(prog_fd, 0, BPF_MODIFY_RETURN, &link_opts); + if (!ASSERT_GE(fmod_ret_fd, 0, "fmod_ret.opoen")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.fentry_test1); + bpf_prog_test_run_opts(prog_fd, &opts); + + prog_fd = bpf_program__fd(skel->progs.fmod_ret_test); + bpf_prog_test_run_opts(prog_fd, &opts); + + ASSERT_EQ(skel->bss->fentry_res, 0x100000, "fentry_res"); + ASSERT_EQ(skel->bss->fexit_res, 0x200000, "fexit_res"); + ASSERT_EQ(skel->bss->fmod_ret_res, 0x300000, "fmod_ret_res"); + +cleanup: + if (fentry_fd >= 0) + close(fentry_fd); + if (fexit_fd >= 0) + close(fexit_fd); + if (fmod_ret_fd >= 0) + close(fmod_ret_fd); +} + void test_bpf_cookie(void) { struct test_bpf_cookie *skel; @@ -432,6 +482,8 @@ void test_bpf_cookie(void) tp_subtest(skel); if (test__start_subtest("perf_event")) pe_subtest(skel); + if (test__start_subtest("trampoline")) + tracing_subtest(skel); test_bpf_cookie__destroy(skel); } diff --git a/tools/testing/selftests/bpf/progs/test_bpf_cookie.c b/tools/testing/selftests/bpf/progs/test_bpf_cookie.c index 0e2222968918..c37bc6ad0b70 100644 --- a/tools/testing/selftests/bpf/progs/test_bpf_cookie.c +++ b/tools/testing/selftests/bpf/progs/test_bpf_cookie.c @@ -14,6 +14,9 @@ int uprobe_res; int uretprobe_res; int tp_res; int pe_res; +int fentry_res; +int fexit_res; +int fmod_ret_res; static void update(void *ctx, int *res) { @@ -82,4 +85,25 @@ int handle_pe(struct pt_regs *ctx) return 0; } +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(fentry_test1, int a) +{ + update(ctx, &fentry_res); + return 0; +} + +SEC("fexit/bpf_fentry_test1") +int BPF_PROG(fexit_test1, int a, int ret) +{ + update(ctx, &fexit_res); + return 0; +} + +SEC("fmod_ret/bpf_modify_return_test") +int BPF_PROG(fmod_ret_test, int _a, int *_b, int _ret) +{ + update(ctx, &fmod_ret_res); + return 1234; +} + char _license[] SEC("license") = "GPL"; -- 2.30.2 ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret. 2022-04-12 16:55 ` [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret Kui-Feng Lee @ 2022-04-13 3:17 ` Andrii Nakryiko 2022-04-13 18:30 ` Kui-Feng Lee 0 siblings, 1 reply; 17+ messages in thread From: Andrii Nakryiko @ 2022-04-13 3:17 UTC (permalink / raw) To: Kui-Feng Lee Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > Make sure BPF cookies are correct for fentry/fexit/fmod_ret. > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > --- > .../selftests/bpf/prog_tests/bpf_cookie.c | 52 +++++++++++++++++++ > .../selftests/bpf/progs/test_bpf_cookie.c | 24 +++++++++ > 2 files changed, 76 insertions(+) > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > index 923a6139b2d8..7f05056c66d4 100644 > --- a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > @@ -410,6 +410,56 @@ static void pe_subtest(struct test_bpf_cookie *skel) > bpf_link__destroy(link); > } > > +static void tracing_subtest(struct test_bpf_cookie *skel) > +{ > + __u64 cookie; > + int prog_fd; > + int fentry_fd = -1, fexit_fd = -1, fmod_ret_fd = -1; > + unnecessary empty line > + LIBBPF_OPTS(bpf_test_run_opts, opts, .repeat = 1); .repeat = 1 is not necessary, I think, .repeat = 0 is equivalent to that > + LIBBPF_OPTS(bpf_link_create_opts, link_opts); > + > + skel->bss->fentry_res = 0; > + skel->bss->fexit_res = 0; > + > + cookie = 0x100000; nit: make this value bigger to make sure higher 32 bits of u64 are preserved properly. Maybe 0x1000000010000000 (and similarly with 2 and 3) > + prog_fd = bpf_program__fd(skel->progs.fentry_test1); > + link_opts.tracing.bpf_cookie = cookie; > + fentry_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &link_opts); > + ASSERT_GE? > + cookie = 0x200000; > + prog_fd = bpf_program__fd(skel->progs.fexit_test1); > + link_opts.tracing.bpf_cookie = cookie; > + fexit_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &link_opts); > + if (!ASSERT_GE(fexit_fd, 0, "fexit.open")) > + goto cleanup; > + [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret. 2022-04-13 3:17 ` Andrii Nakryiko @ 2022-04-13 18:30 ` Kui-Feng Lee 0 siblings, 0 replies; 17+ messages in thread From: Kui-Feng Lee @ 2022-04-13 18:30 UTC (permalink / raw) To: andrii.nakryiko; +Cc: daniel, Kernel Team, ast, andrii, bpf On Tue, 2022-04-12 at 20:17 -0700, Andrii Nakryiko wrote: > On Tue, Apr 12, 2022 at 9:56 AM Kui-Feng Lee <kuifeng@fb.com> wrote: > > > > Make sure BPF cookies are correct for fentry/fexit/fmod_ret. > > > > Signed-off-by: Kui-Feng Lee <kuifeng@fb.com> > > --- > > .../selftests/bpf/prog_tests/bpf_cookie.c | 52 > > +++++++++++++++++++ > > .../selftests/bpf/progs/test_bpf_cookie.c | 24 +++++++++ > > 2 files changed, 76 insertions(+) > > > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > > b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > > index 923a6139b2d8..7f05056c66d4 100644 > > --- a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c > > @@ -410,6 +410,56 @@ static void pe_subtest(struct test_bpf_cookie > > *skel) > > bpf_link__destroy(link); > > } > > > > +static void tracing_subtest(struct test_bpf_cookie *skel) > > +{ > > + __u64 cookie; > > + int prog_fd; > > + int fentry_fd = -1, fexit_fd = -1, fmod_ret_fd = -1; > > + > > unnecessary empty line Got it! > > > + LIBBPF_OPTS(bpf_test_run_opts, opts, .repeat = 1); > > .repeat = 1 is not necessary, I think, .repeat = 0 is equivalent to > that I will test it. > > > + LIBBPF_OPTS(bpf_link_create_opts, link_opts); > > + > > + skel->bss->fentry_res = 0; > > + skel->bss->fexit_res = 0; > > + > > + cookie = 0x100000; > > nit: make this value bigger to make sure higher 32 bits of u64 are > preserved properly. Maybe 0x1000000010000000 (and similarly with 2 > and > 3) Ok! > > > + prog_fd = bpf_program__fd(skel->progs.fentry_test1); > > + link_opts.tracing.bpf_cookie = cookie; > > + fentry_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, > > &link_opts); > > + > > ASSERT_GE? sure! > > > + cookie = 0x200000; > > + prog_fd = bpf_program__fd(skel->progs.fexit_test1); > > + link_opts.tracing.bpf_cookie = cookie; > > + fexit_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, > > &link_opts); > > + if (!ASSERT_GE(fexit_fd, 0, "fexit.open")) > > + goto cleanup; > > + > > [...] ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2022-04-13 20:14 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-12 16:55 [PATCH bpf-next v5 0/5] Attach a cookie to a tracing program Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 1/5] bpf, x86: Generate trampolines from bpf_tramp_links Kui-Feng Lee 2022-04-13 2:43 ` Andrii Nakryiko 2022-04-13 17:53 ` Kui-Feng Lee 2022-04-13 19:08 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 2/5] bpf, x86: Create bpf_tramp_run_ctx on the caller thread's stack Kui-Feng Lee 2022-04-13 2:55 ` Andrii Nakryiko 2022-04-13 18:06 ` Kui-Feng Lee 2022-04-13 20:14 ` Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 3/5] bpf, x86: Attach a cookie to fentry/fexit/fmod_ret Kui-Feng Lee 2022-04-13 3:03 ` Andrii Nakryiko 2022-04-13 18:14 ` Kui-Feng Lee 2022-04-12 16:55 ` [PATCH bpf-next v5 4/5] lib/bpf: Assign cookies to links in libbpf Kui-Feng Lee 2022-04-13 3:14 ` Andrii Nakryiko 2022-04-12 16:55 ` [PATCH bpf-next v5 5/5] selftest/bpf: The test cses of BPF cookie for fentry/fexit/fmod_ret Kui-Feng Lee 2022-04-13 3:17 ` Andrii Nakryiko 2022-04-13 18:30 ` Kui-Feng Lee
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.