* [PATCH bpf-next 01/29] ftrace: Use direct_ops hash in unregister_ftrace_direct
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 02/29] ftrace: Add cleanup to unregister_ftrace_direct_multi Jiri Olsa
` (27 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: Steven Rostedt, netdev, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
Now when we have *direct_multi interface the direct_functions
hash is no longer owned just by direct_ops. It's also used by
any other ftrace_ops passed to *direct_multi interface.
Thus to find out that we are unregistering the last function
from direct_ops, we need to check directly direct_ops's hash.
Cc: Steven Rostedt <srostedt@vmware.com>
Fixes: f64dd4627ec6 ("ftrace: Add multi direct register/unregister interface")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
kernel/trace/ftrace.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 30bc880c3849..7f0594e28226 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5217,6 +5217,7 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr)
{
struct ftrace_direct_func *direct;
struct ftrace_func_entry *entry;
+ struct ftrace_hash *hash;
int ret = -ENODEV;
mutex_lock(&direct_mutex);
@@ -5225,7 +5226,8 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr)
if (!entry)
goto out_unlock;
- if (direct_functions->count == 1)
+ hash = direct_ops.func_hash->filter_hash;
+ if (hash->count == 1)
unregister_ftrace_function(&direct_ops);
ret = ftrace_set_filter_ip(&direct_ops, ip, 1, 0);
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 02/29] ftrace: Add cleanup to unregister_ftrace_direct_multi
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 01/29] ftrace: Use direct_ops hash in unregister_ftrace_direct Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 03/29] ftrace: Add ftrace_set_filter_ips function Jiri Olsa
` (26 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: Steven Rostedt, netdev, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
Adding ops cleanup to unregister_ftrace_direct_multi,
so it can be reused in another register call.
Cc: Steven Rostedt <srostedt@vmware.com>
Fixes: f64dd4627ec6 ("ftrace: Add multi direct register/unregister interface")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
kernel/trace/ftrace.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 7f0594e28226..be5f6b32a012 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5542,6 +5542,10 @@ int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)
err = unregister_ftrace_function(ops);
remove_direct_functions_hash(hash, addr);
mutex_unlock(&direct_mutex);
+
+ /* cleanup for possible another register call */
+ ops->func = NULL;
+ ops->trampoline = 0;
return err;
}
EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 03/29] ftrace: Add ftrace_set_filter_ips function
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 01/29] ftrace: Use direct_ops hash in unregister_ftrace_direct Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 02/29] ftrace: Add cleanup to unregister_ftrace_direct_multi Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 04/29] bpf: Factor bpf_check_attach_target function Jiri Olsa
` (25 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: Steven Rostedt, netdev, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
Adding ftrace_set_filter_ips function to be able to set filter on
multiple ip addresses at once.
With the *direct_multi interface we have cases where we need to
initialize ftrace_ops object with thousands of functions, so having
single function diving into ftrace_hash_move_and_update_ops with
ftrace_lock is better.
The functions ips are passed as unsigned ong array with count.
Cc: Steven Rostedt <srostedt@vmware.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 3 +++
kernel/trace/ftrace.c | 53 +++++++++++++++++++++++++++++++++++-------
2 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 9999e29187de..60847cbce0da 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -512,6 +512,8 @@ struct dyn_ftrace {
int ftrace_set_filter_ip(struct ftrace_ops *ops, unsigned long ip,
int remove, int reset);
+int ftrace_set_filter_ips(struct ftrace_ops *ops, unsigned long *ips,
+ unsigned int cnt, int remove, int reset);
int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf,
int len, int reset);
int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf,
@@ -802,6 +804,7 @@ static inline unsigned long ftrace_location(unsigned long ip)
#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; })
#define ftrace_set_early_filter(ops, buf, enable) do { } while (0)
#define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; })
+#define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; })
#define ftrace_set_filter(ops, buf, len, reset) ({ -ENODEV; })
#define ftrace_set_notrace(ops, buf, len, reset) ({ -ENODEV; })
#define ftrace_free_filter(ops) do { } while (0)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index be5f6b32a012..39350aa38649 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -4958,7 +4958,7 @@ ftrace_notrace_write(struct file *file, const char __user *ubuf,
}
static int
-ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove)
+__ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove)
{
struct ftrace_func_entry *entry;
@@ -4976,9 +4976,25 @@ ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove)
return add_hash_entry(hash, ip);
}
+static int
+ftrace_match_addr(struct ftrace_hash *hash, unsigned long *ips,
+ unsigned int cnt, int remove)
+{
+ unsigned int i;
+ int err;
+
+ for (i = 0; i < cnt; i++) {
+ err = __ftrace_match_addr(hash, ips[i], remove);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
static int
ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len,
- unsigned long ip, int remove, int reset, int enable)
+ unsigned long *ips, unsigned int cnt,
+ int remove, int reset, int enable)
{
struct ftrace_hash **orig_hash;
struct ftrace_hash *hash;
@@ -5008,8 +5024,8 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len,
ret = -EINVAL;
goto out_regex_unlock;
}
- if (ip) {
- ret = ftrace_match_addr(hash, ip, remove);
+ if (ips) {
+ ret = ftrace_match_addr(hash, ips, cnt, remove);
if (ret < 0)
goto out_regex_unlock;
}
@@ -5026,10 +5042,10 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len,
}
static int
-ftrace_set_addr(struct ftrace_ops *ops, unsigned long ip, int remove,
- int reset, int enable)
+ftrace_set_addr(struct ftrace_ops *ops, unsigned long *ips, unsigned int cnt,
+ int remove, int reset, int enable)
{
- return ftrace_set_hash(ops, NULL, 0, ip, remove, reset, enable);
+ return ftrace_set_hash(ops, NULL, 0, ips, cnt, remove, reset, enable);
}
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
@@ -5634,10 +5650,29 @@ int ftrace_set_filter_ip(struct ftrace_ops *ops, unsigned long ip,
int remove, int reset)
{
ftrace_ops_init(ops);
- return ftrace_set_addr(ops, ip, remove, reset, 1);
+ return ftrace_set_addr(ops, &ip, 1, remove, reset, 1);
}
EXPORT_SYMBOL_GPL(ftrace_set_filter_ip);
+/**
+ * ftrace_set_filter_ips - set a functions to filter on in ftrace by addresses
+ * @ops - the ops to set the filter with
+ * @ips - the array of addresses to add to or remove from the filter.
+ * @cnt - the number of addresses in @ips
+ * @remove - non zero to remove ips from the filter
+ * @reset - non zero to reset all filters before applying this filter.
+ *
+ * Filters denote which functions should be enabled when tracing is enabled
+ * If @ips array or any ip specified within is NULL , it fails to update filter.
+ */
+int ftrace_set_filter_ips(struct ftrace_ops *ops, unsigned long *ips,
+ unsigned int cnt, int remove, int reset)
+{
+ ftrace_ops_init(ops);
+ return ftrace_set_addr(ops, ips, cnt, remove, reset, 1);
+}
+EXPORT_SYMBOL_GPL(ftrace_set_filter_ips);
+
/**
* ftrace_ops_set_global_filter - setup ops to use global filters
* @ops - the ops which will use the global filters
@@ -5659,7 +5694,7 @@ static int
ftrace_set_regex(struct ftrace_ops *ops, unsigned char *buf, int len,
int reset, int enable)
{
- return ftrace_set_hash(ops, buf, len, 0, 0, reset, enable);
+ return ftrace_set_hash(ops, buf, len, NULL, 0, 0, reset, enable);
}
/**
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 04/29] bpf: Factor bpf_check_attach_target function
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (2 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 03/29] ftrace: Add ftrace_set_filter_ips function Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 05/29] bpf: Add bpf_check_attach_model function Jiri Olsa
` (24 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Separating the check itself from model distilling and
address search into __bpf_check_attach_target function.
This way we can easily add function in following patch
that gets only function model without the address search,
while using the same code as bpf_check_attach_target.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
kernel/bpf/verifier.c | 79 ++++++++++++++++++++++++++++++++-----------
1 file changed, 59 insertions(+), 20 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0763cca139a7..cbbbf47e1832 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13568,20 +13568,26 @@ static int check_non_sleepable_error_inject(u32 btf_id)
return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
}
-int bpf_check_attach_target(struct bpf_verifier_log *log,
- const struct bpf_prog *prog,
- const struct bpf_prog *tgt_prog,
- u32 btf_id,
- struct bpf_attach_target_info *tgt_info)
+struct attach_target {
+ const struct btf_type *t;
+ const char *tname;
+ int subprog;
+ struct btf *btf;
+};
+
+static int __bpf_check_attach_target(struct bpf_verifier_log *log,
+ const struct bpf_prog *prog,
+ const struct bpf_prog *tgt_prog,
+ u32 btf_id,
+ struct attach_target *target)
{
bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;
const char prefix[] = "btf_trace_";
- int ret = 0, subprog = -1, i;
+ int subprog = -1, i;
const struct btf_type *t;
bool conservative = true;
const char *tname;
struct btf *btf;
- long addr = 0;
if (!btf_id) {
bpf_log(log, "Tracing programs must provide btf_id\n");
@@ -13706,9 +13712,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_func_proto(t))
return -EINVAL;
- ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
- if (ret)
- return ret;
break;
default:
if (!prog_extension)
@@ -13737,22 +13740,57 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (tgt_prog && conservative)
t = NULL;
+ }
+
+ target->t = t;
+ target->tname = tname;
+ target->subprog = subprog;
+ target->btf = btf;
+ return 0;
+}
+
+int bpf_check_attach_target(struct bpf_verifier_log *log,
+ const struct bpf_prog *prog,
+ const struct bpf_prog *tgt_prog,
+ u32 btf_id,
+ struct bpf_attach_target_info *tgt_info)
+{
+ struct attach_target target = { };
+ long addr = 0;
+ int ret;
- ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
+ ret = __bpf_check_attach_target(log, prog, tgt_prog, btf_id, &target);
+ if (ret)
+ return ret;
+
+ switch (prog->expected_attach_type) {
+ case BPF_TRACE_RAW_TP:
+ break;
+ case BPF_TRACE_ITER:
+ ret = btf_distill_func_proto(log, target.btf, target.t, target.tname, &tgt_info->fmodel);
+ if (ret)
+ return ret;
+ break;
+ default:
+ case BPF_MODIFY_RETURN:
+ case BPF_LSM_MAC:
+ case BPF_TRACE_FENTRY:
+ case BPF_TRACE_FEXIT:
+ ret = btf_distill_func_proto(log, target.btf, target.t, target.tname, &tgt_info->fmodel);
if (ret < 0)
return ret;
if (tgt_prog) {
- if (subprog == 0)
+ if (target.subprog == 0)
addr = (long) tgt_prog->bpf_func;
else
- addr = (long) tgt_prog->aux->func[subprog]->bpf_func;
+ addr = (long) tgt_prog->aux->func[target.subprog]->bpf_func;
} else {
- addr = kallsyms_lookup_name(tname);
+ addr = kallsyms_lookup_name(target.tname);
if (!addr) {
bpf_log(log,
"The address of function %s cannot be found\n",
- tname);
+ target.tname);
return -ENOENT;
}
}
@@ -13779,7 +13817,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
break;
}
if (ret) {
- bpf_log(log, "%s is not sleepable\n", tname);
+ bpf_log(log, "%s is not sleepable\n", target.tname);
return ret;
}
} else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
@@ -13787,18 +13825,19 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
bpf_log(log, "can't modify return codes of BPF programs\n");
return -EINVAL;
}
- ret = check_attach_modify_return(addr, tname);
+ ret = check_attach_modify_return(addr, target.tname);
if (ret) {
- bpf_log(log, "%s() is not modifiable\n", tname);
+ bpf_log(log, "%s() is not modifiable\n", target.tname);
return ret;
}
}
break;
}
+
tgt_info->tgt_addr = addr;
- tgt_info->tgt_name = tname;
- tgt_info->tgt_type = t;
+ tgt_info->tgt_name = target.tname;
+ tgt_info->tgt_type = target.t;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 05/29] bpf: Add bpf_check_attach_model function
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (3 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 04/29] bpf: Factor bpf_check_attach_target function Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs Jiri Olsa
` (23 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding bpf_check_attach_model function that returns
model for function specified by btf_id. It will be
used in following patches.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf_verifier.h | 4 ++++
kernel/bpf/verifier.c | 29 +++++++++++++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c8a78e830fca..b561c0b08e68 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -527,6 +527,10 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *tgt_prog,
u32 btf_id,
struct bpf_attach_target_info *tgt_info);
+int bpf_check_attach_model(const struct bpf_prog *prog,
+ const struct bpf_prog *tgt_prog,
+ u32 btf_id,
+ struct btf_func_model *fmodel);
void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index cbbbf47e1832..fac0c3518add 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13749,6 +13749,35 @@ static int __bpf_check_attach_target(struct bpf_verifier_log *log,
return 0;
}
+int bpf_check_attach_model(const struct bpf_prog *prog,
+ const struct bpf_prog *tgt_prog,
+ u32 btf_id,
+ struct btf_func_model *fmodel)
+{
+ struct attach_target target = { };
+ int ret;
+
+ ret = __bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, &target);
+ if (ret)
+ return ret;
+
+ switch (prog->expected_attach_type) {
+ case BPF_TRACE_RAW_TP:
+ break;
+ case BPF_TRACE_ITER:
+ ret = btf_distill_func_proto(NULL, target.btf, target.t, target.tname, fmodel);
+ break;
+ default:
+ case BPF_MODIFY_RETURN:
+ case BPF_LSM_MAC:
+ case BPF_TRACE_FENTRY:
+ case BPF_TRACE_FEXIT:
+ ret = btf_distill_func_proto(NULL, target.btf, target.t, target.tname, fmodel);
+ }
+
+ return ret;
+}
+
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (4 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 05/29] bpf: Add bpf_check_attach_model function Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-24 21:43 ` Andrii Nakryiko
2021-11-18 11:24 ` [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack Jiri Olsa
` (22 subsequent siblings)
28 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding bpf_arg/bpf_ret_value helpers for tracing programs
that returns traced function arguments.
Get n-th argument of the traced function:
long bpf_arg(void *ctx, int n)
Get return value of the traced function:
long bpf_ret_value(void *ctx)
The trampoline now stores number of arguments on ctx-8
address, so it's easy to verify argument index and find
return value argument.
Moving function ip address on the trampoline stack behind
the number of functions arguments, so it's now stored
on ctx-16 address.
Both helpers are inlined by verifier.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 18 +++++++++++---
include/uapi/linux/bpf.h | 14 +++++++++++
kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++--
kernel/trace/bpf_trace.c | 38 +++++++++++++++++++++++++++-
tools/include/uapi/linux/bpf.h | 14 +++++++++++
5 files changed, 122 insertions(+), 7 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 631847907786..67e8ac9aaf0d 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1941,7 +1941,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
void *orig_call)
{
int ret, i, nr_args = m->nr_args;
- int stack_size = nr_args * 8;
+ int stack_size = nr_args * 8 + 8 /* nr_args */;
struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY];
struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT];
struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN];
@@ -1987,12 +1987,22 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
EMIT4(0x48, 0x83, 0xe8, X86_PATCH_SIZE);
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
- /* Continue with stack_size for regs storage, stack will
- * be correctly restored with 'leave' instruction.
- */
+ /* Continue with stack_size for 'nr_args' storage */
stack_size -= 8;
}
+ /* Store number of arguments of the traced function:
+ * mov rax, nr_args
+ * mov QWORD PTR [rbp - stack_size], rax
+ */
+ emit_mov_imm64(&prog, BPF_REG_0, 0, (u32) nr_args);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
+
+ /* Continue with stack_size for regs storage, stack will
+ * be correctly restored with 'leave' instruction.
+ */
+ stack_size -= 8;
+
save_regs(m, &prog, nr_args, stack_size);
if (flags & BPF_TRAMP_F_CALL_ORIG) {
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index a69e4b04ffeb..fc8b344eecba 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -4957,6 +4957,18 @@ union bpf_attr {
* **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
* **-EBUSY** if failed to try lock mmap_lock.
* **-EINVAL** for invalid **flags**.
+ *
+ * long bpf_arg(void *ctx, int n)
+ * Description
+ * Get n-th argument of the traced function (for tracing programs).
+ * Return
+ * Value of the argument.
+ *
+ * long bpf_ret_value(void *ctx)
+ * Description
+ * Get return value of the traced function (for tracing programs).
+ * Return
+ * Return value of the traced function.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -5140,6 +5152,8 @@ union bpf_attr {
FN(skc_to_unix_sock), \
FN(kallsyms_lookup_name), \
FN(find_vma), \
+ FN(arg), \
+ FN(ret_value), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fac0c3518add..d4249ef6ca7e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13246,11 +13246,52 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
continue;
}
+ /* Implement bpf_arg inline. */
+ if (prog_type == BPF_PROG_TYPE_TRACING &&
+ insn->imm == BPF_FUNC_arg) {
+ /* Load nr_args from ctx - 8 */
+ insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
+ insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
+ insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
+ insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
+ insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
+ insn_buf[5] = BPF_JMP_A(1);
+ insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
+
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
+ if (!new_prog)
+ return -ENOMEM;
+
+ delta += 6;
+ env->prog = prog = new_prog;
+ insn = new_prog->insnsi + i + delta;
+ continue;
+ }
+
+ /* Implement bpf_ret_value inline. */
+ if (prog_type == BPF_PROG_TYPE_TRACING &&
+ insn->imm == BPF_FUNC_ret_value) {
+ /* Load nr_args from ctx - 8 */
+ insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -8);
+ insn_buf[1] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
+ insn_buf[2] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
+ insn_buf[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
+
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 4);
+ if (!new_prog)
+ return -ENOMEM;
+
+ delta += 3;
+ env->prog = prog = new_prog;
+ insn = new_prog->insnsi + i + delta;
+ continue;
+ }
+
/* Implement bpf_get_func_ip inline. */
if (prog_type == BPF_PROG_TYPE_TRACING &&
insn->imm == BPF_FUNC_get_func_ip) {
- /* Load IP address from ctx - 8 */
- insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
+ /* Load IP address from ctx - 16 */
+ insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -16);
new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 1);
if (!new_prog)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 25ea521fb8f1..3844cfb45490 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1012,7 +1012,7 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = {
BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
{
/* This helper call is inlined by verifier. */
- return ((u64 *)ctx)[-1];
+ return ((u64 *)ctx)[-2];
}
static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
@@ -1091,6 +1091,38 @@ static const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
};
+BPF_CALL_2(bpf_arg, void *, ctx, int, n)
+{
+ /* This helper call is inlined by verifier. */
+ u64 nr_args = ((u64 *)ctx)[-1];
+
+ if ((u64) n >= nr_args)
+ return 0;
+ return ((u64 *)ctx)[n];
+}
+
+static const struct bpf_func_proto bpf_arg_proto = {
+ .func = bpf_arg,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+ .arg1_type = ARG_ANYTHING,
+};
+
+BPF_CALL_1(bpf_ret_value, void *, ctx)
+{
+ /* This helper call is inlined by verifier. */
+ u64 nr_args = ((u64 *)ctx)[-1];
+
+ return ((u64 *)ctx)[nr_args];
+}
+
+static const struct bpf_func_proto bpf_ret_value_proto = {
+ .func = bpf_ret_value,
+ .gpl_only = true,
+ .ret_type = RET_INTEGER,
+};
+
static const struct bpf_func_proto *
bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
@@ -1212,6 +1244,10 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return &bpf_find_vma_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
+ case BPF_FUNC_arg:
+ return &bpf_arg_proto;
+ case BPF_FUNC_ret_value:
+ return &bpf_ret_value_proto;
default:
return bpf_base_func_proto(func_id);
}
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index a69e4b04ffeb..fc8b344eecba 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -4957,6 +4957,18 @@ union bpf_attr {
* **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
* **-EBUSY** if failed to try lock mmap_lock.
* **-EINVAL** for invalid **flags**.
+ *
+ * long bpf_arg(void *ctx, int n)
+ * Description
+ * Get n-th argument of the traced function (for tracing programs).
+ * Return
+ * Value of the argument.
+ *
+ * long bpf_ret_value(void *ctx)
+ * Description
+ * Get return value of the traced function (for tracing programs).
+ * Return
+ * Return value of the traced function.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -5140,6 +5152,8 @@ union bpf_attr {
FN(skc_to_unix_sock), \
FN(kallsyms_lookup_name), \
FN(find_vma), \
+ FN(arg), \
+ FN(ret_value), \
/* */
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-18 11:24 ` [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs Jiri Olsa
@ 2021-11-24 21:43 ` Andrii Nakryiko
2021-11-25 16:14 ` Alexei Starovoitov
2021-11-28 18:06 ` Jiri Olsa
0 siblings, 2 replies; 49+ messages in thread
From: Andrii Nakryiko @ 2021-11-24 21:43 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Networking,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 3:25 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> Adding bpf_arg/bpf_ret_value helpers for tracing programs
> that returns traced function arguments.
>
> Get n-th argument of the traced function:
> long bpf_arg(void *ctx, int n)
>
> Get return value of the traced function:
> long bpf_ret_value(void *ctx)
>
> The trampoline now stores number of arguments on ctx-8
> address, so it's easy to verify argument index and find
> return value argument.
>
> Moving function ip address on the trampoline stack behind
> the number of functions arguments, so it's now stored
> on ctx-16 address.
>
> Both helpers are inlined by verifier.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
It would be great to land these changes separate from your huge patch
set. There are some upcoming BPF trampoline related changes that will
touch this (to add BPF cookie support for fentry/fexit progs), so
would be nice to minimize the interdependencies. So maybe post this
patch separately (probably after holidays ;) ).
> arch/x86/net/bpf_jit_comp.c | 18 +++++++++++---
> include/uapi/linux/bpf.h | 14 +++++++++++
> kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++--
> kernel/trace/bpf_trace.c | 38 +++++++++++++++++++++++++++-
> tools/include/uapi/linux/bpf.h | 14 +++++++++++
> 5 files changed, 122 insertions(+), 7 deletions(-)
>
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 631847907786..67e8ac9aaf0d 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -1941,7 +1941,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> void *orig_call)
> {
> int ret, i, nr_args = m->nr_args;
> - int stack_size = nr_args * 8;
> + int stack_size = nr_args * 8 + 8 /* nr_args */;
this /* nr_args */ next to 8 is super confusing, would be better to
expand the comment; might be a good idea to have some sort of a
description of possible stack layouts (e.g., fexit has some extra
stuff on the stack, I think, but it's impossible to remember and need
to recover that knowledge from the assembly code, basically).
> struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY];
> struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT];
> struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN];
> @@ -1987,12 +1987,22 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> EMIT4(0x48, 0x83, 0xe8, X86_PATCH_SIZE);
> emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
>
> - /* Continue with stack_size for regs storage, stack will
> - * be correctly restored with 'leave' instruction.
> - */
> + /* Continue with stack_size for 'nr_args' storage */
same, I don't think this comment really helps, just confuses some more
> stack_size -= 8;
> }
>
> + /* Store number of arguments of the traced function:
> + * mov rax, nr_args
> + * mov QWORD PTR [rbp - stack_size], rax
> + */
> + emit_mov_imm64(&prog, BPF_REG_0, 0, (u32) nr_args);
> + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
> +
> + /* Continue with stack_size for regs storage, stack will
> + * be correctly restored with 'leave' instruction.
> + */
> + stack_size -= 8;
I think "stack_size" as a name outlived itself and it just makes
everything harder to understand. It's used more like a stack offset
(relative to rsp or rbp) for different things. Would it make code
worse if we had few offset variables instead (or rather in addition,
we still need to calculate a full stack_size; it's just it's constant
re-adjustment is what's hard to keep track of), like regs_off,
ret_ip_off, arg_cnt_off, etc?
> +
> save_regs(m, &prog, nr_args, stack_size);
>
> if (flags & BPF_TRAMP_F_CALL_ORIG) {
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index a69e4b04ffeb..fc8b344eecba 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -4957,6 +4957,18 @@ union bpf_attr {
> * **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
> * **-EBUSY** if failed to try lock mmap_lock.
> * **-EINVAL** for invalid **flags**.
> + *
> + * long bpf_arg(void *ctx, int n)
__u32 n ?
> + * Description
> + * Get n-th argument of the traced function (for tracing programs).
> + * Return
> + * Value of the argument.
What about errors? those need to be documented.
> + *
> + * long bpf_ret_value(void *ctx)
> + * Description
> + * Get return value of the traced function (for tracing programs).
> + * Return
> + * Return value of the traced function.
Same, errors not documented. Also would be good to document what
happens when ret_value is requested in the context where there is no
ret value (e.g., fentry)
> */
> #define __BPF_FUNC_MAPPER(FN) \
> FN(unspec), \
> @@ -5140,6 +5152,8 @@ union bpf_attr {
> FN(skc_to_unix_sock), \
> FN(kallsyms_lookup_name), \
> FN(find_vma), \
> + FN(arg), \
> + FN(ret_value), \
We already have bpf_get_func_ip, so why not continue a tradition and
call these bpf_get_func_arg() and bpf_get_func_ret(). Nice, short,
clean, consistent.
BTW, a wild thought. Wouldn't it be cool to have these functions work
with kprobe/kretprobe as well? Do you think it's possible?
> /* */
>
> /* integer value in 'imm' field of BPF_CALL instruction selects which helper
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index fac0c3518add..d4249ef6ca7e 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -13246,11 +13246,52 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
> continue;
> }
>
> + /* Implement bpf_arg inline. */
> + if (prog_type == BPF_PROG_TYPE_TRACING &&
> + insn->imm == BPF_FUNC_arg) {
> + /* Load nr_args from ctx - 8 */
> + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> + insn_buf[5] = BPF_JMP_A(1);
> + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
> +
> + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
> + if (!new_prog)
> + return -ENOMEM;
> +
> + delta += 6;
> + env->prog = prog = new_prog;
> + insn = new_prog->insnsi + i + delta;
> + continue;
nit: this whole sequence of steps and calculations seems like
something that might be abstracted and hidden behind a macro or helper
func? Not related to your change, though. But wouldn't it be easier to
understand if it was just written as:
PATCH_INSNS(
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
BPF_JMP_A(1);
BPF_MOV64_IMM(BPF_REG_0, 0));
continue;
?
> + }
> +
> + /* Implement bpf_ret_value inline. */
> + if (prog_type == BPF_PROG_TYPE_TRACING &&
> + insn->imm == BPF_FUNC_ret_value) {
> + /* Load nr_args from ctx - 8 */
> + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -8);
> + insn_buf[1] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> + insn_buf[2] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> + insn_buf[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> +
> + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 4);
> + if (!new_prog)
> + return -ENOMEM;
> +
> + delta += 3;
> + env->prog = prog = new_prog;
> + insn = new_prog->insnsi + i + delta;
> + continue;
> + }
> +
> /* Implement bpf_get_func_ip inline. */
> if (prog_type == BPF_PROG_TYPE_TRACING &&
> insn->imm == BPF_FUNC_get_func_ip) {
> - /* Load IP address from ctx - 8 */
> - insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> + /* Load IP address from ctx - 16 */
> + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -16);
>
> new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 1);
> if (!new_prog)
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index 25ea521fb8f1..3844cfb45490 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -1012,7 +1012,7 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = {
> BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
> {
> /* This helper call is inlined by verifier. */
> - return ((u64 *)ctx)[-1];
> + return ((u64 *)ctx)[-2];
> }
>
> static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
> @@ -1091,6 +1091,38 @@ static const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
> .arg2_type = ARG_CONST_SIZE_OR_ZERO,
> };
>
> +BPF_CALL_2(bpf_arg, void *, ctx, int, n)
> +{
> + /* This helper call is inlined by verifier. */
> + u64 nr_args = ((u64 *)ctx)[-1];
> +
> + if ((u64) n >= nr_args)
> + return 0;
We'll need bpf_get_func_arg_cnt() helper as well to be able to know
the actual number of arguments traced function has. It's impossible to
know whether the argument is zero or there is no argument, otherwise.
> + return ((u64 *)ctx)[n];
> +}
> +
> +static const struct bpf_func_proto bpf_arg_proto = {
> + .func = bpf_arg,
> + .gpl_only = true,
> + .ret_type = RET_INTEGER,
> + .arg1_type = ARG_PTR_TO_CTX,
> + .arg1_type = ARG_ANYTHING,
> +};
> +
> +BPF_CALL_1(bpf_ret_value, void *, ctx)
> +{
> + /* This helper call is inlined by verifier. */
> + u64 nr_args = ((u64 *)ctx)[-1];
> +
> + return ((u64 *)ctx)[nr_args];
we should return 0 for fentry or disable this helper for anything but
fexit? It's going to return garbage otherwise.
> +}
> +
> +static const struct bpf_func_proto bpf_ret_value_proto = {
> + .func = bpf_ret_value,
> + .gpl_only = true,
> + .ret_type = RET_INTEGER,
> +};
> +
[...]
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-24 21:43 ` Andrii Nakryiko
@ 2021-11-25 16:14 ` Alexei Starovoitov
2021-11-28 18:07 ` Jiri Olsa
2021-11-28 18:06 ` Jiri Olsa
1 sibling, 1 reply; 49+ messages in thread
From: Alexei Starovoitov @ 2021-11-25 16:14 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Jiri Olsa, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Networking, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
On Wed, Nov 24, 2021 at 2:43 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
> > + /* Implement bpf_arg inline. */
> > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > + insn->imm == BPF_FUNC_arg) {
> > + /* Load nr_args from ctx - 8 */
> > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > + insn_buf[5] = BPF_JMP_A(1);
> > + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
> > +
> > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
> > + if (!new_prog)
> > + return -ENOMEM;
> > +
> > + delta += 6;
> > + env->prog = prog = new_prog;
> > + insn = new_prog->insnsi + i + delta;
> > + continue;
>
> nit: this whole sequence of steps and calculations seems like
> something that might be abstracted and hidden behind a macro or helper
> func? Not related to your change, though. But wouldn't it be easier to
> understand if it was just written as:
>
> PATCH_INSNS(
> BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> BPF_JMP_A(1);
> BPF_MOV64_IMM(BPF_REG_0, 0));
Daniel and myself tried to do similar macro magic in the past,
but it suffers unnecessary stack increase and extra copies.
So eventually we got rid of it.
I suggest staying with Jiri's approach.
Independent from anything else...
Just noticed BPF_MUL in the above...
Please use BPF_LSH instead. JITs don't optimize such things.
It's a job of gcc/llvm to do so. JITs assume that all
normal optimizations were done by the compiler.
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-25 16:14 ` Alexei Starovoitov
@ 2021-11-28 18:07 ` Jiri Olsa
0 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-28 18:07 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Andrii Nakryiko, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Thu, Nov 25, 2021 at 09:14:15AM -0700, Alexei Starovoitov wrote:
> On Wed, Nov 24, 2021 at 2:43 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> > > + /* Implement bpf_arg inline. */
> > > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > > + insn->imm == BPF_FUNC_arg) {
> > > + /* Load nr_args from ctx - 8 */
> > > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > > + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > > + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > > + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > > + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > > + insn_buf[5] = BPF_JMP_A(1);
> > > + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
> > > +
> > > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
> > > + if (!new_prog)
> > > + return -ENOMEM;
> > > +
> > > + delta += 6;
> > > + env->prog = prog = new_prog;
> > > + insn = new_prog->insnsi + i + delta;
> > > + continue;
> >
> > nit: this whole sequence of steps and calculations seems like
> > something that might be abstracted and hidden behind a macro or helper
> > func? Not related to your change, though. But wouldn't it be easier to
> > understand if it was just written as:
> >
> > PATCH_INSNS(
> > BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > BPF_JMP_A(1);
> > BPF_MOV64_IMM(BPF_REG_0, 0));
>
> Daniel and myself tried to do similar macro magic in the past,
> but it suffers unnecessary stack increase and extra copies.
> So eventually we got rid of it.
> I suggest staying with Jiri's approach.
>
> Independent from anything else...
> Just noticed BPF_MUL in the above...
> Please use BPF_LSH instead. JITs don't optimize such things.
> It's a job of gcc/llvm to do so. JITs assume that all
> normal optimizations were done by the compiler.
>
will change
thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-24 21:43 ` Andrii Nakryiko
2021-11-25 16:14 ` Alexei Starovoitov
@ 2021-11-28 18:06 ` Jiri Olsa
2021-12-01 7:13 ` Andrii Nakryiko
1 sibling, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-28 18:06 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Networking,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Wed, Nov 24, 2021 at 01:43:22PM -0800, Andrii Nakryiko wrote:
> On Thu, Nov 18, 2021 at 3:25 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > Adding bpf_arg/bpf_ret_value helpers for tracing programs
> > that returns traced function arguments.
> >
> > Get n-th argument of the traced function:
> > long bpf_arg(void *ctx, int n)
> >
> > Get return value of the traced function:
> > long bpf_ret_value(void *ctx)
> >
> > The trampoline now stores number of arguments on ctx-8
> > address, so it's easy to verify argument index and find
> > return value argument.
> >
> > Moving function ip address on the trampoline stack behind
> > the number of functions arguments, so it's now stored
> > on ctx-16 address.
> >
> > Both helpers are inlined by verifier.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
>
> It would be great to land these changes separate from your huge patch
> set. There are some upcoming BPF trampoline related changes that will
> touch this (to add BPF cookie support for fentry/fexit progs), so
> would be nice to minimize the interdependencies. So maybe post this
> patch separately (probably after holidays ;) ).
ok
>
> > arch/x86/net/bpf_jit_comp.c | 18 +++++++++++---
> > include/uapi/linux/bpf.h | 14 +++++++++++
> > kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++--
> > kernel/trace/bpf_trace.c | 38 +++++++++++++++++++++++++++-
> > tools/include/uapi/linux/bpf.h | 14 +++++++++++
> > 5 files changed, 122 insertions(+), 7 deletions(-)
> >
> > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> > index 631847907786..67e8ac9aaf0d 100644
> > --- a/arch/x86/net/bpf_jit_comp.c
> > +++ b/arch/x86/net/bpf_jit_comp.c
> > @@ -1941,7 +1941,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> > void *orig_call)
> > {
> > int ret, i, nr_args = m->nr_args;
> > - int stack_size = nr_args * 8;
> > + int stack_size = nr_args * 8 + 8 /* nr_args */;
>
> this /* nr_args */ next to 8 is super confusing, would be better to
> expand the comment; might be a good idea to have some sort of a
> description of possible stack layouts (e.g., fexit has some extra
> stuff on the stack, I think, but it's impossible to remember and need
> to recover that knowledge from the assembly code, basically).
>
> > struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY];
> > struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT];
> > struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN];
> > @@ -1987,12 +1987,22 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> > EMIT4(0x48, 0x83, 0xe8, X86_PATCH_SIZE);
> > emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
> >
> > - /* Continue with stack_size for regs storage, stack will
> > - * be correctly restored with 'leave' instruction.
> > - */
> > + /* Continue with stack_size for 'nr_args' storage */
>
> same, I don't think this comment really helps, just confuses some more
ok, I'll put some more comments with list of possible the stack layouts
>
> > stack_size -= 8;
> > }
> >
> > + /* Store number of arguments of the traced function:
> > + * mov rax, nr_args
> > + * mov QWORD PTR [rbp - stack_size], rax
> > + */
> > + emit_mov_imm64(&prog, BPF_REG_0, 0, (u32) nr_args);
> > + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
> > +
> > + /* Continue with stack_size for regs storage, stack will
> > + * be correctly restored with 'leave' instruction.
> > + */
> > + stack_size -= 8;
>
> I think "stack_size" as a name outlived itself and it just makes
> everything harder to understand. It's used more like a stack offset
> (relative to rsp or rbp) for different things. Would it make code
> worse if we had few offset variables instead (or rather in addition,
> we still need to calculate a full stack_size; it's just it's constant
> re-adjustment is what's hard to keep track of), like regs_off,
> ret_ip_off, arg_cnt_off, etc?
let's see, I'll try that
>
> > +
> > save_regs(m, &prog, nr_args, stack_size);
> >
> > if (flags & BPF_TRAMP_F_CALL_ORIG) {
> > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> > index a69e4b04ffeb..fc8b344eecba 100644
> > --- a/include/uapi/linux/bpf.h
> > +++ b/include/uapi/linux/bpf.h
> > @@ -4957,6 +4957,18 @@ union bpf_attr {
> > * **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
> > * **-EBUSY** if failed to try lock mmap_lock.
> > * **-EINVAL** for invalid **flags**.
> > + *
> > + * long bpf_arg(void *ctx, int n)
>
> __u32 n ?
ok
>
> > + * Description
> > + * Get n-th argument of the traced function (for tracing programs).
> > + * Return
> > + * Value of the argument.
>
> What about errors? those need to be documented.
ok
>
> > + *
> > + * long bpf_ret_value(void *ctx)
> > + * Description
> > + * Get return value of the traced function (for tracing programs).
> > + * Return
> > + * Return value of the traced function.
>
> Same, errors not documented. Also would be good to document what
> happens when ret_value is requested in the context where there is no
> ret value (e.g., fentry)
ugh, that's not handled at the moment.. should we fail when
we see bpf_ret_value helper call in fentry program?
>
> > */
> > #define __BPF_FUNC_MAPPER(FN) \
> > FN(unspec), \
> > @@ -5140,6 +5152,8 @@ union bpf_attr {
> > FN(skc_to_unix_sock), \
> > FN(kallsyms_lookup_name), \
> > FN(find_vma), \
> > + FN(arg), \
> > + FN(ret_value), \
>
> We already have bpf_get_func_ip, so why not continue a tradition and
> call these bpf_get_func_arg() and bpf_get_func_ret(). Nice, short,
> clean, consistent.
ok
>
> BTW, a wild thought. Wouldn't it be cool to have these functions work
> with kprobe/kretprobe as well? Do you think it's possible?
right, bpf_get_func_ip already works for kprobes
struct kprobe could have the btf_func_model of the traced function,
so in case we trace function directly on the entry point we could
read arguments registers based on the btf_func_model
I'll check with Massami
>
> > /* */
> >
> > /* integer value in 'imm' field of BPF_CALL instruction selects which helper
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index fac0c3518add..d4249ef6ca7e 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> > @@ -13246,11 +13246,52 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
> > continue;
> > }
> >
> > + /* Implement bpf_arg inline. */
> > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > + insn->imm == BPF_FUNC_arg) {
> > + /* Load nr_args from ctx - 8 */
> > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > + insn_buf[5] = BPF_JMP_A(1);
> > + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
> > +
> > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
> > + if (!new_prog)
> > + return -ENOMEM;
> > +
> > + delta += 6;
> > + env->prog = prog = new_prog;
> > + insn = new_prog->insnsi + i + delta;
> > + continue;
>
> nit: this whole sequence of steps and calculations seems like
> something that might be abstracted and hidden behind a macro or helper
> func? Not related to your change, though. But wouldn't it be easier to
> understand if it was just written as:
>
> PATCH_INSNS(
> BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> BPF_JMP_A(1);
> BPF_MOV64_IMM(BPF_REG_0, 0));
> continue;
yep, looks better ;-) I'll check
>
> ?
>
>
> > + }
> > +
> > + /* Implement bpf_ret_value inline. */
> > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > + insn->imm == BPF_FUNC_ret_value) {
> > + /* Load nr_args from ctx - 8 */
> > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -8);
> > + insn_buf[1] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > + insn_buf[2] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > + insn_buf[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > +
> > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 4);
> > + if (!new_prog)
> > + return -ENOMEM;
> > +
> > + delta += 3;
> > + env->prog = prog = new_prog;
> > + insn = new_prog->insnsi + i + delta;
> > + continue;
> > + }
> > +
> > /* Implement bpf_get_func_ip inline. */
> > if (prog_type == BPF_PROG_TYPE_TRACING &&
> > insn->imm == BPF_FUNC_get_func_ip) {
> > - /* Load IP address from ctx - 8 */
> > - insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > + /* Load IP address from ctx - 16 */
> > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -16);
> >
> > new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 1);
> > if (!new_prog)
> > diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> > index 25ea521fb8f1..3844cfb45490 100644
> > --- a/kernel/trace/bpf_trace.c
> > +++ b/kernel/trace/bpf_trace.c
> > @@ -1012,7 +1012,7 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = {
> > BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
> > {
> > /* This helper call is inlined by verifier. */
> > - return ((u64 *)ctx)[-1];
> > + return ((u64 *)ctx)[-2];
> > }
> >
> > static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
> > @@ -1091,6 +1091,38 @@ static const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
> > .arg2_type = ARG_CONST_SIZE_OR_ZERO,
> > };
> >
> > +BPF_CALL_2(bpf_arg, void *, ctx, int, n)
> > +{
> > + /* This helper call is inlined by verifier. */
> > + u64 nr_args = ((u64 *)ctx)[-1];
> > +
> > + if ((u64) n >= nr_args)
> > + return 0;
>
> We'll need bpf_get_func_arg_cnt() helper as well to be able to know
> the actual number of arguments traced function has. It's impossible to
> know whether the argument is zero or there is no argument, otherwise.
my idea was that the program will call those helpers based
on get_func_ip with proper argument indexes
but with bpf_get_func_arg_cnt we could make a simple program
that would just print function with all its arguments easily, ok ;-)
>
> > + return ((u64 *)ctx)[n];
> > +}
> > +
> > +static const struct bpf_func_proto bpf_arg_proto = {
> > + .func = bpf_arg,
> > + .gpl_only = true,
> > + .ret_type = RET_INTEGER,
> > + .arg1_type = ARG_PTR_TO_CTX,
> > + .arg1_type = ARG_ANYTHING,
> > +};
> > +
> > +BPF_CALL_1(bpf_ret_value, void *, ctx)
> > +{
> > + /* This helper call is inlined by verifier. */
> > + u64 nr_args = ((u64 *)ctx)[-1];
> > +
> > + return ((u64 *)ctx)[nr_args];
>
> we should return 0 for fentry or disable this helper for anything but
> fexit? It's going to return garbage otherwise.
disabling seems like right choice to me
thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-11-28 18:06 ` Jiri Olsa
@ 2021-12-01 7:13 ` Andrii Nakryiko
2021-12-01 17:37 ` Alexei Starovoitov
0 siblings, 1 reply; 49+ messages in thread
From: Andrii Nakryiko @ 2021-12-01 7:13 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Networking,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Sun, Nov 28, 2021 at 10:06 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Wed, Nov 24, 2021 at 01:43:22PM -0800, Andrii Nakryiko wrote:
> > On Thu, Nov 18, 2021 at 3:25 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > >
> > > Adding bpf_arg/bpf_ret_value helpers for tracing programs
> > > that returns traced function arguments.
> > >
> > > Get n-th argument of the traced function:
> > > long bpf_arg(void *ctx, int n)
> > >
> > > Get return value of the traced function:
> > > long bpf_ret_value(void *ctx)
> > >
> > > The trampoline now stores number of arguments on ctx-8
> > > address, so it's easy to verify argument index and find
> > > return value argument.
> > >
> > > Moving function ip address on the trampoline stack behind
> > > the number of functions arguments, so it's now stored
> > > on ctx-16 address.
> > >
> > > Both helpers are inlined by verifier.
> > >
> > > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > > ---
> >
> > It would be great to land these changes separate from your huge patch
> > set. There are some upcoming BPF trampoline related changes that will
> > touch this (to add BPF cookie support for fentry/fexit progs), so
> > would be nice to minimize the interdependencies. So maybe post this
> > patch separately (probably after holidays ;) ).
>
> ok
>
> >
> > > arch/x86/net/bpf_jit_comp.c | 18 +++++++++++---
> > > include/uapi/linux/bpf.h | 14 +++++++++++
> > > kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++--
> > > kernel/trace/bpf_trace.c | 38 +++++++++++++++++++++++++++-
> > > tools/include/uapi/linux/bpf.h | 14 +++++++++++
> > > 5 files changed, 122 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> > > index 631847907786..67e8ac9aaf0d 100644
> > > --- a/arch/x86/net/bpf_jit_comp.c
> > > +++ b/arch/x86/net/bpf_jit_comp.c
> > > @@ -1941,7 +1941,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> > > void *orig_call)
> > > {
> > > int ret, i, nr_args = m->nr_args;
> > > - int stack_size = nr_args * 8;
> > > + int stack_size = nr_args * 8 + 8 /* nr_args */;
> >
> > this /* nr_args */ next to 8 is super confusing, would be better to
> > expand the comment; might be a good idea to have some sort of a
> > description of possible stack layouts (e.g., fexit has some extra
> > stuff on the stack, I think, but it's impossible to remember and need
> > to recover that knowledge from the assembly code, basically).
> >
> > > struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY];
> > > struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT];
> > > struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN];
> > > @@ -1987,12 +1987,22 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> > > EMIT4(0x48, 0x83, 0xe8, X86_PATCH_SIZE);
> > > emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
> > >
> > > - /* Continue with stack_size for regs storage, stack will
> > > - * be correctly restored with 'leave' instruction.
> > > - */
> > > + /* Continue with stack_size for 'nr_args' storage */
> >
> > same, I don't think this comment really helps, just confuses some more
>
> ok, I'll put some more comments with list of possible the stack layouts
>
> >
> > > stack_size -= 8;
> > > }
> > >
> > > + /* Store number of arguments of the traced function:
> > > + * mov rax, nr_args
> > > + * mov QWORD PTR [rbp - stack_size], rax
> > > + */
> > > + emit_mov_imm64(&prog, BPF_REG_0, 0, (u32) nr_args);
> > > + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size);
> > > +
> > > + /* Continue with stack_size for regs storage, stack will
> > > + * be correctly restored with 'leave' instruction.
> > > + */
> > > + stack_size -= 8;
> >
> > I think "stack_size" as a name outlived itself and it just makes
> > everything harder to understand. It's used more like a stack offset
> > (relative to rsp or rbp) for different things. Would it make code
> > worse if we had few offset variables instead (or rather in addition,
> > we still need to calculate a full stack_size; it's just it's constant
> > re-adjustment is what's hard to keep track of), like regs_off,
> > ret_ip_off, arg_cnt_off, etc?
>
> let's see, I'll try that
>
> >
> > > +
> > > save_regs(m, &prog, nr_args, stack_size);
> > >
> > > if (flags & BPF_TRAMP_F_CALL_ORIG) {
> > > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> > > index a69e4b04ffeb..fc8b344eecba 100644
> > > --- a/include/uapi/linux/bpf.h
> > > +++ b/include/uapi/linux/bpf.h
> > > @@ -4957,6 +4957,18 @@ union bpf_attr {
> > > * **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
> > > * **-EBUSY** if failed to try lock mmap_lock.
> > > * **-EINVAL** for invalid **flags**.
> > > + *
> > > + * long bpf_arg(void *ctx, int n)
> >
> > __u32 n ?
>
> ok
>
> >
> > > + * Description
> > > + * Get n-th argument of the traced function (for tracing programs).
> > > + * Return
> > > + * Value of the argument.
> >
> > What about errors? those need to be documented.
>
> ok
>
> >
> > > + *
> > > + * long bpf_ret_value(void *ctx)
> > > + * Description
> > > + * Get return value of the traced function (for tracing programs).
> > > + * Return
> > > + * Return value of the traced function.
> >
> > Same, errors not documented. Also would be good to document what
> > happens when ret_value is requested in the context where there is no
> > ret value (e.g., fentry)
>
> ugh, that's not handled at the moment.. should we fail when
> we see bpf_ret_value helper call in fentry program?
Well, two options, really. Either return zero or detect at
verification time and fail verifications. I find myself leaning
towards less restrictions at verification time, so I'd probably go
with runtime check and zero. This allows to have the same BPF
subprogram that can be called both from fentry/fexit with a proper
if() guard to not do anything with the result of bpf_ret_value (as one
example).
>
> >
> > > */
> > > #define __BPF_FUNC_MAPPER(FN) \
> > > FN(unspec), \
> > > @@ -5140,6 +5152,8 @@ union bpf_attr {
> > > FN(skc_to_unix_sock), \
> > > FN(kallsyms_lookup_name), \
> > > FN(find_vma), \
> > > + FN(arg), \
> > > + FN(ret_value), \
> >
> > We already have bpf_get_func_ip, so why not continue a tradition and
> > call these bpf_get_func_arg() and bpf_get_func_ret(). Nice, short,
> > clean, consistent.
>
> ok
>
> >
> > BTW, a wild thought. Wouldn't it be cool to have these functions work
> > with kprobe/kretprobe as well? Do you think it's possible?
>
> right, bpf_get_func_ip already works for kprobes
>
> struct kprobe could have the btf_func_model of the traced function,
> so in case we trace function directly on the entry point we could
> read arguments registers based on the btf_func_model
>
> I'll check with Massami
Hm... I'd actually try to keep kprobe BTF-free. We have fentry for
cases where BTF is present and the function is simple enough (like <=6
args, etc). Kprobe is an escape hatch mechanism when all the BTF
fanciness just gets in the way (retsnoop being a primary example from
my side). What I meant here was that bpf_get_arg(int n) would read
correct fields from pt_regs that map to first N arguments passed in
the registers. What we currently have with PT_REGS_PARM macros in
bpf_tracing.h, but with a proper unified BPF helper.
>
> >
> > > /* */
> > >
> > > /* integer value in 'imm' field of BPF_CALL instruction selects which helper
> > > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > > index fac0c3518add..d4249ef6ca7e 100644
> > > --- a/kernel/bpf/verifier.c
> > > +++ b/kernel/bpf/verifier.c
> > > @@ -13246,11 +13246,52 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
> > > continue;
> > > }
> > >
> > > + /* Implement bpf_arg inline. */
> > > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > > + insn->imm == BPF_FUNC_arg) {
> > > + /* Load nr_args from ctx - 8 */
> > > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > > + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > > + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > > + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > > + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > > + insn_buf[5] = BPF_JMP_A(1);
> > > + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0);
> > > +
> > > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7);
> > > + if (!new_prog)
> > > + return -ENOMEM;
> > > +
> > > + delta += 6;
> > > + env->prog = prog = new_prog;
> > > + insn = new_prog->insnsi + i + delta;
> > > + continue;
> >
> > nit: this whole sequence of steps and calculations seems like
> > something that might be abstracted and hidden behind a macro or helper
> > func? Not related to your change, though. But wouldn't it be easier to
> > understand if it was just written as:
> >
> > PATCH_INSNS(
> > BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4);
> > BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > BPF_JMP_A(1);
> > BPF_MOV64_IMM(BPF_REG_0, 0));
> > continue;
>
> yep, looks better ;-) I'll check
as Alexei mentioned, might not be possible, but if variadic
implementation turns out to be not too ugly, I think it might work.
Macro can assume that insn_buf and all the other variables are there,
so there shouldn't be any increase in stack size use, I think.
But this is just an item on a wishlist, so don't overstress about that.
>
> >
> > ?
> >
> >
> > > + }
> > > +
> > > + /* Implement bpf_ret_value inline. */
> > > + if (prog_type == BPF_PROG_TYPE_TRACING &&
> > > + insn->imm == BPF_FUNC_ret_value) {
> > > + /* Load nr_args from ctx - 8 */
> > > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -8);
> > > + insn_buf[1] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8);
> > > + insn_buf[2] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1);
> > > + insn_buf[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0);
> > > +
> > > + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 4);
> > > + if (!new_prog)
> > > + return -ENOMEM;
> > > +
> > > + delta += 3;
> > > + env->prog = prog = new_prog;
> > > + insn = new_prog->insnsi + i + delta;
> > > + continue;
> > > + }
> > > +
> > > /* Implement bpf_get_func_ip inline. */
> > > if (prog_type == BPF_PROG_TYPE_TRACING &&
> > > insn->imm == BPF_FUNC_get_func_ip) {
> > > - /* Load IP address from ctx - 8 */
> > > - insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);
> > > + /* Load IP address from ctx - 16 */
> > > + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -16);
> > >
> > > new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 1);
> > > if (!new_prog)
> > > diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> > > index 25ea521fb8f1..3844cfb45490 100644
> > > --- a/kernel/trace/bpf_trace.c
> > > +++ b/kernel/trace/bpf_trace.c
> > > @@ -1012,7 +1012,7 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = {
> > > BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
> > > {
> > > /* This helper call is inlined by verifier. */
> > > - return ((u64 *)ctx)[-1];
> > > + return ((u64 *)ctx)[-2];
> > > }
> > >
> > > static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
> > > @@ -1091,6 +1091,38 @@ static const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
> > > .arg2_type = ARG_CONST_SIZE_OR_ZERO,
> > > };
> > >
> > > +BPF_CALL_2(bpf_arg, void *, ctx, int, n)
> > > +{
> > > + /* This helper call is inlined by verifier. */
> > > + u64 nr_args = ((u64 *)ctx)[-1];
> > > +
> > > + if ((u64) n >= nr_args)
> > > + return 0;
> >
> > We'll need bpf_get_func_arg_cnt() helper as well to be able to know
> > the actual number of arguments traced function has. It's impossible to
> > know whether the argument is zero or there is no argument, otherwise.
>
> my idea was that the program will call those helpers based
> on get_func_ip with proper argument indexes
see my comments on multi-attach kprobes, get_func_ip() is nice, but
BPF cookies are often much better. So I wouldn't design everything
with the assumption that user always has to use hashmap +
get_func_ip().
>
> but with bpf_get_func_arg_cnt we could make a simple program
> that would just print function with all its arguments easily, ok ;-)
right, and many other more complicated functions that don't have to do
runtime ip lookups ;)
>
> >
> > > + return ((u64 *)ctx)[n];
> > > +}
> > > +
> > > +static const struct bpf_func_proto bpf_arg_proto = {
> > > + .func = bpf_arg,
> > > + .gpl_only = true,
> > > + .ret_type = RET_INTEGER,
> > > + .arg1_type = ARG_PTR_TO_CTX,
> > > + .arg1_type = ARG_ANYTHING,
> > > +};
> > > +
> > > +BPF_CALL_1(bpf_ret_value, void *, ctx)
> > > +{
> > > + /* This helper call is inlined by verifier. */
> > > + u64 nr_args = ((u64 *)ctx)[-1];
> > > +
> > > + return ((u64 *)ctx)[nr_args];
> >
> > we should return 0 for fentry or disable this helper for anything but
> > fexit? It's going to return garbage otherwise.
>
> disabling seems like right choice to me
>
well, see above. I think we should prefer statically disabling
something if it's harmful to enable otherwise, but for more
flexibility and less headache with "proving to BPF verifier", I lean
more and more towards runtime checks, if they are safe and not overly
expensive or complicated.
> thanks,
> jirka
>
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-12-01 7:13 ` Andrii Nakryiko
@ 2021-12-01 17:37 ` Alexei Starovoitov
2021-12-01 17:59 ` Andrii Nakryiko
0 siblings, 1 reply; 49+ messages in thread
From: Alexei Starovoitov @ 2021-12-01 17:37 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Jiri Olsa, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Networking, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
On Tue, Nov 30, 2021 at 11:13 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> Hm... I'd actually try to keep kprobe BTF-free. We have fentry for
> cases where BTF is present and the function is simple enough (like <=6
> args, etc). Kprobe is an escape hatch mechanism when all the BTF
> fanciness just gets in the way (retsnoop being a primary example from
> my side). What I meant here was that bpf_get_arg(int n) would read
> correct fields from pt_regs that map to first N arguments passed in
> the registers. What we currently have with PT_REGS_PARM macros in
> bpf_tracing.h, but with a proper unified BPF helper.
and these macros are arch specific.
which means that it won't be a trivial patch to add bpf_get_arg()
support for kprobes.
Plenty of things to consider. Like should it return an error
at run-time or verification time when a particular arch is not supported.
Or argument 6 might be available on one arch, but not on the other.
32-bit CPU regs vs 64-bit regs of BPF, etc.
I wouldn't attempt to mix this work with current patches.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-12-01 17:37 ` Alexei Starovoitov
@ 2021-12-01 17:59 ` Andrii Nakryiko
2021-12-01 20:36 ` Alexei Starovoitov
2021-12-01 21:16 ` Jiri Olsa
0 siblings, 2 replies; 49+ messages in thread
From: Andrii Nakryiko @ 2021-12-01 17:59 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Jiri Olsa, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Networking, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
On Wed, Dec 1, 2021 at 9:37 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Nov 30, 2021 at 11:13 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > Hm... I'd actually try to keep kprobe BTF-free. We have fentry for
> > cases where BTF is present and the function is simple enough (like <=6
> > args, etc). Kprobe is an escape hatch mechanism when all the BTF
> > fanciness just gets in the way (retsnoop being a primary example from
> > my side). What I meant here was that bpf_get_arg(int n) would read
> > correct fields from pt_regs that map to first N arguments passed in
> > the registers. What we currently have with PT_REGS_PARM macros in
> > bpf_tracing.h, but with a proper unified BPF helper.
>
> and these macros are arch specific.
> which means that it won't be a trivial patch to add bpf_get_arg()
> support for kprobes.
no one suggested it would be trivial :) things worth doing are usually
non-trivial, as can be evidenced by Jiri's patch set
> Plenty of things to consider. Like should it return an error
> at run-time or verification time when a particular arch is not supported.
See my other replies to Jiri, I'm more and more convinced that dynamic
is the way to go for things like this, where the safety of the kernel
or BPF program are not compromised.
But you emphasized an important point, that it's probably good to
allow users to distinguish errors from reading actual value 0. There
are and will be situations where argument isn't available or some
combination of conditions are not supported. So I think, while it's a
bit more verbose, these forms are generally better:
int bpf_get_func_arg(int n, u64 *value);
int bpf_get_func_ret(u64 *value);
WDYT?
> Or argument 6 might be available on one arch, but not on the other.
> 32-bit CPU regs vs 64-bit regs of BPF, etc.
> I wouldn't attempt to mix this work with current patches.
Oh, I didn't suggest doing it as part of this already huge and
complicated set. But I think it's good to think a bit ahead and design
the helper API appropriately, at the very least.
And again, I think bpf_get_func_arg/bpf_get_func_ret deserve their own
patch set where we can discuss all this independently from
multi-attach.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-12-01 17:59 ` Andrii Nakryiko
@ 2021-12-01 20:36 ` Alexei Starovoitov
2021-12-01 21:16 ` Jiri Olsa
1 sibling, 0 replies; 49+ messages in thread
From: Alexei Starovoitov @ 2021-12-01 20:36 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Jiri Olsa, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Networking, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
On Wed, Dec 1, 2021 at 10:00 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> But you emphasized an important point, that it's probably good to
> allow users to distinguish errors from reading actual value 0. There
> are and will be situations where argument isn't available or some
> combination of conditions are not supported. So I think, while it's a
> bit more verbose, these forms are generally better:
>
> int bpf_get_func_arg(int n, u64 *value);
> int bpf_get_func_ret(u64 *value);
>
> WDYT?
Makes sense to me.
The verifier will be able to inline it just fine.
Two extra insns only compared to direct return.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs
2021-12-01 17:59 ` Andrii Nakryiko
2021-12-01 20:36 ` Alexei Starovoitov
@ 2021-12-01 21:16 ` Jiri Olsa
1 sibling, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-12-01 21:16 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Wed, Dec 01, 2021 at 09:59:57AM -0800, Andrii Nakryiko wrote:
> On Wed, Dec 1, 2021 at 9:37 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Tue, Nov 30, 2021 at 11:13 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > Hm... I'd actually try to keep kprobe BTF-free. We have fentry for
> > > cases where BTF is present and the function is simple enough (like <=6
> > > args, etc). Kprobe is an escape hatch mechanism when all the BTF
> > > fanciness just gets in the way (retsnoop being a primary example from
> > > my side). What I meant here was that bpf_get_arg(int n) would read
> > > correct fields from pt_regs that map to first N arguments passed in
> > > the registers. What we currently have with PT_REGS_PARM macros in
> > > bpf_tracing.h, but with a proper unified BPF helper.
> >
> > and these macros are arch specific.
> > which means that it won't be a trivial patch to add bpf_get_arg()
> > support for kprobes.
>
> no one suggested it would be trivial :) things worth doing are usually
> non-trivial, as can be evidenced by Jiri's patch set
>
> > Plenty of things to consider. Like should it return an error
> > at run-time or verification time when a particular arch is not supported.
>
> See my other replies to Jiri, I'm more and more convinced that dynamic
> is the way to go for things like this, where the safety of the kernel
> or BPF program are not compromised.
>
> But you emphasized an important point, that it's probably good to
> allow users to distinguish errors from reading actual value 0. There
> are and will be situations where argument isn't available or some
> combination of conditions are not supported. So I think, while it's a
> bit more verbose, these forms are generally better:
>
> int bpf_get_func_arg(int n, u64 *value);
> int bpf_get_func_ret(u64 *value);
>
> WDYT?
ok, good preparation for kprobe code quirks described by Alexei
>
> > Or argument 6 might be available on one arch, but not on the other.
> > 32-bit CPU regs vs 64-bit regs of BPF, etc.
> > I wouldn't attempt to mix this work with current patches.
>
> Oh, I didn't suggest doing it as part of this already huge and
> complicated set. But I think it's good to think a bit ahead and design
> the helper API appropriately, at the very least.
>
> And again, I think bpf_get_func_arg/bpf_get_func_ret deserve their own
> patch set where we can discuss all this independently from
> multi-attach.
>
good ;-) thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (5 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-19 4:14 ` Alexei Starovoitov
2021-11-18 11:24 ` [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog Jiri Olsa
` (21 subsequent siblings)
28 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Currently we call the original function by using the absolute address
given at the JIT generation. That's not usable when having trampoline
attached to multiple functions. In this case we need to take the
return address from the stack.
Adding support to retrieve the original function address from the stack
by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline
function.
Basically we take the return address of the 'fentry' call:
function + 0: call fentry # stores 'function + 5' address on stack
function + 5: ...
The 'function + 5' address will be used as the address for the
original function to call.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 13 +++++++++----
include/linux/bpf.h | 5 +++++
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 67e8ac9aaf0d..d87001073033 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2035,10 +2035,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
if (flags & BPF_TRAMP_F_CALL_ORIG) {
restore_regs(m, &prog, nr_args, stack_size);
- /* call original function */
- if (emit_call(&prog, orig_call, prog)) {
- ret = -EINVAL;
- goto cleanup;
+ if (flags & BPF_TRAMP_F_ORIG_STACK) {
+ emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8);
+ EMIT2(0xff, 0xd0); /* call *rax */
+ } else {
+ /* call original function */
+ if (emit_call(&prog, orig_call, prog)) {
+ ret = -EINVAL;
+ goto cleanup;
+ }
}
/* remember return value in a stack for bpf prog to access */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index cc7a0c36e7df..77c76e2fa9ff 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -594,6 +594,11 @@ struct btf_func_model {
/* Return the return value of fentry prog. Only used by bpf_struct_ops. */
#define BPF_TRAMP_F_RET_FENTRY_RET BIT(4)
+/* Get original function from stack instead of from provided direct address.
+ * Makes sense for fexit programs only.
+ */
+#define BPF_TRAMP_F_ORIG_STACK BIT(5)
+
/* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50
* bytes on x86. Pick a number to fit into BPF_IMAGE_SIZE / 2
*/
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack
2021-11-18 11:24 ` [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack Jiri Olsa
@ 2021-11-19 4:14 ` Alexei Starovoitov
2021-11-19 21:46 ` Jiri Olsa
0 siblings, 1 reply; 49+ messages in thread
From: Alexei Starovoitov @ 2021-11-19 4:14 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, netdev,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 12:24:33PM +0100, Jiri Olsa wrote:
> Currently we call the original function by using the absolute address
> given at the JIT generation. That's not usable when having trampoline
> attached to multiple functions. In this case we need to take the
> return address from the stack.
>
> Adding support to retrieve the original function address from the stack
> by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline
> function.
>
> Basically we take the return address of the 'fentry' call:
>
> function + 0: call fentry # stores 'function + 5' address on stack
> function + 5: ...
>
> The 'function + 5' address will be used as the address for the
> original function to call.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> arch/x86/net/bpf_jit_comp.c | 13 +++++++++----
> include/linux/bpf.h | 5 +++++
> 2 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index 67e8ac9aaf0d..d87001073033 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -2035,10 +2035,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> if (flags & BPF_TRAMP_F_CALL_ORIG) {
> restore_regs(m, &prog, nr_args, stack_size);
>
> - /* call original function */
> - if (emit_call(&prog, orig_call, prog)) {
> - ret = -EINVAL;
> - goto cleanup;
> + if (flags & BPF_TRAMP_F_ORIG_STACK) {
> + emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8);
> + EMIT2(0xff, 0xd0); /* call *rax */
Either return an eror if repoline is on
or use emit_indirect_jump().
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack
2021-11-19 4:14 ` Alexei Starovoitov
@ 2021-11-19 21:46 ` Jiri Olsa
0 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-19 21:46 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, netdev,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 08:14:09PM -0800, Alexei Starovoitov wrote:
> On Thu, Nov 18, 2021 at 12:24:33PM +0100, Jiri Olsa wrote:
> > Currently we call the original function by using the absolute address
> > given at the JIT generation. That's not usable when having trampoline
> > attached to multiple functions. In this case we need to take the
> > return address from the stack.
> >
> > Adding support to retrieve the original function address from the stack
> > by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline
> > function.
> >
> > Basically we take the return address of the 'fentry' call:
> >
> > function + 0: call fentry # stores 'function + 5' address on stack
> > function + 5: ...
> >
> > The 'function + 5' address will be used as the address for the
> > original function to call.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> > arch/x86/net/bpf_jit_comp.c | 13 +++++++++----
> > include/linux/bpf.h | 5 +++++
> > 2 files changed, 14 insertions(+), 4 deletions(-)
> >
> > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> > index 67e8ac9aaf0d..d87001073033 100644
> > --- a/arch/x86/net/bpf_jit_comp.c
> > +++ b/arch/x86/net/bpf_jit_comp.c
> > @@ -2035,10 +2035,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
> > if (flags & BPF_TRAMP_F_CALL_ORIG) {
> > restore_regs(m, &prog, nr_args, stack_size);
> >
> > - /* call original function */
> > - if (emit_call(&prog, orig_call, prog)) {
> > - ret = -EINVAL;
> > - goto cleanup;
> > + if (flags & BPF_TRAMP_F_ORIG_STACK) {
> > + emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8);
> > + EMIT2(0xff, 0xd0); /* call *rax */
>
> Either return an eror if repoline is on
> or use emit_indirect_jump().
>
ok, will check
thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (6 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-24 21:48 ` Andrii Nakryiko
2021-11-18 11:24 ` [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program Jiri Olsa
` (20 subsequent siblings)
28 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Keeping active attached trampoline in bpf_prog so it can be used
in following changes to account for multiple functions attachments
in program.
As EXT programs are not going to be supported in multiple functions
attachment for now, I'm keeping them stored in link.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 1 +
kernel/bpf/syscall.c | 34 +++++++++++++++++++++++++++++-----
2 files changed, 30 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 77c76e2fa9ff..c93c629b5725 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -872,6 +872,7 @@ struct bpf_prog_aux {
struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
struct bpf_prog *dst_prog;
struct bpf_trampoline *dst_trampoline;
+ struct bpf_trampoline *trampoline;
enum bpf_prog_type saved_dst_prog_type;
enum bpf_attach_type saved_dst_attach_type;
bool verifier_zext; /* Zero extensions has been inserted by verifier. */
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 50f96ea4452a..0df8b2f3d982 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2623,15 +2623,28 @@ struct bpf_tracing_link {
struct bpf_prog *tgt_prog;
};
+static struct bpf_trampoline *link_trampoline(struct bpf_tracing_link *link)
+{
+ struct bpf_prog *prog = link->link.prog;
+
+ if (prog->type == BPF_PROG_TYPE_EXT)
+ return link->trampoline;
+ else
+ return prog->aux->trampoline;
+}
+
static void bpf_tracing_link_release(struct bpf_link *link)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link);
+ struct bpf_trampoline *tr = link_trampoline(tr_link);
+ struct bpf_prog *prog = link->prog;
- WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog,
- tr_link->trampoline));
+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, tr));
- bpf_trampoline_put(tr_link->trampoline);
+ if (prog->type != BPF_PROG_TYPE_EXT)
+ prog->aux->trampoline = NULL;
+ bpf_trampoline_put(tr);
/* tgt_prog is NULL if target is a kernel function */
if (tr_link->tgt_prog)
@@ -2662,9 +2675,10 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link);
+ struct bpf_trampoline *tr = link_trampoline(tr_link);
info->tracing.attach_type = tr_link->attach_type;
- bpf_trampoline_unpack_key(tr_link->trampoline->key,
+ bpf_trampoline_unpack_key(tr->key,
&info->tracing.target_obj_id,
&info->tracing.target_btf_id);
@@ -2682,6 +2696,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
int tgt_prog_fd,
u32 btf_id)
{
+ bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;
struct bpf_link_primer link_primer;
struct bpf_prog *tgt_prog = NULL;
struct bpf_trampoline *tr = NULL;
@@ -2748,6 +2763,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
mutex_lock(&prog->aux->dst_mutex);
+ if (!prog_extension && prog->aux->trampoline) {
+ err = -EBUSY;
+ goto out_unlock;
+ }
+
/* There are a few possible cases here:
*
* - if prog->aux->dst_trampoline is set, the program was just loaded
@@ -2824,7 +2844,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
}
link->tgt_prog = tgt_prog;
- link->trampoline = tr;
+
+ if (prog_extension)
+ link->trampoline = tr;
+ else
+ prog->aux->trampoline = tr;
/* Always clear the trampoline and target prog from prog->aux to make
* sure the original attach destination is not kept alive after a
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog
2021-11-18 11:24 ` [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog Jiri Olsa
@ 2021-11-24 21:48 ` Andrii Nakryiko
2021-11-28 17:24 ` Jiri Olsa
0 siblings, 1 reply; 49+ messages in thread
From: Andrii Nakryiko @ 2021-11-24 21:48 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Networking,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 3:25 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> Keeping active attached trampoline in bpf_prog so it can be used
> in following changes to account for multiple functions attachments
> in program.
>
> As EXT programs are not going to be supported in multiple functions
> attachment for now, I'm keeping them stored in link.
can the same EXT program be attached twice? If not, why can't you just
use the same prog->aux->trampoline instead of the if/else everywhere?
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
> include/linux/bpf.h | 1 +
> kernel/bpf/syscall.c | 34 +++++++++++++++++++++++++++++-----
> 2 files changed, 30 insertions(+), 5 deletions(-)
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog
2021-11-24 21:48 ` Andrii Nakryiko
@ 2021-11-28 17:24 ` Jiri Olsa
0 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-28 17:24 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Networking,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Wed, Nov 24, 2021 at 01:48:09PM -0800, Andrii Nakryiko wrote:
> On Thu, Nov 18, 2021 at 3:25 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > Keeping active attached trampoline in bpf_prog so it can be used
> > in following changes to account for multiple functions attachments
> > in program.
> >
> > As EXT programs are not going to be supported in multiple functions
> > attachment for now, I'm keeping them stored in link.
>
> can the same EXT program be attached twice? If not, why can't you just
> use the same prog->aux->trampoline instead of the if/else everywhere?
I recall that was my initial change, but it was clashing with
fentry/fexit programs because extensions are special
I'll re-check and try to make this generic
jirka
>
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> > include/linux/bpf.h | 1 +
> > kernel/bpf/syscall.c | 34 +++++++++++++++++++++++++++++-----
> > 2 files changed, 30 insertions(+), 5 deletions(-)
> >
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (7 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-19 4:11 ` Alexei Starovoitov
2021-11-18 11:24 ` [PATCH bpf-next 10/29] bpf: Add bpf_trampoline_id object Jiri Olsa
` (19 subsequent siblings)
28 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding support to load tracing program with new BPF_F_MULTI_FUNC flag,
that allows the program to be loaded without specific function to be
attached to.
Such program will be allowed to be attached to multiple functions
in following patches.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 1 +
include/uapi/linux/bpf.h | 7 +++++++
kernel/bpf/syscall.c | 35 +++++++++++++++++++++++++++++-----
kernel/bpf/verifier.c | 3 ++-
tools/include/uapi/linux/bpf.h | 7 +++++++
5 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index c93c629b5725..35f484f323f3 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -881,6 +881,7 @@ struct bpf_prog_aux {
bool func_proto_unreliable;
bool sleepable;
bool tail_call_reachable;
+ bool multi_func;
struct hlist_node tramp_hlist;
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index fc8b344eecba..ca05e35e0478 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1111,6 +1111,13 @@ enum bpf_link_type {
*/
#define BPF_F_SLEEPABLE (1U << 4)
+/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does
+ * not expect BTF ID for the program, instead it assumes it's function
+ * with 6 u64 arguments. No trampoline is created for the program. Such
+ * program can be attached to multiple functions.
+ */
+#define BPF_F_MULTI_FUNC (1U << 5)
+
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
* the following extensions:
*
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0df8b2f3d982..648155f0a4b1 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -31,6 +31,7 @@
#include <linux/bpf-netns.h>
#include <linux/rcupdate_trace.h>
#include <linux/memcontrol.h>
+#include <linux/btf_ids.h>
#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -2040,7 +2041,8 @@ static int
bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
enum bpf_attach_type expected_attach_type,
struct btf *attach_btf, u32 btf_id,
- struct bpf_prog *dst_prog)
+ struct bpf_prog *dst_prog,
+ bool multi_func)
{
if (btf_id) {
if (btf_id > BTF_MAX_TYPE)
@@ -2060,6 +2062,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
}
}
+ if (multi_func) {
+ if (prog_type != BPF_PROG_TYPE_TRACING)
+ return -EINVAL;
+ if (!attach_btf || btf_id)
+ return -EINVAL;
+ return 0;
+ }
+
if (attach_btf && (!btf_id || dst_prog))
return -EINVAL;
@@ -2183,6 +2193,16 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
}
}
+#define DEFINE_BPF_MULTI_FUNC(args...) \
+ extern int bpf_multi_func(args); \
+ int __init bpf_multi_func(args) { return 0; }
+
+DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
+ unsigned long a3, unsigned long a4,
+ unsigned long a5, unsigned long a6)
+
+BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
+
/* last field in 'union bpf_attr' used by this command */
#define BPF_PROG_LOAD_LAST_FIELD fd_array
@@ -2193,6 +2213,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
struct btf *attach_btf = NULL;
int err;
char license[128];
+ bool multi_func;
bool is_gpl;
if (CHECK_ATTR(BPF_PROG_LOAD))
@@ -2202,7 +2223,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
BPF_F_ANY_ALIGNMENT |
BPF_F_TEST_STATE_FREQ |
BPF_F_SLEEPABLE |
- BPF_F_TEST_RND_HI32))
+ BPF_F_TEST_RND_HI32 |
+ BPF_F_MULTI_FUNC))
return -EINVAL;
if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
@@ -2233,6 +2255,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
if (is_perfmon_prog_type(type) && !perfmon_capable())
return -EPERM;
+ multi_func = attr->prog_flags & BPF_F_MULTI_FUNC;
+
/* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
* or btf, we need to check which one it is
*/
@@ -2251,7 +2275,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
return -ENOTSUPP;
}
}
- } else if (attr->attach_btf_id) {
+ } else if (attr->attach_btf_id || multi_func) {
/* fall back to vmlinux BTF, if BTF type ID is specified */
attach_btf = bpf_get_btf_vmlinux();
if (IS_ERR(attach_btf))
@@ -2264,7 +2288,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
bpf_prog_load_fixup_attach_type(attr);
if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
attach_btf, attr->attach_btf_id,
- dst_prog)) {
+ dst_prog, multi_func)) {
if (dst_prog)
bpf_prog_put(dst_prog);
if (attach_btf)
@@ -2284,10 +2308,11 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr)
prog->expected_attach_type = attr->expected_attach_type;
prog->aux->attach_btf = attach_btf;
- prog->aux->attach_btf_id = attr->attach_btf_id;
+ prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
prog->aux->dst_prog = dst_prog;
prog->aux->offload_requested = !!attr->prog_ifindex;
prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
+ prog->aux->multi_func = multi_func;
err = security_bpf_prog_alloc(prog->aux);
if (err)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d4249ef6ca7e..af6a39bbb0dc 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13983,7 +13983,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
if (!bpf_iter_prog_supported(prog))
return -EINVAL;
return 0;
- }
+ } else if (prog->aux->multi_func)
+ return prog->type == BPF_PROG_TYPE_TRACING ? 0 : -EINVAL;
if (prog->type == BPF_PROG_TYPE_LSM) {
ret = bpf_lsm_verify_prog(&env->log, prog);
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index fc8b344eecba..ca05e35e0478 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1111,6 +1111,13 @@ enum bpf_link_type {
*/
#define BPF_F_SLEEPABLE (1U << 4)
+/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does
+ * not expect BTF ID for the program, instead it assumes it's function
+ * with 6 u64 arguments. No trampoline is created for the program. Such
+ * program can be attached to multiple functions.
+ */
+#define BPF_F_MULTI_FUNC (1U << 5)
+
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
* the following extensions:
*
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-18 11:24 ` [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program Jiri Olsa
@ 2021-11-19 4:11 ` Alexei Starovoitov
2021-11-22 20:15 ` Jiri Olsa
0 siblings, 1 reply; 49+ messages in thread
From: Alexei Starovoitov @ 2021-11-19 4:11 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, netdev,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> +
> +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> + unsigned long a3, unsigned long a4,
> + unsigned long a5, unsigned long a6)
This is probably a bit too x86 specific. May be make add all 12 args?
Or other places would need to be tweaked?
> +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
...
> - prog->aux->attach_btf_id = attr->attach_btf_id;
> + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
Just ignoring that was passed in uattr?
Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
point to that btf_id instead?
Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-19 4:11 ` Alexei Starovoitov
@ 2021-11-22 20:15 ` Jiri Olsa
2021-11-24 21:51 ` Andrii Nakryiko
0 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-22 20:15 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, netdev,
bpf, Martin KaFai Lau, Song Liu, Yonghong Song, John Fastabend,
KP Singh
On Thu, Nov 18, 2021 at 08:11:59PM -0800, Alexei Starovoitov wrote:
> On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> > +
> > +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> > + unsigned long a3, unsigned long a4,
> > + unsigned long a5, unsigned long a6)
>
> This is probably a bit too x86 specific. May be make add all 12 args?
> Or other places would need to be tweaked?
I think si, I'll check
>
> > +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
> ...
> > - prog->aux->attach_btf_id = attr->attach_btf_id;
> > + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
>
> Just ignoring that was passed in uattr?
> Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
> point to that btf_id instead?
> Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
>
nice idea, it might fit better than the flag
thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-22 20:15 ` Jiri Olsa
@ 2021-11-24 21:51 ` Andrii Nakryiko
2021-11-28 17:41 ` Jiri Olsa
0 siblings, 1 reply; 49+ messages in thread
From: Andrii Nakryiko @ 2021-11-24 21:51 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Mon, Nov 22, 2021 at 12:15 PM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Thu, Nov 18, 2021 at 08:11:59PM -0800, Alexei Starovoitov wrote:
> > On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> > > +
> > > +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> > > + unsigned long a3, unsigned long a4,
> > > + unsigned long a5, unsigned long a6)
> >
> > This is probably a bit too x86 specific. May be make add all 12 args?
> > Or other places would need to be tweaked?
>
> I think si, I'll check
>
> >
> > > +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
> > ...
> > > - prog->aux->attach_btf_id = attr->attach_btf_id;
> > > + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
> >
> > Just ignoring that was passed in uattr?
> > Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
> > point to that btf_id instead?
> > Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
> >
>
> nice idea, it might fit better than the flag
Instead of a flag we can also use a different expected_attach_type
(FENTRY vs FENTRY_MULTI, etc). As for attach_btf_id, why can't we just
enforce it as 0?
>
> thanks,
> jirka
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-24 21:51 ` Andrii Nakryiko
@ 2021-11-28 17:41 ` Jiri Olsa
2021-12-01 7:17 ` Andrii Nakryiko
0 siblings, 1 reply; 49+ messages in thread
From: Jiri Olsa @ 2021-11-28 17:41 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Wed, Nov 24, 2021 at 01:51:36PM -0800, Andrii Nakryiko wrote:
> On Mon, Nov 22, 2021 at 12:15 PM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Thu, Nov 18, 2021 at 08:11:59PM -0800, Alexei Starovoitov wrote:
> > > On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> > > > +
> > > > +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> > > > + unsigned long a3, unsigned long a4,
> > > > + unsigned long a5, unsigned long a6)
> > >
> > > This is probably a bit too x86 specific. May be make add all 12 args?
> > > Or other places would need to be tweaked?
> >
> > I think si, I'll check
> >
> > >
> > > > +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
> > > ...
> > > > - prog->aux->attach_btf_id = attr->attach_btf_id;
> > > > + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
> > >
> > > Just ignoring that was passed in uattr?
> > > Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
> > > point to that btf_id instead?
> > > Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
> > >
> >
> > nice idea, it might fit better than the flag
>
> Instead of a flag we can also use a different expected_attach_type
> (FENTRY vs FENTRY_MULTI, etc).
right, you already asked for that - https://lore.kernel.org/bpf/YS9k26rRcUJVS%2Fvx@krava/
I still think it'd mean more code while this way we just use
current fentry/fexit code paths with few special handling
for multi programs
> As for attach_btf_id, why can't we just
> enforce it as 0?
there's prog->aux->attach_func_proto that needs to be set based
on attach_btf_id, and is checked later in btf_ctx_access
jirka
>
> >
> > thanks,
> > jirka
> >
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-11-28 17:41 ` Jiri Olsa
@ 2021-12-01 7:17 ` Andrii Nakryiko
2021-12-01 21:20 ` Jiri Olsa
0 siblings, 1 reply; 49+ messages in thread
From: Andrii Nakryiko @ 2021-12-01 7:17 UTC (permalink / raw)
To: Jiri Olsa
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Sun, Nov 28, 2021 at 9:41 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Wed, Nov 24, 2021 at 01:51:36PM -0800, Andrii Nakryiko wrote:
> > On Mon, Nov 22, 2021 at 12:15 PM Jiri Olsa <jolsa@redhat.com> wrote:
> > >
> > > On Thu, Nov 18, 2021 at 08:11:59PM -0800, Alexei Starovoitov wrote:
> > > > On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> > > > > +
> > > > > +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> > > > > + unsigned long a3, unsigned long a4,
> > > > > + unsigned long a5, unsigned long a6)
> > > >
> > > > This is probably a bit too x86 specific. May be make add all 12 args?
> > > > Or other places would need to be tweaked?
> > >
> > > I think si, I'll check
> > >
> > > >
> > > > > +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
> > > > ...
> > > > > - prog->aux->attach_btf_id = attr->attach_btf_id;
> > > > > + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
> > > >
> > > > Just ignoring that was passed in uattr?
> > > > Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
> > > > point to that btf_id instead?
> > > > Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
> > > >
> > >
> > > nice idea, it might fit better than the flag
> >
> > Instead of a flag we can also use a different expected_attach_type
> > (FENTRY vs FENTRY_MULTI, etc).
>
> right, you already asked for that - https://lore.kernel.org/bpf/YS9k26rRcUJVS%2Fvx@krava/
>
> I still think it'd mean more code while this way we just use
> current fentry/fexit code paths with few special handling
> for multi programs
>
I don't see how it makes much difference for kernel implementation.
Checking expected_attach_type vs checking prog_flags is about the same
amount of code. The big advantage of new expected_attach_type (or
prog_type) is that it will be very obvious in all sorts of diagnostics
tooling (think bpftool prog show output, etc). prog_flags are almost
invisible and it will be the last thing that users will think about.
I'd try to minimize the usage of prog_flags overall.
> > As for attach_btf_id, why can't we just
> > enforce it as 0?
>
> there's prog->aux->attach_func_proto that needs to be set based
> on attach_btf_id, and is checked later in btf_ctx_access
right:
if (attach_btf_id == 0)
prog->aux->attach_func_proto =
&special_func_model_or_proto_or_whatever_that_does_not_have_to_be_known_to_libbpf_and_outside_world_ever;
;) let's keep implementation details as internal implementation
details, instead of dumping all that to UAPI
>
> jirka
>
> >
> > >
> > > thanks,
> > > jirka
> > >
> >
>
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program
2021-12-01 7:17 ` Andrii Nakryiko
@ 2021-12-01 21:20 ` Jiri Olsa
0 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-12-01 21:20 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Networking, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
On Tue, Nov 30, 2021 at 11:17:44PM -0800, Andrii Nakryiko wrote:
> On Sun, Nov 28, 2021 at 9:41 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Wed, Nov 24, 2021 at 01:51:36PM -0800, Andrii Nakryiko wrote:
> > > On Mon, Nov 22, 2021 at 12:15 PM Jiri Olsa <jolsa@redhat.com> wrote:
> > > >
> > > > On Thu, Nov 18, 2021 at 08:11:59PM -0800, Alexei Starovoitov wrote:
> > > > > On Thu, Nov 18, 2021 at 12:24:35PM +0100, Jiri Olsa wrote:
> > > > > > +
> > > > > > +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2,
> > > > > > + unsigned long a3, unsigned long a4,
> > > > > > + unsigned long a5, unsigned long a6)
> > > > >
> > > > > This is probably a bit too x86 specific. May be make add all 12 args?
> > > > > Or other places would need to be tweaked?
> > > >
> > > > I think si, I'll check
> > > >
> > > > >
> > > > > > +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
> > > > > ...
> > > > > > - prog->aux->attach_btf_id = attr->attach_btf_id;
> > > > > > + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
> > > > >
> > > > > Just ignoring that was passed in uattr?
> > > > > Maybe instead of ignoring dopr BPF_F_MULTI_FUNC and make libbpf
> > > > > point to that btf_id instead?
> > > > > Then multi or not can be checked with if (attr->attach_btf_id == bpf_multi_func_btf_id[0]).
> > > > >
> > > >
> > > > nice idea, it might fit better than the flag
> > >
> > > Instead of a flag we can also use a different expected_attach_type
> > > (FENTRY vs FENTRY_MULTI, etc).
> >
> > right, you already asked for that - https://lore.kernel.org/bpf/YS9k26rRcUJVS%2Fvx@krava/
> >
> > I still think it'd mean more code while this way we just use
> > current fentry/fexit code paths with few special handling
> > for multi programs
> >
>
> I don't see how it makes much difference for kernel implementation.
> Checking expected_attach_type vs checking prog_flags is about the same
> amount of code. The big advantage of new expected_attach_type (or
> prog_type) is that it will be very obvious in all sorts of diagnostics
> tooling (think bpftool prog show output, etc). prog_flags are almost
> invisible and it will be the last thing that users will think about.
> I'd try to minimize the usage of prog_flags overall.
ok, I'll check on that.. I recall adding this new type in
many expected_attach_type switches, which made me think
the new flag will be easier
>
> > > As for attach_btf_id, why can't we just
> > > enforce it as 0?
> >
> > there's prog->aux->attach_func_proto that needs to be set based
> > on attach_btf_id, and is checked later in btf_ctx_access
>
> right:
>
> if (attach_btf_id == 0)
> prog->aux->attach_func_proto =
> &special_func_model_or_proto_or_whatever_that_does_not_have_to_be_known_to_libbpf_and_outside_world_ever;
>
> ;) let's keep implementation details as internal implementation
> details, instead of dumping all that to UAPI
ok, we can do that ;-)
thanks,
jirka
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH bpf-next 10/29] bpf: Add bpf_trampoline_id object
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (8 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 11/29] bpf: Add addr to " Jiri Olsa
` (18 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Replacing the bpf_trampoline's key with struct bpf_tramp_id object,
that currently holds only obj_id/btf_id, so same data as key.
Having the key in the struct will allow us to add more ids (functions)
to single trampoline in following patches.
No functional change is intended.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 18 +++++++++--
include/linux/bpf_verifier.h | 19 ------------
kernel/bpf/syscall.c | 30 +++++++++++++-----
kernel/bpf/trampoline.c | 59 +++++++++++++++++++++++++++++++-----
kernel/bpf/verifier.c | 14 ++++++---
5 files changed, 99 insertions(+), 41 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 35f484f323f3..2ce8b1c49af7 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -671,13 +671,18 @@ struct bpf_tramp_image {
};
};
+struct bpf_tramp_id {
+ u32 obj_id;
+ u32 btf_id;
+};
+
struct bpf_trampoline {
/* hlist for trampoline_table */
struct hlist_node hlist;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
- u64 key;
+ struct bpf_tramp_id *id;
struct {
struct btf_func_model model;
void *addr;
@@ -732,9 +737,16 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
return bpf_func(ctx, insnsi);
}
#ifdef CONFIG_BPF_JIT
+struct bpf_tramp_id *bpf_tramp_id_alloc(void);
+void bpf_tramp_id_free(struct bpf_tramp_id *id);
+bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id);
+int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
+void bpf_tramp_id_init(struct bpf_tramp_id *id,
+ const struct bpf_prog *tgt_prog,
+ struct btf *btf, u32 btf_id);
int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
-struct bpf_trampoline *bpf_trampoline_get(u64 key,
+struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
#define BPF_DISPATCHER_INIT(_name) { \
@@ -792,7 +804,7 @@ static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog,
{
return -ENOTSUPP;
}
-static inline struct bpf_trampoline *bpf_trampoline_get(u64 key,
+static inline struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
struct bpf_attach_target_info *tgt_info)
{
return ERR_PTR(-EOPNOTSUPP);
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index b561c0b08e68..0c01a50180a0 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -503,25 +503,6 @@ int check_ctx_reg(struct bpf_verifier_env *env,
int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
u32 regno, u32 mem_size);
-/* this lives here instead of in bpf.h because it needs to dereference tgt_prog */
-static inline u64 bpf_trampoline_compute_key(const struct bpf_prog *tgt_prog,
- struct btf *btf, u32 btf_id)
-{
- if (tgt_prog)
- return ((u64)tgt_prog->aux->id << 32) | btf_id;
- else
- return ((u64)btf_obj_id(btf) << 32) | 0x80000000 | btf_id;
-}
-
-/* unpack the IDs from the key as constructed above */
-static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
-{
- if (obj_id)
- *obj_id = key >> 32;
- if (btf_id)
- *btf_id = key & 0x7FFFFFFF;
-}
-
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 648155f0a4b1..f99ea3237f9c 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2703,9 +2703,8 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
struct bpf_trampoline *tr = link_trampoline(tr_link);
info->tracing.attach_type = tr_link->attach_type;
- bpf_trampoline_unpack_key(tr->key,
- &info->tracing.target_obj_id,
- &info->tracing.target_btf_id);
+ info->tracing.target_obj_id = tr->id->obj_id;
+ info->tracing.target_btf_id = tr->id->btf_id;
return 0;
}
@@ -2726,7 +2725,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
struct bpf_prog *tgt_prog = NULL;
struct bpf_trampoline *tr = NULL;
struct bpf_tracing_link *link;
- u64 key = 0;
+ struct bpf_tramp_id *id = NULL;
int err;
switch (prog->type) {
@@ -2767,6 +2766,12 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
+ id = bpf_tramp_id_alloc();
+ if (!id) {
+ err = -ENOMEM;
+ goto out_put_prog;
+ }
+
tgt_prog = bpf_prog_get(tgt_prog_fd);
if (IS_ERR(tgt_prog)) {
err = PTR_ERR(tgt_prog);
@@ -2774,7 +2779,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
- key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);
+ bpf_tramp_id_init(id, tgt_prog, NULL, btf_id);
}
link = kzalloc(sizeof(*link), GFP_USER);
@@ -2823,12 +2828,20 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
err = -EINVAL;
goto out_unlock;
}
+
+ id = bpf_tramp_id_alloc();
+ if (!id) {
+ err = -ENOMEM;
+ goto out_unlock;
+ }
+
btf_id = prog->aux->attach_btf_id;
- key = bpf_trampoline_compute_key(NULL, prog->aux->attach_btf, btf_id);
+ bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id);
}
if (!prog->aux->dst_trampoline ||
- (key && key != prog->aux->dst_trampoline->key)) {
+ (!bpf_tramp_id_is_empty(id) &&
+ bpf_tramp_id_is_equal(id, prog->aux->dst_trampoline->id))) {
/* If there is no saved target, or the specified target is
* different from the destination specified at load time, we
* need a new trampoline and a check for compatibility
@@ -2840,7 +2853,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- tr = bpf_trampoline_get(key, &tgt_info);
+ tr = bpf_trampoline_get(id, &tgt_info);
if (!tr) {
err = -ENOMEM;
goto out_unlock;
@@ -2900,6 +2913,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
out_put_prog:
if (tgt_prog_fd && tgt_prog)
bpf_prog_put(tgt_prog);
+ bpf_tramp_id_free(id);
return err;
}
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index e98de5e73ba5..ae2573c36653 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -59,16 +59,57 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
PAGE_SIZE, true, ksym->name);
}
-static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
+static u64 bpf_tramp_id_key(struct bpf_tramp_id *id)
+{
+ return ((u64) id->obj_id << 32) | id->btf_id;
+}
+
+bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id)
+{
+ return !id || (!id->obj_id && !id->btf_id);
+}
+
+int bpf_tramp_id_is_equal(struct bpf_tramp_id *a,
+ struct bpf_tramp_id *b)
+{
+ return !memcmp(a, b, sizeof(*a));
+}
+
+struct bpf_tramp_id *bpf_tramp_id_alloc(void)
+{
+ struct bpf_tramp_id *id;
+
+ return kzalloc(sizeof(*id), GFP_KERNEL);
+}
+
+void bpf_tramp_id_init(struct bpf_tramp_id *id,
+ const struct bpf_prog *tgt_prog,
+ struct btf *btf, u32 btf_id)
+{
+ if (tgt_prog)
+ id->obj_id = tgt_prog->aux->id;
+ else
+ id->obj_id = btf_obj_id(btf);
+ id->btf_id = btf_id;
+}
+
+void bpf_tramp_id_free(struct bpf_tramp_id *id)
+{
+ kfree(id);
+}
+
+static struct bpf_trampoline *bpf_trampoline_lookup(struct bpf_tramp_id *id)
{
struct bpf_trampoline *tr;
struct hlist_head *head;
+ u64 key;
int i;
+ key = bpf_tramp_id_key(id);
mutex_lock(&trampoline_mutex);
head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
hlist_for_each_entry(tr, head, hlist) {
- if (tr->key == key) {
+ if (bpf_tramp_id_is_equal(tr->id, id)) {
refcount_inc(&tr->refcnt);
goto out;
}
@@ -77,7 +118,7 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
if (!tr)
goto out;
- tr->key = key;
+ tr->id = id;
INIT_HLIST_NODE(&tr->hlist);
hlist_add_head(&tr->hlist, head);
refcount_set(&tr->refcnt, 1);
@@ -291,12 +332,14 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im)
call_rcu_tasks_trace(&im->rcu, __bpf_tramp_image_put_rcu_tasks);
}
-static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx)
+static struct bpf_tramp_image*
+bpf_tramp_image_alloc(struct bpf_tramp_id *id, u32 idx)
{
struct bpf_tramp_image *im;
struct bpf_ksym *ksym;
void *image;
int err = -ENOMEM;
+ u64 key;
im = kzalloc(sizeof(*im), GFP_KERNEL);
if (!im)
@@ -317,6 +360,7 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx)
ksym = &im->ksym;
INIT_LIST_HEAD_RCU(&ksym->lnode);
+ key = bpf_tramp_id_key(id);
snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u", key, idx);
bpf_image_ksym_add(image, ksym);
return im;
@@ -351,7 +395,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr)
goto out;
}
- im = bpf_tramp_image_alloc(tr->key, tr->selector);
+ im = bpf_tramp_image_alloc(tr->id, tr->selector);
if (IS_ERR(im)) {
err = PTR_ERR(im);
goto out;
@@ -482,12 +526,12 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
return err;
}
-struct bpf_trampoline *bpf_trampoline_get(u64 key,
+struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
struct bpf_attach_target_info *tgt_info)
{
struct bpf_trampoline *tr;
- tr = bpf_trampoline_lookup(key);
+ tr = bpf_trampoline_lookup(id);
if (!tr)
return NULL;
@@ -521,6 +565,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
* multiple rcu callbacks.
*/
hlist_del(&tr->hlist);
+ bpf_tramp_id_free(tr->id);
kfree(tr);
out:
mutex_unlock(&trampoline_mutex);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index af6a39bbb0dc..a1e4389b0e9e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13929,8 +13929,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
struct bpf_attach_target_info tgt_info = {};
u32 btf_id = prog->aux->attach_btf_id;
struct bpf_trampoline *tr;
+ struct bpf_tramp_id *id;
int ret;
- u64 key;
if (prog->type == BPF_PROG_TYPE_SYSCALL) {
if (prog->aux->sleepable)
@@ -13995,11 +13995,17 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
- key = bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_id);
- tr = bpf_trampoline_get(key, &tgt_info);
- if (!tr)
+ id = bpf_tramp_id_alloc();
+ if (!id)
return -ENOMEM;
+ bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id);
+ tr = bpf_trampoline_get(id, &tgt_info);
+ if (!tr) {
+ bpf_tramp_id_free(id);
+ return -ENOMEM;
+ }
+
prog->aux->dst_trampoline = tr;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 11/29] bpf: Add addr to bpf_trampoline_id object
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (9 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 10/29] bpf: Add bpf_trampoline_id object Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 12/29] bpf: Add struct bpf_tramp_node layer Jiri Olsa
` (17 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding addr to bpf_trampoline_id object so it's not associated
directly with trampoline directly. This will help us to easily
support multiple ids/addresses support for trampolines coming
in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 4 ++--
kernel/bpf/trampoline.c | 18 +++++++++---------
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 2ce8b1c49af7..c57141a76e7b 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -674,6 +674,7 @@ struct bpf_tramp_image {
struct bpf_tramp_id {
u32 obj_id;
u32 btf_id;
+ void *addr;
};
struct bpf_trampoline {
@@ -685,11 +686,10 @@ struct bpf_trampoline {
struct bpf_tramp_id *id;
struct {
struct btf_func_model model;
- void *addr;
bool ftrace_managed;
} func;
/* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF
- * program by replacing one of its functions. func.addr is the address
+ * program by replacing one of its functions. id->addr is the address
* of the function it replaced.
*/
struct bpf_prog *extension_prog;
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index ae2573c36653..e19c5112be67 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -136,7 +136,7 @@ static int bpf_trampoline_module_get(struct bpf_trampoline *tr)
int err = 0;
preempt_disable();
- mod = __module_text_address((unsigned long) tr->func.addr);
+ mod = __module_text_address((unsigned long) tr->id->addr);
if (mod && !try_module_get(mod))
err = -ENOENT;
preempt_enable();
@@ -164,7 +164,7 @@ static int is_ftrace_location(void *ip)
static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
{
- void *ip = tr->func.addr;
+ void *ip = tr->id->addr;
int ret;
if (tr->func.ftrace_managed)
@@ -179,7 +179,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_addr)
{
- void *ip = tr->func.addr;
+ void *ip = tr->id->addr;
int ret;
if (tr->func.ftrace_managed)
@@ -192,7 +192,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
/* first time registering */
static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
{
- void *ip = tr->func.addr;
+ void *ip = tr->id->addr;
int ret;
ret = is_ftrace_location(ip);
@@ -410,7 +410,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr)
err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE,
&tr->func.model, flags, tprogs,
- tr->func.addr);
+ tr->id->addr);
if (err < 0)
goto out;
@@ -478,7 +478,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
goto out;
}
tr->extension_prog = prog;
- err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL,
+ err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, NULL,
prog->bpf_func);
goto out;
}
@@ -513,7 +513,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
mutex_lock(&tr->mutex);
if (kind == BPF_TRAMP_REPLACE) {
WARN_ON_ONCE(!tr->extension_prog);
- err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
+ err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP,
tr->extension_prog->bpf_func, NULL);
tr->extension_prog = NULL;
goto out;
@@ -536,11 +536,11 @@ struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
return NULL;
mutex_lock(&tr->mutex);
- if (tr->func.addr)
+ if (tr->id->addr)
goto out;
memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
- tr->func.addr = (void *)tgt_info->tgt_addr;
+ tr->id->addr = (void *)tgt_info->tgt_addr;
out:
mutex_unlock(&tr->mutex);
return tr;
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 12/29] bpf: Add struct bpf_tramp_node layer
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (10 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 11/29] bpf: Add addr to " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 13/29] bpf: Add bpf_tramp_attach layer for trampoline attachment Jiri Olsa
` (16 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Currently each trampoline holds a list of programs that
are attached to it. With multi func attach support we need
a way for a single program to be connected to multiple
trampolines.
Adding struct bpf_tramp_node object that holds bpf_prog
pointer, so it can be resolved directly. We can now
have multiple struct bpf_tramp_node being attached to
different trampolines pointing to single bpf_prog.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 15 ++++++++++-----
kernel/bpf/core.c | 1 +
kernel/bpf/syscall.c | 4 ++--
kernel/bpf/trampoline.c | 22 ++++++++++++----------
4 files changed, 25 insertions(+), 17 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index c57141a76e7b..21f8dbcf3f48 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -677,6 +677,11 @@ struct bpf_tramp_id {
void *addr;
};
+struct bpf_tramp_node {
+ struct hlist_node hlist_tramp;
+ struct bpf_prog *prog;
+};
+
struct bpf_trampoline {
/* hlist for trampoline_table */
struct hlist_node hlist;
@@ -744,8 +749,8 @@ int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
void bpf_tramp_id_init(struct bpf_tramp_id *id,
const struct bpf_prog *tgt_prog,
struct btf *btf, u32 btf_id);
-int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
-int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr);
+int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
+int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
@@ -794,12 +799,12 @@ void bpf_ksym_del(struct bpf_ksym *ksym);
int bpf_jit_charge_modmem(u32 pages);
void bpf_jit_uncharge_modmem(u32 pages);
#else
-static inline int bpf_trampoline_link_prog(struct bpf_prog *prog,
+static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr)
{
return -ENOTSUPP;
}
-static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog,
+static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr)
{
return -ENOTSUPP;
@@ -894,7 +899,7 @@ struct bpf_prog_aux {
bool sleepable;
bool tail_call_reachable;
bool multi_func;
- struct hlist_node tramp_hlist;
+ struct bpf_tramp_node tramp_node;
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
/* function name for valid attach_btf_id */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index b52dc845ecea..2eed91153a3f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -105,6 +105,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
fp->aux = aux;
fp->aux->prog = fp;
fp->jit_requested = ebpf_jit_enabled();
+ fp->aux->tramp_node.prog = fp;
INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
mutex_init(&fp->aux->used_maps_mutex);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index f99ea3237f9c..0d916e3b7676 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2665,7 +2665,7 @@ static void bpf_tracing_link_release(struct bpf_link *link)
struct bpf_trampoline *tr = link_trampoline(tr_link);
struct bpf_prog *prog = link->prog;
- WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, tr));
+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(&link->prog->aux->tramp_node, tr));
if (prog->type != BPF_PROG_TYPE_EXT)
prog->aux->trampoline = NULL;
@@ -2874,7 +2874,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- err = bpf_trampoline_link_prog(prog, tr);
+ err = bpf_trampoline_link_prog(&prog->aux->tramp_node, tr);
if (err) {
bpf_link_cleanup(&link_primer);
link = NULL;
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index e19c5112be67..b6af3e0982d7 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -216,8 +216,8 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
static struct bpf_tramp_progs *
bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)
{
- const struct bpf_prog_aux *aux;
struct bpf_tramp_progs *tprogs;
+ struct bpf_tramp_node *node;
struct bpf_prog **progs;
int kind;
@@ -231,9 +231,9 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
*total += tr->progs_cnt[kind];
progs = tprogs[kind].progs;
- hlist_for_each_entry(aux, &tr->progs_hlist[kind], tramp_hlist) {
- *ip_arg |= aux->prog->call_get_func_ip;
- *progs++ = aux->prog;
+ hlist_for_each_entry(node, &tr->progs_hlist[kind], hlist_tramp) {
+ *ip_arg |= node->prog->call_get_func_ip;
+ *progs++ = node->prog;
}
}
return tprogs;
@@ -455,8 +455,9 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
}
}
-int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
+int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr)
{
+ struct bpf_prog *prog = node->prog;
enum bpf_tramp_prog_type kind;
int err = 0;
int cnt;
@@ -486,16 +487,16 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
err = -E2BIG;
goto out;
}
- if (!hlist_unhashed(&prog->aux->tramp_hlist)) {
+ if (!hlist_unhashed(&node->hlist_tramp)) {
/* prog already linked */
err = -EBUSY;
goto out;
}
- hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]);
+ hlist_add_head(&node->hlist_tramp, &tr->progs_hlist[kind]);
tr->progs_cnt[kind]++;
err = bpf_trampoline_update(tr);
if (err) {
- hlist_del_init(&prog->aux->tramp_hlist);
+ hlist_del_init(&node->hlist_tramp);
tr->progs_cnt[kind]--;
}
out:
@@ -504,8 +505,9 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
}
/* bpf_trampoline_unlink_prog() should never fail. */
-int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
+int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr)
{
+ struct bpf_prog *prog = node->prog;
enum bpf_tramp_prog_type kind;
int err;
@@ -518,7 +520,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
tr->extension_prog = NULL;
goto out;
}
- hlist_del_init(&prog->aux->tramp_hlist);
+ hlist_del_init(&node->hlist_tramp);
tr->progs_cnt[kind]--;
err = bpf_trampoline_update(tr);
out:
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 13/29] bpf: Add bpf_tramp_attach layer for trampoline attachment
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (11 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 12/29] bpf: Add struct bpf_tramp_node layer Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 14/29] bpf: Add support to store multiple ids in bpf_tramp_id object Jiri Olsa
` (15 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding bpf_tramp_attach layer for trampoline attachment to
have extra layer on top of the trampoline. The reason is
that in following changes we will add multiple trampolines
for single program and we need entity to hold them.
The api in nutshell:
- each bpf_prog holds 'bpf_tramp_attach' object, which holds
list of 'struct bpf_tramp_node' objects:
struct bpf_tramp_attach {
struct bpf_tramp_id *id;
struct hlist_head nodes;
};
This allow us to hold multiple trampolines for each program.
- bpf_tramp_attach returns 'bpf_tramp_attach' object that
finds trampoline for given 'id' and adds it to the attach
object, no actuall program attachment is done, just trampoline
allocation
- bpf_tramp_attach_link does the actual attachment of the
program to trampoline
- bpf_tramp_attach_unlink unlinks all the trampolines present
in the attach object
- bpf_tramp_detach frees all the trampolines in attach object
Currently there'll be only single node added in attach object.
Following patches add support for multiple id trampolines,
and uses multiple nodes in attach object to hold trampoline
for given program.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 21 +++++--
kernel/bpf/core.c | 5 +-
kernel/bpf/syscall.c | 61 ++++++++++----------
kernel/bpf/trampoline.c | 122 ++++++++++++++++++++++++++++++++--------
kernel/bpf/verifier.c | 12 ++--
5 files changed, 156 insertions(+), 65 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 21f8dbcf3f48..2dbc00904a84 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -679,7 +679,14 @@ struct bpf_tramp_id {
struct bpf_tramp_node {
struct hlist_node hlist_tramp;
+ struct hlist_node hlist_attach;
struct bpf_prog *prog;
+ struct bpf_trampoline *tr;
+};
+
+struct bpf_tramp_attach {
+ struct bpf_tramp_id *id;
+ struct hlist_head nodes;
};
struct bpf_trampoline {
@@ -751,9 +758,14 @@ void bpf_tramp_id_init(struct bpf_tramp_id *id,
struct btf *btf, u32 btf_id);
int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
-struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
- struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
+
+struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
+ struct bpf_prog *tgt_prog,
+ struct bpf_prog *prog);
+void bpf_tramp_detach(struct bpf_tramp_attach *attach);
+int bpf_tramp_attach_link(struct bpf_tramp_attach *attach);
+int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach);
#define BPF_DISPATCHER_INIT(_name) { \
.mutex = __MUTEX_INITIALIZER(_name.mutex), \
.func = &_name##_func, \
@@ -888,8 +900,8 @@ struct bpf_prog_aux {
const struct bpf_ctx_arg_aux *ctx_arg_info;
struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
struct bpf_prog *dst_prog;
- struct bpf_trampoline *dst_trampoline;
- struct bpf_trampoline *trampoline;
+ struct bpf_tramp_attach *dst_attach;
+ struct bpf_tramp_attach *attach;
enum bpf_prog_type saved_dst_prog_type;
enum bpf_attach_type saved_dst_attach_type;
bool verifier_zext; /* Zero extensions has been inserted by verifier. */
@@ -899,7 +911,6 @@ struct bpf_prog_aux {
bool sleepable;
bool tail_call_reachable;
bool multi_func;
- struct bpf_tramp_node tramp_node;
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
/* function name for valid attach_btf_id */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 2eed91153a3f..993ae224e371 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -105,7 +105,6 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
fp->aux = aux;
fp->aux->prog = fp;
fp->jit_requested = ebpf_jit_enabled();
- fp->aux->tramp_node.prog = fp;
INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
mutex_init(&fp->aux->used_maps_mutex);
@@ -2284,8 +2283,8 @@ static void bpf_prog_free_deferred(struct work_struct *work)
if (aux->prog->has_callchain_buf)
put_callchain_buffers();
#endif
- if (aux->dst_trampoline)
- bpf_trampoline_put(aux->dst_trampoline);
+ if (aux->dst_attach)
+ bpf_tramp_detach(aux->dst_attach);
for (i = 0; i < aux->func_cnt; i++) {
/* We can just unlink the subprog poke descriptor table as
* it was originally linked to the main program and is also
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0d916e3b7676..a65c1862ab68 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2644,32 +2644,32 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd)
struct bpf_tracing_link {
struct bpf_link link;
enum bpf_attach_type attach_type;
- struct bpf_trampoline *trampoline;
+ struct bpf_tramp_attach *attach;
struct bpf_prog *tgt_prog;
};
-static struct bpf_trampoline *link_trampoline(struct bpf_tracing_link *link)
+static struct bpf_tramp_attach *link_attach(struct bpf_tracing_link *link)
{
struct bpf_prog *prog = link->link.prog;
if (prog->type == BPF_PROG_TYPE_EXT)
- return link->trampoline;
+ return link->attach;
else
- return prog->aux->trampoline;
+ return prog->aux->attach;
}
static void bpf_tracing_link_release(struct bpf_link *link)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link);
- struct bpf_trampoline *tr = link_trampoline(tr_link);
+ struct bpf_tramp_attach *attach = link_attach(tr_link);
struct bpf_prog *prog = link->prog;
- WARN_ON_ONCE(bpf_trampoline_unlink_prog(&link->prog->aux->tramp_node, tr));
+ WARN_ON_ONCE(bpf_tramp_attach_unlink(attach));
if (prog->type != BPF_PROG_TYPE_EXT)
- prog->aux->trampoline = NULL;
- bpf_trampoline_put(tr);
+ prog->aux->attach = NULL;
+ bpf_tramp_detach(attach);
/* tgt_prog is NULL if target is a kernel function */
if (tr_link->tgt_prog)
@@ -2700,11 +2700,11 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link);
- struct bpf_trampoline *tr = link_trampoline(tr_link);
+ struct bpf_tramp_attach *attach = link_attach(tr_link);
info->tracing.attach_type = tr_link->attach_type;
- info->tracing.target_obj_id = tr->id->obj_id;
- info->tracing.target_btf_id = tr->id->btf_id;
+ info->tracing.target_obj_id = attach->id->obj_id;
+ info->tracing.target_btf_id = attach->id->btf_id;
return 0;
}
@@ -2721,9 +2721,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
u32 btf_id)
{
bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;
+ struct bpf_tramp_attach *attach = NULL;
struct bpf_link_primer link_primer;
struct bpf_prog *tgt_prog = NULL;
- struct bpf_trampoline *tr = NULL;
struct bpf_tracing_link *link;
struct bpf_tramp_id *id = NULL;
int err;
@@ -2793,7 +2793,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
mutex_lock(&prog->aux->dst_mutex);
- if (!prog_extension && prog->aux->trampoline) {
+ if (!prog_extension && prog->aux->attach) {
err = -EBUSY;
goto out_unlock;
}
@@ -2816,7 +2816,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
* - if prog->aux->dst_trampoline and tgt_prog is NULL, the program
* was detached and is going for re-attachment.
*/
- if (!prog->aux->dst_trampoline && !tgt_prog) {
+ if (!prog->aux->dst_attach && !tgt_prog) {
/*
* Allow re-attach for TRACING and LSM programs. If it's
* currently linked, bpf_trampoline_link_prog will fail.
@@ -2839,9 +2839,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id);
}
- if (!prog->aux->dst_trampoline ||
+ if (!prog->aux->dst_attach ||
(!bpf_tramp_id_is_empty(id) &&
- bpf_tramp_id_is_equal(id, prog->aux->dst_trampoline->id))) {
+ bpf_tramp_id_is_equal(id, prog->aux->dst_attach->id))) {
/* If there is no saved target, or the specified target is
* different from the destination specified at load time, we
* need a new trampoline and a check for compatibility
@@ -2853,9 +2853,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- tr = bpf_trampoline_get(id, &tgt_info);
- if (!tr) {
- err = -ENOMEM;
+ id->addr = (void *) tgt_info.tgt_addr;
+
+ attach = bpf_tramp_attach(id, tgt_prog, prog);
+ if (IS_ERR(attach)) {
+ err = PTR_ERR(attach);
goto out_unlock;
}
} else {
@@ -2866,7 +2868,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
* can only happen once for any program, as the saved values in
* prog->aux are cleared below.
*/
- tr = prog->aux->dst_trampoline;
+ attach = prog->aux->dst_attach;
tgt_prog = prog->aux->dst_prog;
}
@@ -2874,7 +2876,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- err = bpf_trampoline_link_prog(&prog->aux->tramp_node, tr);
+ err = bpf_tramp_attach_link(attach);
if (err) {
bpf_link_cleanup(&link_primer);
link = NULL;
@@ -2882,32 +2884,31 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
}
link->tgt_prog = tgt_prog;
-
if (prog_extension)
- link->trampoline = tr;
+ link->attach = attach;
else
- prog->aux->trampoline = tr;
+ prog->aux->attach = attach;
/* Always clear the trampoline and target prog from prog->aux to make
* sure the original attach destination is not kept alive after a
* program is (re-)attached to another target.
*/
if (prog->aux->dst_prog &&
- (tgt_prog_fd || tr != prog->aux->dst_trampoline))
+ (tgt_prog_fd || attach != prog->aux->dst_attach))
/* got extra prog ref from syscall, or attaching to different prog */
bpf_prog_put(prog->aux->dst_prog);
- if (prog->aux->dst_trampoline && tr != prog->aux->dst_trampoline)
+ if (prog->aux->dst_attach && attach != prog->aux->dst_attach)
/* we allocated a new trampoline, so free the old one */
- bpf_trampoline_put(prog->aux->dst_trampoline);
+ bpf_tramp_detach(prog->aux->dst_attach);
prog->aux->dst_prog = NULL;
- prog->aux->dst_trampoline = NULL;
+ prog->aux->dst_attach = NULL;
mutex_unlock(&prog->aux->dst_mutex);
return bpf_link_settle(&link_primer);
out_unlock:
- if (tr && tr != prog->aux->dst_trampoline)
- bpf_trampoline_put(tr);
+ if (attach && attach != prog->aux->dst_attach)
+ bpf_tramp_detach(attach);
mutex_unlock(&prog->aux->dst_mutex);
kfree(link);
out_put_prog:
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index b6af3e0982d7..16fc4c14319b 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -11,6 +11,7 @@
#include <linux/rcupdate_wait.h>
#include <linux/module.h>
#include <linux/static_call.h>
+#include <linux/bpf_verifier.h>
/* dummy _ops. The verifier will operate on target program's ops. */
const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -98,7 +99,7 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id)
kfree(id);
}
-static struct bpf_trampoline *bpf_trampoline_lookup(struct bpf_tramp_id *id)
+static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id)
{
struct bpf_trampoline *tr;
struct hlist_head *head;
@@ -528,26 +529,6 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin
return err;
}
-struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
- struct bpf_attach_target_info *tgt_info)
-{
- struct bpf_trampoline *tr;
-
- tr = bpf_trampoline_lookup(id);
- if (!tr)
- return NULL;
-
- mutex_lock(&tr->mutex);
- if (tr->id->addr)
- goto out;
-
- memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
- tr->id->addr = (void *)tgt_info->tgt_addr;
-out:
- mutex_unlock(&tr->mutex);
- return tr;
-}
-
void bpf_trampoline_put(struct bpf_trampoline *tr)
{
if (!tr)
@@ -567,12 +548,109 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
* multiple rcu callbacks.
*/
hlist_del(&tr->hlist);
- bpf_tramp_id_free(tr->id);
kfree(tr);
out:
mutex_unlock(&trampoline_mutex);
}
+static struct bpf_tramp_node *node_alloc(struct bpf_trampoline *tr, struct bpf_prog *prog)
+{
+ struct bpf_tramp_node *node;
+
+ node = kzalloc(sizeof(*node), GFP_KERNEL);
+ if (!node)
+ return NULL;
+
+ INIT_HLIST_NODE(&node->hlist_tramp);
+ INIT_HLIST_NODE(&node->hlist_attach);
+ node->prog = prog;
+ node->tr = tr;
+ return node;
+}
+
+static void node_free(struct bpf_tramp_node *node)
+{
+ bpf_trampoline_put(node->tr);
+ kfree(node);
+}
+
+struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
+ struct bpf_prog *tgt_prog,
+ struct bpf_prog *prog)
+{
+ struct bpf_trampoline *tr = NULL;
+ struct bpf_tramp_attach *attach;
+ struct bpf_tramp_node *node;
+ int err;
+
+ attach = kzalloc(sizeof(*attach), GFP_KERNEL);
+ if (!attach)
+ return ERR_PTR(-ENOMEM);
+
+ tr = bpf_trampoline_get(id);
+ if (!tr) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ node = node_alloc(tr, prog);
+ if (!node)
+ goto out;
+
+ err = bpf_check_attach_model(prog, tgt_prog, id->btf_id, &tr->func.model);
+ if (err)
+ goto out;
+
+ attach->id = id;
+ hlist_add_head(&node->hlist_attach, &attach->nodes);
+ return attach;
+
+out:
+ bpf_trampoline_put(tr);
+ kfree(attach);
+ return ERR_PTR(err);
+}
+
+void bpf_tramp_detach(struct bpf_tramp_attach *attach)
+{
+ struct bpf_tramp_node *node;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach)
+ node_free(node);
+
+ bpf_tramp_id_free(attach->id);
+ kfree(attach);
+}
+
+int bpf_tramp_attach_link(struct bpf_tramp_attach *attach)
+{
+ struct bpf_tramp_node *node;
+ int err;
+
+ hlist_for_each_entry(node, &attach->nodes, hlist_attach) {
+ err = bpf_trampoline_link_prog(node, node->tr);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach)
+{
+ struct bpf_tramp_node *node;
+ int err;
+
+ hlist_for_each_entry(node, &attach->nodes, hlist_attach) {
+ err = bpf_trampoline_unlink_prog(node, node->tr);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
#define NO_START_TIME 1
static __always_inline u64 notrace bpf_prog_start_time(void)
{
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a1e4389b0e9e..e05f39fd2708 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13928,7 +13928,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
struct bpf_attach_target_info tgt_info = {};
u32 btf_id = prog->aux->attach_btf_id;
- struct bpf_trampoline *tr;
+ struct bpf_tramp_attach *attach;
struct bpf_tramp_id *id;
int ret;
@@ -14000,13 +14000,15 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -ENOMEM;
bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id);
- tr = bpf_trampoline_get(id, &tgt_info);
- if (!tr) {
+ id->addr = (void *) tgt_info.tgt_addr;
+
+ attach = bpf_tramp_attach(id, tgt_prog, prog);
+ if (IS_ERR(attach)) {
bpf_tramp_id_free(id);
- return -ENOMEM;
+ return PTR_ERR(attach);
}
- prog->aux->dst_trampoline = tr;
+ prog->aux->dst_attach = attach;
return 0;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 14/29] bpf: Add support to store multiple ids in bpf_tramp_id object
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (12 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 13/29] bpf: Add bpf_tramp_attach layer for trampoline attachment Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 15/29] bpf: Add support to store multiple addrs " Jiri Olsa
` (14 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding support to store multiple ids in bpf_tramp_id object,
to have id for trampolines with multiple functions assigned.
Extra array of u32 values is allocated within bpf_tramp_id
object allocation.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 6 ++++--
kernel/bpf/syscall.c | 6 +++---
kernel/bpf/trampoline.c | 39 +++++++++++++++++++++++++++++++--------
kernel/bpf/verifier.c | 2 +-
4 files changed, 39 insertions(+), 14 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 2dbc00904a84..47e25d8be600 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -672,8 +672,10 @@ struct bpf_tramp_image {
};
struct bpf_tramp_id {
+ u32 max;
+ u32 cnt;
u32 obj_id;
- u32 btf_id;
+ u32 *id;
void *addr;
};
@@ -749,7 +751,7 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
return bpf_func(ctx, insnsi);
}
#ifdef CONFIG_BPF_JIT
-struct bpf_tramp_id *bpf_tramp_id_alloc(void);
+struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt);
void bpf_tramp_id_free(struct bpf_tramp_id *id);
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id);
int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index a65c1862ab68..216fcce07326 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2704,7 +2704,7 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
info->tracing.attach_type = tr_link->attach_type;
info->tracing.target_obj_id = attach->id->obj_id;
- info->tracing.target_btf_id = attach->id->btf_id;
+ info->tracing.target_btf_id = attach->id->id[0];
return 0;
}
@@ -2766,7 +2766,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
- id = bpf_tramp_id_alloc();
+ id = bpf_tramp_id_alloc(1);
if (!id) {
err = -ENOMEM;
goto out_put_prog;
@@ -2829,7 +2829,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_unlock;
}
- id = bpf_tramp_id_alloc();
+ id = bpf_tramp_id_alloc(1);
if (!id) {
err = -ENOMEM;
goto out_unlock;
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 16fc4c14319b..d65f463c532d 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -60,27 +60,45 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
PAGE_SIZE, true, ksym->name);
}
+static bool bpf_tramp_id_is_multi(struct bpf_tramp_id *id)
+{
+ return id->cnt > 1;
+}
+
static u64 bpf_tramp_id_key(struct bpf_tramp_id *id)
{
- return ((u64) id->obj_id << 32) | id->btf_id;
+ if (bpf_tramp_id_is_multi(id))
+ return (u64) &id;
+ else
+ return ((u64) id->obj_id << 32) | id->id[0];
}
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id)
{
- return !id || (!id->obj_id && !id->btf_id);
+ return !id || id->cnt == 0;
}
int bpf_tramp_id_is_equal(struct bpf_tramp_id *a,
struct bpf_tramp_id *b)
{
- return !memcmp(a, b, sizeof(*a));
+ return a->obj_id == b->obj_id && a->cnt == b->cnt &&
+ !memcmp(a->id, b->id, a->cnt * sizeof(*a->id));
}
-struct bpf_tramp_id *bpf_tramp_id_alloc(void)
+struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max)
{
struct bpf_tramp_id *id;
- return kzalloc(sizeof(*id), GFP_KERNEL);
+ id = kzalloc(sizeof(*id), GFP_KERNEL);
+ if (id) {
+ id->id = kzalloc(sizeof(u32) * max, GFP_KERNEL);
+ if (!id->id) {
+ kfree(id);
+ return NULL;
+ }
+ id->max = max;
+ }
+ return id;
}
void bpf_tramp_id_init(struct bpf_tramp_id *id,
@@ -91,11 +109,15 @@ void bpf_tramp_id_init(struct bpf_tramp_id *id,
id->obj_id = tgt_prog->aux->id;
else
id->obj_id = btf_obj_id(btf);
- id->btf_id = btf_id;
+ id->id[0] = btf_id;
+ id->cnt = 1;
}
void bpf_tramp_id_free(struct bpf_tramp_id *id)
{
+ if (!id)
+ return;
+ kfree(id->id);
kfree(id);
}
@@ -362,7 +384,8 @@ bpf_tramp_image_alloc(struct bpf_tramp_id *id, u32 idx)
ksym = &im->ksym;
INIT_LIST_HEAD_RCU(&ksym->lnode);
key = bpf_tramp_id_key(id);
- snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u", key, idx);
+ snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u%s", key, idx,
+ bpf_tramp_id_is_multi(id) ? "_multi" : "");
bpf_image_ksym_add(image, ksym);
return im;
@@ -597,7 +620,7 @@ struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
if (!node)
goto out;
- err = bpf_check_attach_model(prog, tgt_prog, id->btf_id, &tr->func.model);
+ err = bpf_check_attach_model(prog, tgt_prog, id->id[0], &tr->func.model);
if (err)
goto out;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e05f39fd2708..1903d5d256b6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13995,7 +13995,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
- id = bpf_tramp_id_alloc();
+ id = bpf_tramp_id_alloc(1);
if (!id)
return -ENOMEM;
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 15/29] bpf: Add support to store multiple addrs in bpf_tramp_id object
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (13 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 14/29] bpf: Add support to store multiple ids in bpf_tramp_id object Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 16/29] bpf: Add bpf_tramp_id_single function Jiri Olsa
` (13 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding support to store multiple addrs in bpf_tramp_id object,
to provide address values for id values stored in the object.
The id->addr[idx] returns address value for id->id[idx] id.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 2 +-
kernel/bpf/syscall.c | 2 +-
kernel/bpf/trampoline.c | 20 ++++++++++++--------
kernel/bpf/verifier.c | 2 +-
4 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 47e25d8be600..13e9dcfd47e7 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -676,7 +676,7 @@ struct bpf_tramp_id {
u32 cnt;
u32 obj_id;
u32 *id;
- void *addr;
+ void **addr;
};
struct bpf_tramp_node {
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 216fcce07326..0ae3b5b7419a 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2853,7 +2853,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- id->addr = (void *) tgt_info.tgt_addr;
+ id->addr[0] = (void *) tgt_info.tgt_addr;
attach = bpf_tramp_attach(id, tgt_prog, prog);
if (IS_ERR(attach)) {
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index d65f463c532d..d9675d619963 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -92,7 +92,10 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max)
id = kzalloc(sizeof(*id), GFP_KERNEL);
if (id) {
id->id = kzalloc(sizeof(u32) * max, GFP_KERNEL);
- if (!id->id) {
+ id->addr = kzalloc(sizeof(*id->addr) * max, GFP_KERNEL);
+ if (!id->id || !id->addr) {
+ kfree(id->id);
+ kfree(id->addr);
kfree(id);
return NULL;
}
@@ -117,6 +120,7 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id)
{
if (!id)
return;
+ kfree(id->addr);
kfree(id->id);
kfree(id);
}
@@ -159,7 +163,7 @@ static int bpf_trampoline_module_get(struct bpf_trampoline *tr)
int err = 0;
preempt_disable();
- mod = __module_text_address((unsigned long) tr->id->addr);
+ mod = __module_text_address((unsigned long) tr->id->addr[0]);
if (mod && !try_module_get(mod))
err = -ENOENT;
preempt_enable();
@@ -187,7 +191,7 @@ static int is_ftrace_location(void *ip)
static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
{
- void *ip = tr->id->addr;
+ void *ip = tr->id->addr[0];
int ret;
if (tr->func.ftrace_managed)
@@ -202,7 +206,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_addr)
{
- void *ip = tr->id->addr;
+ void *ip = tr->id->addr[0];
int ret;
if (tr->func.ftrace_managed)
@@ -215,7 +219,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
/* first time registering */
static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
{
- void *ip = tr->id->addr;
+ void *ip = tr->id->addr[0];
int ret;
ret = is_ftrace_location(ip);
@@ -434,7 +438,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr)
err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE,
&tr->func.model, flags, tprogs,
- tr->id->addr);
+ tr->id->addr[0]);
if (err < 0)
goto out;
@@ -503,7 +507,7 @@ int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline
goto out;
}
tr->extension_prog = prog;
- err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, NULL,
+ err = bpf_arch_text_poke(tr->id->addr[0], BPF_MOD_JUMP, NULL,
prog->bpf_func);
goto out;
}
@@ -539,7 +543,7 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin
mutex_lock(&tr->mutex);
if (kind == BPF_TRAMP_REPLACE) {
WARN_ON_ONCE(!tr->extension_prog);
- err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP,
+ err = bpf_arch_text_poke(tr->id->addr[0], BPF_MOD_JUMP,
tr->extension_prog->bpf_func, NULL);
tr->extension_prog = NULL;
goto out;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1903d5d256b6..56c518efa2d2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14000,7 +14000,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -ENOMEM;
bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id);
- id->addr = (void *) tgt_info.tgt_addr;
+ id->addr[0] = (void *) tgt_info.tgt_addr;
attach = bpf_tramp_attach(id, tgt_prog, prog);
if (IS_ERR(attach)) {
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 16/29] bpf: Add bpf_tramp_id_single function
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (14 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 15/29] bpf: Add support to store multiple addrs " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 17/29] bpf: Resolve id in bpf_tramp_id_single Jiri Olsa
` (12 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding bpf_tramp_id_single function as interface to
create trampoline with single ID and grouping together
the trampoline allocation with init that is used on
several places and save us few lines.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 5 ++---
kernel/bpf/syscall.c | 18 +++++++-----------
kernel/bpf/trampoline.c | 11 ++++++++---
kernel/bpf/verifier.c | 3 +--
4 files changed, 18 insertions(+), 19 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 13e9dcfd47e7..894ee812e213 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -755,9 +755,8 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt);
void bpf_tramp_id_free(struct bpf_tramp_id *id);
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id);
int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
-void bpf_tramp_id_init(struct bpf_tramp_id *id,
- const struct bpf_prog *tgt_prog,
- struct btf *btf, u32 btf_id);
+struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
+ struct btf *btf, u32 btf_id);
int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
void bpf_trampoline_put(struct bpf_trampoline *tr);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0ae3b5b7419a..8109b0fc7d2f 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2766,12 +2766,6 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
- id = bpf_tramp_id_alloc(1);
- if (!id) {
- err = -ENOMEM;
- goto out_put_prog;
- }
-
tgt_prog = bpf_prog_get(tgt_prog_fd);
if (IS_ERR(tgt_prog)) {
err = PTR_ERR(tgt_prog);
@@ -2779,7 +2773,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
- bpf_tramp_id_init(id, tgt_prog, NULL, btf_id);
+ id = bpf_tramp_id_single(tgt_prog, NULL, btf_id);
+ if (!id) {
+ err = -ENOMEM;
+ goto out_put_prog;
+ }
}
link = kzalloc(sizeof(*link), GFP_USER);
@@ -2829,14 +2827,12 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_unlock;
}
- id = bpf_tramp_id_alloc(1);
+ btf_id = prog->aux->attach_btf_id;
+ id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id);
if (!id) {
err = -ENOMEM;
goto out_unlock;
}
-
- btf_id = prog->aux->attach_btf_id;
- bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id);
}
if (!prog->aux->dst_attach ||
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index d9675d619963..4a4ef9396b7e 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -104,16 +104,21 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max)
return id;
}
-void bpf_tramp_id_init(struct bpf_tramp_id *id,
- const struct bpf_prog *tgt_prog,
- struct btf *btf, u32 btf_id)
+struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
+ struct btf *btf, u32 btf_id)
{
+ struct bpf_tramp_id *id;
+
+ id = bpf_tramp_id_alloc(1);
+ if (!id)
+ return NULL;
if (tgt_prog)
id->obj_id = tgt_prog->aux->id;
else
id->obj_id = btf_obj_id(btf);
id->id[0] = btf_id;
id->cnt = 1;
+ return id;
}
void bpf_tramp_id_free(struct bpf_tramp_id *id)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 56c518efa2d2..9914487f2281 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13995,11 +13995,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
- id = bpf_tramp_id_alloc(1);
+ id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id);
if (!id)
return -ENOMEM;
- bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id);
id->addr[0] = (void *) tgt_info.tgt_addr;
attach = bpf_tramp_attach(id, tgt_prog, prog);
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 17/29] bpf: Resolve id in bpf_tramp_id_single
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (15 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 16/29] bpf: Add bpf_tramp_id_single function Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 18/29] bpf: Add refcount_t to struct bpf_tramp_id Jiri Olsa
` (11 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Moving the id resolving in the bpf_tramp_id_single function
so it's centralized together with the trampoline's allocation
and init.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 3 ++-
kernel/bpf/syscall.c | 21 ++++++---------------
kernel/bpf/trampoline.c | 18 +++++++++++++++---
kernel/bpf/verifier.c | 4 +---
4 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 894ee812e213..dda24339e4b1 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -756,7 +756,8 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id);
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id);
int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
- struct btf *btf, u32 btf_id);
+ struct bpf_prog *prog, u32 btf_id,
+ struct bpf_attach_target_info *tgt_info);
int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
void bpf_trampoline_put(struct bpf_trampoline *tr);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8109b0fc7d2f..0acf6cb0fdc7 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2773,9 +2773,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
goto out_put_prog;
}
- id = bpf_tramp_id_single(tgt_prog, NULL, btf_id);
- if (!id) {
- err = -ENOMEM;
+ id = bpf_tramp_id_single(tgt_prog, prog, btf_id, NULL);
+ if (IS_ERR(id)) {
+ err = PTR_ERR(id);
goto out_put_prog;
}
}
@@ -2828,9 +2828,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
}
btf_id = prog->aux->attach_btf_id;
- id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id);
- if (!id) {
- err = -ENOMEM;
+ id = bpf_tramp_id_single(NULL, prog, btf_id, NULL);
+ if (IS_ERR(id)) {
+ err = PTR_ERR(id);
goto out_unlock;
}
}
@@ -2842,15 +2842,6 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
* different from the destination specified at load time, we
* need a new trampoline and a check for compatibility
*/
- struct bpf_attach_target_info tgt_info = {};
-
- err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id,
- &tgt_info);
- if (err)
- goto out_unlock;
-
- id->addr[0] = (void *) tgt_info.tgt_addr;
-
attach = bpf_tramp_attach(id, tgt_prog, prog);
if (IS_ERR(attach)) {
err = PTR_ERR(attach);
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 4a4ef9396b7e..e6a73088ecee 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -105,18 +105,30 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max)
}
struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
- struct btf *btf, u32 btf_id)
+ struct bpf_prog *prog, u32 btf_id,
+ struct bpf_attach_target_info *tgt_info)
{
struct bpf_tramp_id *id;
+ if (!tgt_info) {
+ struct bpf_attach_target_info __tgt_info = {};
+ int err;
+
+ tgt_info = &__tgt_info;
+ err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id,
+ tgt_info);
+ if (err)
+ return ERR_PTR(err);
+ }
id = bpf_tramp_id_alloc(1);
if (!id)
- return NULL;
+ return ERR_PTR(-ENOMEM);
if (tgt_prog)
id->obj_id = tgt_prog->aux->id;
else
- id->obj_id = btf_obj_id(btf);
+ id->obj_id = btf_obj_id(prog->aux->attach_btf);
id->id[0] = btf_id;
+ id->addr[0] = (void *) tgt_info->tgt_addr;
id->cnt = 1;
return id;
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9914487f2281..8d56d43489aa 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13995,12 +13995,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
- id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id);
+ id = bpf_tramp_id_single(tgt_prog, prog, btf_id, &tgt_info);
if (!id)
return -ENOMEM;
- id->addr[0] = (void *) tgt_info.tgt_addr;
-
attach = bpf_tramp_attach(id, tgt_prog, prog);
if (IS_ERR(attach)) {
bpf_tramp_id_free(id);
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 18/29] bpf: Add refcount_t to struct bpf_tramp_id
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (16 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 17/29] bpf: Resolve id in bpf_tramp_id_single Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 19/29] bpf: Add support to attach trampolines with multiple IDs Jiri Olsa
` (10 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding refcount_t to struct bpf_tramp_id so we can
track its allocation and safely use one object on
more places in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 3 +++
kernel/bpf/syscall.c | 2 +-
kernel/bpf/trampoline.c | 16 +++++++++++++---
kernel/bpf/verifier.c | 2 +-
4 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index dda24339e4b1..04ada1d2495e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -23,6 +23,7 @@
#include <linux/slab.h>
#include <linux/percpu-refcount.h>
#include <linux/bpfptr.h>
+#include <linux/refcount.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
@@ -677,6 +678,7 @@ struct bpf_tramp_id {
u32 obj_id;
u32 *id;
void **addr;
+ refcount_t refcnt;
};
struct bpf_tramp_node {
@@ -753,6 +755,7 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
#ifdef CONFIG_BPF_JIT
struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt);
void bpf_tramp_id_free(struct bpf_tramp_id *id);
+void bpf_tramp_id_put(struct bpf_tramp_id *id);
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id);
int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b);
struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0acf6cb0fdc7..bfbd81869818 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2901,7 +2901,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
out_put_prog:
if (tgt_prog_fd && tgt_prog)
bpf_prog_put(tgt_prog);
- bpf_tramp_id_free(id);
+ bpf_tramp_id_put(id);
return err;
}
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index e6a73088ecee..39600fb78c9e 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -100,6 +100,7 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max)
return NULL;
}
id->max = max;
+ refcount_set(&id->refcnt, 1);
}
return id;
}
@@ -133,10 +134,18 @@ struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
return id;
}
-void bpf_tramp_id_free(struct bpf_tramp_id *id)
+static struct bpf_tramp_id *bpf_tramp_id_get(struct bpf_tramp_id *id)
+{
+ refcount_inc(&id->refcnt);
+ return id;
+}
+
+void bpf_tramp_id_put(struct bpf_tramp_id *id)
{
if (!id)
return;
+ if (!refcount_dec_and_test(&id->refcnt))
+ return;
kfree(id->addr);
kfree(id->id);
kfree(id);
@@ -162,7 +171,7 @@ static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id)
if (!tr)
goto out;
- tr->id = id;
+ tr->id = bpf_tramp_id_get(id);
INIT_HLIST_NODE(&tr->hlist);
hlist_add_head(&tr->hlist, head);
refcount_set(&tr->refcnt, 1);
@@ -592,6 +601,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
* multiple rcu callbacks.
*/
hlist_del(&tr->hlist);
+ bpf_tramp_id_put(tr->id);
kfree(tr);
out:
mutex_unlock(&trampoline_mutex);
@@ -663,7 +673,7 @@ void bpf_tramp_detach(struct bpf_tramp_attach *attach)
hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach)
node_free(node);
- bpf_tramp_id_free(attach->id);
+ bpf_tramp_id_put(attach->id);
kfree(attach);
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8d56d43489aa..6a87180ac2bb 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14001,7 +14001,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
attach = bpf_tramp_attach(id, tgt_prog, prog);
if (IS_ERR(attach)) {
- bpf_tramp_id_free(id);
+ bpf_tramp_id_put(id);
return PTR_ERR(attach);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 19/29] bpf: Add support to attach trampolines with multiple IDs
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (17 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 18/29] bpf: Add refcount_t to struct bpf_tramp_id Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 20/29] bpf: Add support for tracing multi link Jiri Olsa
` (9 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding support to attach trampolines with multiple IDs.
This patch adds support to bpf_tramp_attach function to
attach given program to bpf_tramp_id object that holds
multiple BTF function IDs.
The process of attaching in bpf_tramp_attach is as follows:
- IDs in bpf_tramp_id object are sorted out to several new
bpf_tramp_id objects based on number of arguments of each
ID - so we end up with up to 6 bpf_tramp_id objects, that
we will create or find suitable trampoline for
- separating function IDs that have same number of arguments
save us troubles of handling different argument counts
within one trampoline
- now for each such bpf_tramp_id object we do following:
* search existing trampolines to find match or intersection
* if there's full match on IDs, we add program to existing
trampoline and we are done
* if there's intersection with existing trampoline,
we split it and add new program to the common part,
the rest of the IDs are attached to new trampoline
- we keep trampoline_table as place holder for all trampolines,
(while the has works only for single ID trampolines) so in case
there is no multi-id trampoline defined, we still use the fast
hash trampoline lookup
The bpf_tramp_attach assumes ID array is coming in sorted so it's
possible to run bsearch on it to do all the needed searches.
The splitting of the trampoline use the fact that we carry
'bpf_tramp_attach' object for each bpf_program, so when we split
trampoline that the program is attached to, we just add new
'bpf_tramp_node' object to the program's attach 'nodes'. This way
we keep track of all program's trampolines and it will be properly
detached when the program goes away.
The splitting of the trampoline is done with following steps:
- lock the trampoline
- unregister trampoline
- alloc the duplicate, which means that for all attached programs
of the original trampoline we create new bpf_tramp_node objects
and add them to these programs' attach objects
- then we assign new IDs (common and the rest) to both (original
and the duplicated) trampolines
- register both trampolines
- unlock the original trampoline
This patch only adds bpf_tramp_attach support to attach multiple
ID bpf_tramp_id object. The actual user interface for that comes
in following patch.
Now when each call to bpf_tramp_attach can change any program's attach
object, we need to take trampoline_mutex in both bpf_tramp_attach_link
and bpf_tramp_attach_unlink functions. Perhaps we could add new lock
to bpf_tramp_attach object to get rid of single lock for all.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 12 +-
kernel/bpf/trampoline.c | 717 +++++++++++++++++++++++++++++++++++++---
2 files changed, 674 insertions(+), 55 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 04ada1d2495e..6ceb3bb39e1d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -24,6 +24,13 @@
#include <linux/percpu-refcount.h>
#include <linux/bpfptr.h>
#include <linux/refcount.h>
+#ifdef CONFIG_FUNCTION_TRACER
+#ifndef CC_USING_FENTRY
+#define CC_USING_FENTRY
+#endif
+#endif
+#include <linux/ftrace.h>
+#include <linux/types.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
@@ -703,6 +710,9 @@ struct bpf_trampoline {
struct {
struct btf_func_model model;
bool ftrace_managed;
+#ifdef CONFIG_FUNCTION_TRACER
+ struct ftrace_ops ops;
+#endif
} func;
/* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF
* program by replacing one of its functions. id->addr is the address
@@ -763,7 +773,6 @@ struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog,
struct bpf_attach_target_info *tgt_info);
int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr);
-void bpf_trampoline_put(struct bpf_trampoline *tr);
struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
struct bpf_prog *tgt_prog,
@@ -831,7 +840,6 @@ static inline struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id,
{
return ERR_PTR(-EOPNOTSUPP);
}
-static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {}
#define DEFINE_BPF_DISPATCHER(name)
#define DECLARE_BPF_DISPATCHER(name)
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_nop_func
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 39600fb78c9e..7a9e3126e256 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -12,6 +12,8 @@
#include <linux/module.h>
#include <linux/static_call.h>
#include <linux/bpf_verifier.h>
+#include <linux/bsearch.h>
+#include <linux/minmax.h>
/* dummy _ops. The verifier will operate on target program's ops. */
const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -24,8 +26,9 @@ const struct bpf_prog_ops bpf_extension_prog_ops = {
#define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS)
static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE];
+static int nr_bpf_trampoline_multi;
-/* serializes access to trampoline_table */
+/* serializes access to trampoline_table, nr_bpf_trampoline_multi */
static DEFINE_MUTEX(trampoline_mutex);
void *bpf_jit_alloc_exec_page(void)
@@ -62,15 +65,12 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
static bool bpf_tramp_id_is_multi(struct bpf_tramp_id *id)
{
- return id->cnt > 1;
+ return id && id->cnt > 1;
}
static u64 bpf_tramp_id_key(struct bpf_tramp_id *id)
{
- if (bpf_tramp_id_is_multi(id))
- return (u64) &id;
- else
- return ((u64) id->obj_id << 32) | id->id[0];
+ return ((u64) id->obj_id << 32) | id->id[0];
}
bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id)
@@ -151,26 +151,14 @@ void bpf_tramp_id_put(struct bpf_tramp_id *id)
kfree(id);
}
-static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id)
+static void bpf_trampoline_init(struct bpf_trampoline *tr, struct bpf_tramp_id *id)
{
- struct bpf_trampoline *tr;
struct hlist_head *head;
u64 key;
int i;
key = bpf_tramp_id_key(id);
- mutex_lock(&trampoline_mutex);
head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
- hlist_for_each_entry(tr, head, hlist) {
- if (bpf_tramp_id_is_equal(tr->id, id)) {
- refcount_inc(&tr->refcnt);
- goto out;
- }
- }
- tr = kzalloc(sizeof(*tr), GFP_KERNEL);
- if (!tr)
- goto out;
-
tr->id = bpf_tramp_id_get(id);
INIT_HLIST_NODE(&tr->hlist);
hlist_add_head(&tr->hlist, head);
@@ -178,11 +166,39 @@ static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id)
mutex_init(&tr->mutex);
for (i = 0; i < BPF_TRAMP_MAX; i++)
INIT_HLIST_HEAD(&tr->progs_hlist[i]);
-out:
- mutex_unlock(&trampoline_mutex);
+ if (bpf_tramp_id_is_multi(id))
+ nr_bpf_trampoline_multi++;
+}
+
+static struct bpf_trampoline *bpf_trampoline_alloc(struct bpf_tramp_id *id)
+{
+ struct bpf_trampoline *tr;
+
+ tr = kzalloc(sizeof(*tr), GFP_KERNEL);
+ if (!tr)
+ return NULL;
+
+ bpf_trampoline_init(tr, id);
return tr;
}
+static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id)
+{
+ struct bpf_trampoline *tr;
+ struct hlist_head *head;
+ u64 key;
+
+ key = bpf_tramp_id_key(id);
+ head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
+ hlist_for_each_entry(tr, head, hlist) {
+ if (bpf_tramp_id_is_equal(tr->id, id)) {
+ refcount_inc(&tr->refcnt);
+ return tr;
+ }
+ }
+ return bpf_trampoline_alloc(id);
+}
+
static int bpf_trampoline_module_get(struct bpf_trampoline *tr)
{
struct module *mod;
@@ -220,6 +236,9 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
void *ip = tr->id->addr[0];
int ret;
+ if (bpf_tramp_id_is_multi(tr->id))
+ return unregister_ftrace_direct_multi(&tr->func.ops, (long) old_addr);
+
if (tr->func.ftrace_managed)
ret = unregister_ftrace_direct((long)ip, (long)old_addr);
else
@@ -235,6 +254,9 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
void *ip = tr->id->addr[0];
int ret;
+ if (bpf_tramp_id_is_multi(tr->id))
+ return modify_ftrace_direct_multi(&tr->func.ops, (long) new_addr);
+
if (tr->func.ftrace_managed)
ret = modify_ftrace_direct((long)ip, (long)old_addr, (long)new_addr);
else
@@ -248,6 +270,9 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
void *ip = tr->id->addr[0];
int ret;
+ if (bpf_tramp_id_is_multi(tr->id))
+ return register_ftrace_direct_multi(&tr->func.ops, (long) new_addr);
+
ret = is_ftrace_location(ip);
if (ret < 0)
return ret;
@@ -435,17 +460,19 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr)
struct bpf_tramp_progs *tprogs;
u32 flags = BPF_TRAMP_F_RESTORE_REGS;
bool ip_arg = false;
- int err, total;
+ int err = 0, total;
tprogs = bpf_trampoline_get_progs(tr, &total, &ip_arg);
if (IS_ERR(tprogs))
return PTR_ERR(tprogs);
if (total == 0) {
- err = unregister_fentry(tr, tr->cur_image->image);
- bpf_tramp_image_put(tr->cur_image);
- tr->cur_image = NULL;
- tr->selector = 0;
+ if (tr->cur_image) {
+ err = unregister_fentry(tr, tr->cur_image->image);
+ bpf_tramp_image_put(tr->cur_image);
+ tr->cur_image = NULL;
+ tr->selector = 0;
+ }
goto out;
}
@@ -456,8 +483,11 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr)
}
if (tprogs[BPF_TRAMP_FEXIT].nr_progs ||
- tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs)
+ tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) {
flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME;
+ if (bpf_tramp_id_is_multi(tr->id))
+ flags |= BPF_TRAMP_F_ORIG_STACK;
+ }
if (ip_arg)
flags |= BPF_TRAMP_F_IP_ARG;
@@ -582,29 +612,29 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin
return err;
}
-void bpf_trampoline_put(struct bpf_trampoline *tr)
+static void bpf_trampoline_put(struct bpf_trampoline *tr)
{
if (!tr)
return;
- mutex_lock(&trampoline_mutex);
if (!refcount_dec_and_test(&tr->refcnt))
- goto out;
+ return;
WARN_ON_ONCE(mutex_is_locked(&tr->mutex));
if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FENTRY])))
- goto out;
+ return;
if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))
- goto out;
+ return;
/* This code will be executed even when the last bpf_tramp_image
* is alive. All progs are detached from the trampoline and the
* trampoline image is patched with jmp into epilogue to skip
* fexit progs. The fentry-only trampoline will be freed via
* multiple rcu callbacks.
*/
+ if (bpf_tramp_id_is_multi(tr->id))
+ nr_bpf_trampoline_multi--;
hlist_del(&tr->hlist);
bpf_tramp_id_put(tr->id);
+ ftrace_free_filter(&tr->func.ops);
kfree(tr);
-out:
- mutex_unlock(&trampoline_mutex);
}
static struct bpf_tramp_node *node_alloc(struct bpf_trampoline *tr, struct bpf_prog *prog)
@@ -628,18 +658,442 @@ static void node_free(struct bpf_tramp_node *node)
kfree(node);
}
-struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
- struct bpf_prog *tgt_prog,
- struct bpf_prog *prog)
+static void bpf_func_model_nargs(struct btf_func_model *m, int nr_args)
+{
+ int i;
+
+ for (i = 0; i < nr_args; i++)
+ m->arg_size[i] = 8;
+ m->ret_size = 8;
+ m->nr_args = nr_args;
+}
+
+struct attach_args {
+ int nr_args;
+ struct bpf_prog *tgt_prog;
+ struct bpf_prog *prog;
+};
+
+static int bpf_trampoline_setup(struct bpf_trampoline *tr,
+ struct attach_args *att)
+{
+ struct bpf_tramp_id *id = tr->id;
+
+ if (bpf_tramp_id_is_multi(id)) {
+ bpf_func_model_nargs(&tr->func.model, att->nr_args);
+ return ftrace_set_filter_ips(&tr->func.ops, (long*) id->addr,
+ id->cnt, 0, 1);
+ } else {
+ return bpf_check_attach_model(att->prog, att->tgt_prog,
+ id->id[0], &tr->func.model);
+ }
+}
+
+static int
+bpf_trampoline_create(struct bpf_tramp_attach *attach,
+ struct bpf_tramp_id *id, struct attach_args *att)
{
struct bpf_trampoline *tr = NULL;
- struct bpf_tramp_attach *attach;
struct bpf_tramp_node *node;
int err;
- attach = kzalloc(sizeof(*attach), GFP_KERNEL);
- if (!attach)
- return ERR_PTR(-ENOMEM);
+ tr = bpf_trampoline_alloc(id);
+ if (!tr) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ err = bpf_trampoline_setup(tr, att);
+ if (err)
+ goto out;
+
+ node = node_alloc(tr, att->prog);
+ if (!node) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ hlist_add_head(&node->hlist_attach, &attach->nodes);
+ return 0;
+
+out:
+ bpf_trampoline_put(tr);
+ return err;
+}
+
+static void bpf_trampoline_dup_destroy(struct bpf_trampoline *tr)
+{
+ struct bpf_tramp_node *node;
+ struct hlist_node *n;
+ int kind;
+
+ if (!tr)
+ return;
+
+ for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
+ hlist_for_each_entry_safe(node, n, &tr->progs_hlist[kind],
+ hlist_tramp) {
+ hlist_del(&node->hlist_tramp);
+ hlist_del(&node->hlist_attach);
+ node_free(node);
+ }
+ }
+
+ WARN_ON_ONCE(refcount_read(&tr->refcnt) != 1);
+ bpf_trampoline_put(tr);
+}
+
+static struct bpf_trampoline*
+bpf_trampoline_dup(struct bpf_trampoline *tr, struct bpf_tramp_id *id)
+{
+ struct bpf_tramp_node *node, *iter;
+ struct bpf_trampoline *dup;
+ int kind;
+
+ /* Allocate new trampoline and duplicate all
+ * the program attachments it has.
+ */
+ dup = bpf_trampoline_alloc(id);
+ if (!dup)
+ return NULL;
+
+ dup->refcnt = tr->refcnt;
+
+ for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
+ hlist_for_each_entry(iter, &tr->progs_hlist[kind], hlist_tramp) {
+ struct bpf_prog *prog = iter->prog;
+
+ node = node_alloc(dup, prog);
+ if (!node)
+ goto out_free;
+ hlist_add_head(&node->hlist_tramp, &dup->progs_hlist[kind]);
+ hlist_add_head(&node->hlist_attach, &prog->aux->attach->nodes);
+ dup->progs_cnt[kind]++;
+ }
+ }
+ return dup;
+
+out_free:
+ bpf_trampoline_dup_destroy(dup);
+ return NULL;
+}
+
+static int btf_id_cmp(const void *a, const void *b)
+{
+ const u32 *x = a;
+ const u32 *y = b;
+
+ if (*x == *y)
+ return 0;
+ return *x < *y ? -1 : 1;
+}
+
+static void id_add(struct bpf_tramp_id *id, u32 btf_id, void *addr)
+{
+ if (WARN_ON_ONCE(id->cnt >= id->max))
+ return;
+ id->id[id->cnt] = btf_id;
+ id->addr[id->cnt] = addr;
+ id->cnt++;
+}
+
+static struct bpf_tramp_id *id_check(struct bpf_tramp_id *id)
+{
+ if (bpf_tramp_id_is_empty(id)) {
+ bpf_tramp_id_put(id);
+ id = NULL;
+ }
+ return id;
+}
+
+static int id_and(struct bpf_tramp_id *a, struct bpf_tramp_id *b,
+ struct bpf_tramp_id **pand, struct bpf_tramp_id **pother)
+{
+ struct bpf_tramp_id *and, *other;
+ u32 i, id;
+
+ and = bpf_tramp_id_alloc(min(a->cnt, b->cnt));
+ other = bpf_tramp_id_alloc(max(a->cnt, b->cnt));
+ if (!and || !other) {
+ bpf_tramp_id_put(and);
+ bpf_tramp_id_put(other);
+ return -ENOMEM;
+ }
+
+ and->obj_id = a->obj_id;
+ other->obj_id = a->obj_id;
+
+ for (i = 0; i < a->cnt; i++) {
+ id = a->id[i];
+ if (bsearch(&id, b->id, b->cnt, sizeof(u32), btf_id_cmp))
+ id_add(and, id, a->addr[i]);
+ else
+ id_add(other, id, a->addr[i]);
+ }
+
+ *pand = id_check(and);
+ *pother = id_check(other);
+ return 0;
+}
+
+static int id_sub(struct bpf_tramp_id *a, struct bpf_tramp_id *b,
+ struct bpf_tramp_id **psub)
+{
+ struct bpf_tramp_id *sub;
+ u32 i, id;
+
+ sub = bpf_tramp_id_alloc(max(a->cnt, b->cnt));
+ if (!sub)
+ return -ENOMEM;
+
+ sub->obj_id = a->obj_id;
+
+ if (a->cnt < b->cnt)
+ swap(a, b);
+
+ for (i = 0; i < a->cnt; i++) {
+ id = a->id[i];
+ if (!bsearch(&id, b->id, b->cnt, sizeof(u32), btf_id_cmp))
+ id_add(sub, id, a->addr[i]);
+ }
+
+ *psub = id_check(sub);
+ return 0;
+}
+
+struct tramp_state {
+ struct bpf_trampoline *tr_common;
+ struct bpf_trampoline *tr_other;
+ struct bpf_tramp_id *id_common;
+ struct bpf_tramp_id *id_other;
+ struct bpf_tramp_id *id;
+};
+
+#define MAX_TRAMP_STATE 20
+
+struct attach_state {
+ struct tramp_state ts[MAX_TRAMP_STATE];
+ int cnt;
+};
+
+static struct tramp_state* tramp_state_get(struct attach_state *state)
+{
+ if (state->cnt == MAX_TRAMP_STATE)
+ return NULL;
+ return &state->ts[state->cnt];
+}
+
+static void state_next(struct attach_state *state)
+{
+ state->cnt++;
+}
+
+static void state_cleanup(struct attach_state *state)
+{
+ struct tramp_state *ts;
+ int i;
+
+ for (i = 0; i < state->cnt; i++) {
+ ts = &state->ts[state->cnt];
+ bpf_tramp_id_put(ts->id_common);
+ bpf_tramp_id_put(ts->id_other);
+ bpf_tramp_id_put(ts->id);
+ }
+}
+
+static int tramp_state_compute(struct attach_state *state,
+ struct bpf_trampoline *tr,
+ struct bpf_tramp_id *id,
+ struct bpf_tramp_id **id_cont)
+{
+ struct bpf_tramp_id *id_new, *id_common, *id_other;
+ struct tramp_state *ts;
+
+ ts = tramp_state_get(state);
+ if (!ts)
+ return -EBUSY;
+
+ /* different playground.. bail out */
+ if (tr->id->obj_id != id->obj_id) {
+ *id_cont = bpf_tramp_id_get(id);
+ return 0;
+ }
+
+ /* complete match with trampoline */
+ if (bpf_tramp_id_is_equal(tr->id, id)) {
+ ts->id_common = bpf_tramp_id_get(id);
+ *id_cont = NULL;
+ goto out;
+ }
+
+ /* find out if there's common set of ids */
+ if (id_and(id, tr->id, &id_common, &id_new))
+ return -ENOMEM;
+
+ /* nothing in common, bail out */
+ if (!id_common) {
+ bpf_tramp_id_put(id_new);
+ *id_cont = bpf_tramp_id_get(id);
+ return 0;
+ }
+
+ /* we have common set, let's get the rest of the matched
+ * trampoline ids as new id for split trampoline
+ */
+ if (id_sub(id_common, tr->id, &id_other)) {
+ bpf_tramp_id_put(id_common);
+ bpf_tramp_id_put(id_new);
+ return -ENOMEM;
+ }
+
+ ts->id_common = id_common;
+ ts->id_other = id_other;
+ ts->id = bpf_tramp_id_get(tr->id);
+ *id_cont = id_new;
+
+out:
+ ts->tr_common = tr;
+ state_next(state);
+ return 0;
+}
+
+static int bpf_trampoline_register(struct bpf_trampoline *tr)
+{
+ return bpf_trampoline_update(tr);
+}
+
+static int bpf_trampoline_unregister(struct bpf_trampoline *tr)
+{
+ int err;
+
+ if (!tr->cur_image)
+ return 0;
+ err = unregister_fentry(tr, tr->cur_image->image);
+ bpf_tramp_image_put(tr->cur_image);
+ tr->cur_image = NULL;
+ tr->selector = 0;
+ return err;
+}
+
+static void bpf_trampoline_id_assign(struct bpf_trampoline *tr, struct bpf_tramp_id *id)
+{
+ bool multi1 = bpf_tramp_id_is_multi(tr->id);
+ bool multi2 = bpf_tramp_id_is_multi(id);
+
+ /* We can split into single ID trampolines and that
+ * might affect nr_bpf_trampoline_multi and the fast
+ * path trigger, so we need to check on that.
+ */
+ if (multi1 && !multi2)
+ nr_bpf_trampoline_multi--;
+ if (!multi1 && multi2)
+ nr_bpf_trampoline_multi++;
+
+ tr->id = id;
+}
+
+static int bpf_trampoline_split(struct tramp_state *ts, struct attach_args *att)
+{
+ struct bpf_trampoline *tr_other, *tr_common = ts->tr_common;
+ struct bpf_tramp_id *id_common = ts->id_common;
+ struct bpf_tramp_id *id_other = ts->id_other;
+ int err;
+
+ mutex_lock(&tr_common->mutex);
+
+ err = bpf_trampoline_unregister(tr_common);
+ if (err)
+ goto out;
+
+ tr_other = bpf_trampoline_dup(tr_common, id_other);
+ if (!tr_other) {
+ err = -ENOMEM;
+ goto out_free;
+ }
+
+ err = bpf_trampoline_setup(tr_other, att);
+ if (err)
+ goto out_free;
+
+ bpf_trampoline_id_assign(tr_common, id_common);
+
+ err = bpf_trampoline_setup(tr_common, att);
+ if (err)
+ goto out_free;
+
+ ts->tr_other = tr_other;
+ WARN_ON_ONCE(bpf_trampoline_register(tr_common));
+ WARN_ON_ONCE(bpf_trampoline_register(tr_other));
+
+ mutex_unlock(&tr_common->mutex);
+ return 0;
+
+out_free:
+ bpf_trampoline_dup_destroy(tr_other);
+ tr_common->id = ts->id;
+ WARN_ON_ONCE(bpf_trampoline_register(tr_common));
+out:
+ mutex_unlock(&tr_common->mutex);
+ return err;
+}
+
+static int tramp_state_apply(struct bpf_tramp_attach *attach,
+ struct tramp_state *ts, struct attach_args *att)
+{
+ struct bpf_tramp_node *node;
+ int err;
+
+ /* The program will be attached to the common part. */
+ node = node_alloc(ts->tr_common, att->prog);
+ if (!node)
+ return -ENOMEM;
+
+ refcount_inc(&ts->tr_common->refcnt);
+
+ /* If there are also 'other' IDs in the trampoline,
+ * we need to do the split. */
+ if (ts->id_other) {
+ err = bpf_trampoline_split(ts, att);
+ if (err) {
+ node_free(node);
+ return err;
+ }
+ }
+
+ hlist_add_head(&node->hlist_attach, &attach->nodes);
+ return 0;
+}
+
+static int tramp_state_revert(struct tramp_state *ts, struct attach_args *att)
+{
+ struct bpf_trampoline *tr_common = ts->tr_common;
+ int err;
+
+ bpf_trampoline_dup_destroy(ts->tr_other);
+
+ mutex_lock(&tr_common->mutex);
+ err = bpf_trampoline_unregister(tr_common);
+ if (err)
+ goto out;
+
+ tr_common->id = ts->id;
+ err = bpf_trampoline_setup(tr_common, att);
+ if (err)
+ goto out;
+
+ WARN_ON_ONCE(bpf_trampoline_register(tr_common));
+out:
+ mutex_unlock(&tr_common->mutex);
+ return err;
+}
+
+static int
+bpf_tramp_attach_single(struct bpf_tramp_attach *attach,
+ struct bpf_tramp_id *id, struct attach_args *att)
+{
+ struct bpf_trampoline *tr = NULL;
+ struct bpf_tramp_node *node;
+ int err;
tr = bpf_trampoline_get(id);
if (!tr) {
@@ -647,22 +1101,175 @@ struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
goto out;
}
- node = node_alloc(tr, prog);
+ node = node_alloc(tr, att->prog);
if (!node)
goto out;
- err = bpf_check_attach_model(prog, tgt_prog, id->id[0], &tr->func.model);
+ err = bpf_check_attach_model(att->prog, att->tgt_prog,
+ id->id[0], &tr->func.model);
if (err)
goto out;
- attach->id = id;
hlist_add_head(&node->hlist_attach, &attach->nodes);
- return attach;
+ return 0;
out:
bpf_trampoline_put(tr);
- kfree(attach);
- return ERR_PTR(err);
+ return err;
+}
+
+#define list_for_each_trampoline(tr, i) \
+ for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++) \
+ hlist_for_each_entry(tr, &trampoline_table[i], hlist)
+
+static int __bpf_tramp_attach(struct bpf_tramp_attach *attach,
+ struct bpf_tramp_id *id,
+ struct attach_args *att)
+{
+ struct attach_state state = {};
+ struct bpf_tramp_id *id_cont;
+ struct bpf_trampoline *tr;
+ bool id_put = false;
+ int err = 0, i, j;
+
+ mutex_lock(&trampoline_mutex);
+
+ /* If we are ataching single ID trampoline and there's no multi ID
+ * trampoline registered, there's no need to iterate all trampolines
+ * for intersection, we can do the fast path and use hash search.
+ * */
+ if (!bpf_tramp_id_is_multi(id) && !nr_bpf_trampoline_multi) {
+ err = bpf_tramp_attach_single(attach, id, att);
+ goto out;
+ }
+
+ /* Iterate all trampolines to find all the interesections. */
+ list_for_each_trampoline(tr, i) {
+ err = tramp_state_compute(&state, tr, id, &id_cont);
+ if (err)
+ goto out_multi;
+ id_put = true;
+ id = id_cont;
+ if (!id)
+ goto out_break;
+ }
+out_break:
+
+ /* Do the actuall trampoline splits if there's any .. */
+ for (i = 0; i < state.cnt; i++) {
+ err = tramp_state_apply(attach, &state.ts[i], att);
+ if (err)
+ goto revert;
+ }
+
+ /* .. and create new trampoline if needed. */
+ if (id)
+ err = bpf_trampoline_create(attach, id, att);
+
+revert:
+ /* Attach failed, let's revert already changed trampolines */
+ if (err) {
+ for (j = 0; j < i; j++)
+ WARN_ON_ONCE(tramp_state_revert(&state.ts[j], att));
+ }
+
+out_multi:
+ if (id_put)
+ bpf_tramp_id_put(id);
+out:
+ mutex_unlock(&trampoline_mutex);
+ state_cleanup(&state);
+ return err;
+}
+
+#define MAX_ARGS 7
+
+static void put_args(struct bpf_tramp_id **args)
+{
+ int i;
+
+ for (i = 0; i < MAX_ARGS; i++)
+ bpf_tramp_id_put(args[i]);
+}
+
+static int get_args(struct bpf_tramp_id *id, struct bpf_tramp_id **args,
+ struct bpf_prog *tgt_prog, struct bpf_prog *prog)
+{
+ const struct btf_type *t;
+ struct bpf_tramp_id *a;
+ const struct btf *btf;
+ int err = -EINVAL;
+ u32 i, nargs;
+
+ btf = tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf;
+ if (!btf)
+ return -EINVAL;
+
+ for (i = 0; i < id->cnt; i++){
+ t = btf_type_by_id(btf, id->id[i]);
+ if (!btf_type_is_func(t))
+ goto out_free;
+ t = btf_type_by_id(btf, t->type);
+ if (!btf_type_is_func_proto(t))
+ goto out_free;
+ nargs = btf_type_vlen(t);
+ if (nargs >= MAX_ARGS)
+ goto out_free;
+ a = args[nargs];
+ if (!a) {
+ a = bpf_tramp_id_alloc(id->cnt);
+ if (!a) {
+ err = -ENOMEM;
+ goto out_free;
+ }
+ a->obj_id = id->obj_id;
+ args[nargs] = a;
+ }
+ id_add(a, id->id[i], id->addr[i]);
+ }
+ err = 0;
+out_free:
+ if (err)
+ put_args(args);
+ return err;
+}
+
+struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id,
+ struct bpf_prog *tgt_prog,
+ struct bpf_prog *prog)
+{
+ struct bpf_tramp_id *args[MAX_ARGS] = {};
+ struct bpf_tramp_attach *attach;
+ struct attach_args att = {
+ .tgt_prog = tgt_prog,
+ .prog = prog,
+ };
+ int i, err;
+
+ err = get_args(id, args, tgt_prog, prog);
+ if (err)
+ return ERR_PTR(err);
+
+ attach = kzalloc(sizeof(*attach), GFP_KERNEL);
+ if (!attach)
+ return ERR_PTR(-ENOMEM);
+
+ for (i = 0; i < MAX_ARGS; i++) {
+ if (!args[i] || !args[i]->cnt)
+ continue;
+ att.nr_args = i;
+ err = __bpf_tramp_attach(attach, args[i], &att);
+ if (err)
+ break;
+ }
+
+ if (err)
+ bpf_tramp_detach(attach);
+ else
+ attach->id = id;
+
+ put_args(args);
+ return err ? ERR_PTR(err) : attach;
}
void bpf_tramp_detach(struct bpf_tramp_attach *attach)
@@ -670,8 +1277,10 @@ void bpf_tramp_detach(struct bpf_tramp_attach *attach)
struct bpf_tramp_node *node;
struct hlist_node *n;
+ mutex_lock(&trampoline_mutex);
hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach)
node_free(node);
+ mutex_unlock(&trampoline_mutex);
bpf_tramp_id_put(attach->id);
kfree(attach);
@@ -682,13 +1291,14 @@ int bpf_tramp_attach_link(struct bpf_tramp_attach *attach)
struct bpf_tramp_node *node;
int err;
+ mutex_lock(&trampoline_mutex);
hlist_for_each_entry(node, &attach->nodes, hlist_attach) {
err = bpf_trampoline_link_prog(node, node->tr);
if (err)
- return err;
+ break;
}
-
- return 0;
+ mutex_unlock(&trampoline_mutex);
+ return err;
}
int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach)
@@ -696,13 +1306,14 @@ int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach)
struct bpf_tramp_node *node;
int err;
+ mutex_lock(&trampoline_mutex);
hlist_for_each_entry(node, &attach->nodes, hlist_attach) {
err = bpf_trampoline_unlink_prog(node, node->tr);
if (err)
- return err;
+ break;
}
-
- return 0;
+ mutex_unlock(&trampoline_mutex);
+ return err;
}
#define NO_START_TIME 1
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 20/29] bpf: Add support for tracing multi link
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (18 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 19/29] bpf: Add support to attach trampolines with multiple IDs Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 21/29] libbpf: Add btf__find_by_glob_kind function Jiri Olsa
` (8 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding new link to allow to attach program to multiple
function BTF IDs.
New fields are added to bpf_attr::link_create to pass
array of BTF IDs:
struct {
__aligned_u64 btf_ids; /* addresses to attach */
__u32 btf_ids_cnt; /* addresses count */
} multi;
The new link code will load these IDs into bpf_tramp_id
and resolve their ips.
The resolve itself is done as per Andrii's suggestion:
- lookup all names for given IDs
- store and sort them by name
- go through all kallsyms symbols and use bsearch
to find it in provided names
- if name is found, store the address for the name
- resort the names array based on ID
If there are multi symbols of the same name the first one
will be used to resolve the address.
The new link will pass them to the bpf_tramp_attach that
does all the work of attaching.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/uapi/linux/bpf.h | 5 +
kernel/bpf/syscall.c | 252 +++++++++++++++++++++++++++++++++
kernel/kallsyms.c | 2 +-
tools/include/uapi/linux/bpf.h | 5 +
4 files changed, 263 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index ca05e35e0478..a03a5bc1d141 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1009,6 +1009,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_NETNS = 5,
BPF_LINK_TYPE_XDP = 6,
BPF_LINK_TYPE_PERF_EVENT = 7,
+ BPF_LINK_TYPE_TRACING_MULTI = 8,
MAX_BPF_LINK_TYPE,
};
@@ -1470,6 +1471,10 @@ union bpf_attr {
*/
__u64 bpf_cookie;
} perf_event;
+ struct {
+ __aligned_u64 btf_ids; /* addresses to attach */
+ __u32 btf_ids_cnt; /* addresses count */
+ } multi;
};
} link_create;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index bfbd81869818..e6f48dc9dd48 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -32,6 +32,9 @@
#include <linux/rcupdate_trace.h>
#include <linux/memcontrol.h>
#include <linux/btf_ids.h>
+#include <linux/ftrace.h>
+#include <linux/sort.h>
+#include <linux/bsearch.h>
#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -2905,6 +2908,251 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
return err;
}
+struct bpf_tracing_multi_link {
+ struct bpf_link link;
+ enum bpf_attach_type attach_type;
+ struct bpf_tramp_attach *attach;
+};
+
+static void bpf_tracing_multi_link_release(struct bpf_link *link)
+{
+ struct bpf_prog *prog = link->prog;
+ struct bpf_tramp_attach *attach = prog->aux->attach;
+
+ WARN_ON_ONCE(bpf_tramp_attach_unlink(attach));
+
+ if (prog->type != BPF_PROG_TYPE_EXT)
+ prog->aux->attach = NULL;
+ bpf_tramp_detach(attach);
+}
+
+static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
+{
+ struct bpf_tracing_multi_link *tr_link =
+ container_of(link, struct bpf_tracing_multi_link, link);
+
+ kfree(tr_link);
+}
+
+static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link,
+ struct seq_file *seq)
+{
+ struct bpf_tracing_multi_link *tr_link =
+ container_of(link, struct bpf_tracing_multi_link, link);
+
+ seq_printf(seq, "attach_type:\t%d\n", tr_link->attach_type);
+}
+
+static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link,
+ struct bpf_link_info *info)
+{
+ struct bpf_tracing_multi_link *tr_link =
+ container_of(link, struct bpf_tracing_multi_link, link);
+
+ info->tracing.attach_type = tr_link->attach_type;
+ return 0;
+}
+
+static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
+ .release = bpf_tracing_multi_link_release,
+ .dealloc = bpf_tracing_multi_link_dealloc,
+ .show_fdinfo = bpf_tracing_multi_link_show_fdinfo,
+ .fill_link_info = bpf_tracing_multi_link_fill_link_info,
+};
+
+static int check_multi_prog_type(struct bpf_prog *prog)
+{
+ if (prog->expected_attach_type != BPF_TRACE_FENTRY &&
+ prog->expected_attach_type != BPF_TRACE_FEXIT)
+ return -EINVAL;
+ return 0;
+}
+
+static int btf_ids_cmp(const void *a, const void *b)
+{
+ const u32 *x = a;
+ const u32 *y = b;
+
+ if (*x == *y)
+ return 0;
+ return *x < *y ? -1 : 1;
+}
+
+struct resolve_id {
+ const char *name;
+ void *addr;
+ u32 id;
+};
+
+static int rid_name_cmp(const void *a, const void *b)
+{
+ const struct resolve_id *x = a;
+ const struct resolve_id *y = b;
+
+ return strcmp(x->name, y->name);
+}
+
+static int rid_id_cmp(const void *a, const void *b)
+{
+ const struct resolve_id *x = a;
+ const struct resolve_id *y = b;
+
+ if (x->id == y->id)
+ return 0;
+ return x->id < y->id ? -1 : 1;
+}
+
+struct kallsyms_data {
+ struct resolve_id *rid;
+ u32 cnt;
+ u32 found;
+};
+
+static int kallsyms_callback(void *data, const char *name,
+ struct module *mod, unsigned long addr)
+{
+ struct kallsyms_data *args = data;
+ struct resolve_id *rid, id = {
+ .name = name,
+ };
+
+ rid = bsearch(&id, args->rid, args->cnt, sizeof(*rid), rid_name_cmp);
+ if (rid && !rid->addr) {
+ rid->addr = (void *) addr;
+ args->found++;
+ }
+ return args->found == args->cnt ? 1 : 0;
+}
+
+static int bpf_tramp_id_resolve(struct bpf_tramp_id *id, struct bpf_prog *prog)
+{
+ struct kallsyms_data args;
+ const struct btf_type *t;
+ struct resolve_id *rid;
+ const char *name;
+ struct btf *btf;
+ int err = 0;
+ u32 i;
+
+ btf = prog->aux->attach_btf;
+ if (!btf)
+ return -EINVAL;
+
+ rid = kzalloc(id->cnt * sizeof(*rid), GFP_KERNEL);
+ if (!rid)
+ return -ENOMEM;
+
+ err = -EINVAL;
+ for (i = 0; i < id->cnt; i++) {
+ t = btf_type_by_id(btf, id->id[i]);
+ if (!t)
+ goto out_free;
+
+ name = btf_name_by_offset(btf, t->name_off);
+ if (!name)
+ goto out_free;
+
+ rid[i].name = name;
+ rid[i].id = id->id[i];
+ }
+
+ sort(rid, id->cnt, sizeof(*rid), rid_name_cmp, NULL);
+
+ args.rid = rid;
+ args.cnt = id->cnt;
+ args.found = 0;
+ kallsyms_on_each_symbol(kallsyms_callback, &args);
+
+ sort(rid, id->cnt, sizeof(*rid), rid_id_cmp, NULL);
+
+ for (i = 0; i < id->cnt; i++) {
+ if (!rid[i].addr) {
+ err = -EINVAL;
+ goto out_free;
+ }
+ id->addr[i] = rid[i].addr;
+ }
+ err = 0;
+out_free:
+ kfree(rid);
+ return err;
+}
+
+static int bpf_tracing_multi_attach(struct bpf_prog *prog,
+ const union bpf_attr *attr)
+{
+ void __user *uids = u64_to_user_ptr(attr->link_create.multi.btf_ids);
+ u32 cnt_size, cnt = attr->link_create.multi.btf_ids_cnt;
+ struct bpf_tracing_multi_link *link = NULL;
+ struct bpf_link_primer link_primer;
+ struct bpf_tramp_attach *attach;
+ struct bpf_tramp_id *id = NULL;
+ int err = -EINVAL;
+
+ if (check_multi_prog_type(prog))
+ return -EINVAL;
+ if (!cnt || !uids)
+ return -EINVAL;
+
+ id = bpf_tramp_id_alloc(cnt);
+ if (!id)
+ return -ENOMEM;
+
+ err = -EFAULT;
+ cnt_size = cnt * sizeof(id->id[0]);
+ if (copy_from_user(id->id, uids, cnt_size))
+ goto out_free_id;
+
+ id->cnt = cnt;
+ id->obj_id = btf_obj_id(prog->aux->attach_btf);
+
+ /* Sort user provided BTF ids, so we can use memcmp
+ * and bsearch on top of it later.
+ */
+ sort(id->id, cnt, sizeof(u32), btf_ids_cmp, NULL);
+
+ err = bpf_tramp_id_resolve(id, prog);
+ if (err)
+ goto out_free_id;
+
+ attach = bpf_tramp_attach(id, NULL, prog);
+ if (IS_ERR(attach)) {
+ err = PTR_ERR(attach);
+ goto out_free_id;
+ }
+
+ link = kzalloc(sizeof(*link), GFP_KERNEL);
+ if (!link) {
+ err = -ENOMEM;
+ goto out_detach;
+ }
+
+ bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
+ &bpf_tracing_multi_link_lops, prog);
+ link->attach_type = prog->expected_attach_type;
+
+ err = bpf_link_prime(&link->link, &link_primer);
+ if (err) {
+ kfree(link);
+ goto out_detach;
+ }
+
+ err = bpf_tramp_attach_link(attach);
+ if (err) {
+ bpf_link_cleanup(&link_primer);
+ goto out_detach;
+ }
+ prog->aux->attach = attach;
+ return bpf_link_settle(&link_primer);
+
+out_detach:
+ bpf_tramp_detach(attach);
+ return err;
+out_free_id:
+ bpf_tramp_id_put(id);
+ return err;
+}
+
struct bpf_raw_tp_link {
struct bpf_link link;
struct bpf_raw_event_map *btp;
@@ -3211,6 +3459,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)
case BPF_CGROUP_SETSOCKOPT:
return BPF_PROG_TYPE_CGROUP_SOCKOPT;
case BPF_TRACE_ITER:
+ case BPF_TRACE_FENTRY:
+ case BPF_TRACE_FEXIT:
return BPF_PROG_TYPE_TRACING;
case BPF_SK_LOOKUP:
return BPF_PROG_TYPE_SK_LOOKUP;
@@ -4270,6 +4520,8 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
if (prog->expected_attach_type == BPF_TRACE_ITER)
return bpf_iter_link_attach(attr, uattr, prog);
+ else if (prog->aux->multi_func)
+ return bpf_tracing_multi_attach(prog, attr);
else if (prog->type == BPF_PROG_TYPE_EXT)
return bpf_tracing_prog_attach(prog,
attr->link_create.target_fd,
diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
index 3011bc33a5ba..904e140c3491 100644
--- a/kernel/kallsyms.c
+++ b/kernel/kallsyms.c
@@ -224,7 +224,7 @@ unsigned long kallsyms_lookup_name(const char *name)
return module_kallsyms_lookup_name(name);
}
-#ifdef CONFIG_LIVEPATCH
+#if defined(CONFIG_LIVEPATCH) || defined(CONFIG_BPF)
/*
* Iterate over all symbols in vmlinux. For symbols from modules use
* module_kallsyms_on_each_symbol instead.
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index ca05e35e0478..a03a5bc1d141 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1009,6 +1009,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_NETNS = 5,
BPF_LINK_TYPE_XDP = 6,
BPF_LINK_TYPE_PERF_EVENT = 7,
+ BPF_LINK_TYPE_TRACING_MULTI = 8,
MAX_BPF_LINK_TYPE,
};
@@ -1470,6 +1471,10 @@ union bpf_attr {
*/
__u64 bpf_cookie;
} perf_event;
+ struct {
+ __aligned_u64 btf_ids; /* addresses to attach */
+ __u32 btf_ids_cnt; /* addresses count */
+ } multi;
};
} link_create;
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 21/29] libbpf: Add btf__find_by_glob_kind function
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (19 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 20/29] bpf: Add support for tracing multi link Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 22/29] libbpf: Add support to link multi func tracing program Jiri Olsa
` (7 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: Andrii Nakryiko, netdev, bpf, Martin KaFai Lau, Song Liu,
Yonghong Song, John Fastabend, KP Singh
Adding btf__find_by_glob_kind function that returns array of
BTF ids that match given kind and allow/deny patterns.
int btf__find_by_glob_kind(const struct btf *btf, __u32 kind,
const char *allow_pattern,
const char *deny_pattern,
__u32 **__ids);
The __ids array is allocated and needs to be manually freed.
At the moment the supported pattern is '*' at the beginning or
the end of the pattern.
Kindly borrowed from retsnoop.
Suggested-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/lib/bpf/btf.c | 77 +++++++++++++++++++++++++++++++++++++++++++++
tools/lib/bpf/btf.h | 3 ++
2 files changed, 80 insertions(+)
diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c
index b6be579e0dc6..ebc02576390d 100644
--- a/tools/lib/bpf/btf.c
+++ b/tools/lib/bpf/btf.c
@@ -749,6 +749,83 @@ __s32 btf__find_by_name_kind(const struct btf *btf, const char *type_name,
return btf_find_by_name_kind(btf, 1, type_name, kind);
}
+/* 'borrowed' from retsnoop */
+static bool glob_matches(const char *glob, const char *s)
+{
+ int n = strlen(glob);
+
+ if (n == 1 && glob[0] == '*')
+ return true;
+
+ if (glob[0] == '*' && glob[n - 1] == '*') {
+ const char *subs;
+ /* substring match */
+
+ /* this is hacky, but we don't want to allocate for no good reason */
+ ((char *)glob)[n - 1] = '\0';
+ subs = strstr(s, glob + 1);
+ ((char *)glob)[n - 1] = '*';
+
+ return subs != NULL;
+ } else if (glob[0] == '*') {
+ size_t nn = strlen(s);
+ /* suffix match */
+
+ /* too short for a given suffix */
+ if (nn < n - 1)
+ return false;
+
+ return strcmp(s + nn - (n - 1), glob + 1) == 0;
+ } else if (glob[n - 1] == '*') {
+ /* prefix match */
+ return strncmp(s, glob, n - 1) == 0;
+ } else {
+ /* exact match */
+ return strcmp(glob, s) == 0;
+ }
+}
+
+int btf__find_by_glob_kind(const struct btf *btf, __u32 kind,
+ const char *allow_pattern, const char *deny_pattern,
+ __u32 **__ids)
+{
+ __u32 i, nr_types = btf__get_nr_types(btf);
+ int cnt = 0, alloc = 0;
+ __u32 *ids = NULL;
+
+ for (i = 1; i <= nr_types; i++) {
+ const struct btf_type *t = btf__type_by_id(btf, i);
+ const char *name;
+ __u32 *p;
+
+ if (btf_kind(t) != kind)
+ continue;
+ name = btf__name_by_offset(btf, t->name_off);
+ if (!name)
+ continue;
+
+ if (deny_pattern && glob_matches(deny_pattern, name))
+ continue;
+ if (allow_pattern && !glob_matches(allow_pattern, name))
+ continue;
+
+ if (cnt == alloc) {
+ alloc = max(16, alloc * 3 / 2);
+ p = libbpf_reallocarray(ids, alloc, sizeof(__u32));
+ if (!p) {
+ free(ids);
+ return -ENOMEM;
+ }
+ ids = p;
+ }
+ ids[cnt] = i;
+ cnt++;
+ }
+
+ *__ids = ids;
+ return cnt;
+}
+
static bool btf_is_modifiable(const struct btf *btf)
{
return (void *)btf->hdr != btf->raw_data;
diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h
index 5c73a5b0a044..408b8e6d913b 100644
--- a/tools/lib/bpf/btf.h
+++ b/tools/lib/bpf/btf.h
@@ -572,6 +572,9 @@ static inline struct btf_decl_tag *btf_decl_tag(const struct btf_type *t)
return (struct btf_decl_tag *)(t + 1);
}
+int btf__find_by_glob_kind(const struct btf *btf, __u32 kind,
+ const char *allow_pattern, const char *deny_pattern,
+ __u32 **__ids);
#ifdef __cplusplus
} /* extern "C" */
#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 22/29] libbpf: Add support to link multi func tracing program
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (20 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 21/29] libbpf: Add btf__find_by_glob_kind function Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 23/29] selftests/bpf: Add bpf_arg/bpf_ret_value test Jiri Olsa
` (6 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding support to link multi func tracing program
through link_create interface.
Adding special types for multi func programs:
fentry.multi
fexit.multi
so you can define multi func programs like:
SEC("fentry.multi/bpf_fentry_test*")
int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
that defines test1 to be attached to bpf_fentry_test* functions.
The test1 program is loaded with BPF_F_MULTI_FUNC flag.
If functions are not specified the program needs to be attached
manually.
Adding new btf_ids/btf_ids_cnt fields to bpf_link_create_opts,
that define functions to attach the program to.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/lib/bpf/bpf.c | 7 +++++
tools/lib/bpf/bpf.h | 6 +++-
tools/lib/bpf/libbpf.c | 66 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index 94560ba31724..86a95419e501 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -784,6 +784,13 @@ int bpf_link_create(int prog_fd, int target_fd,
if (!OPTS_ZEROED(opts, perf_event))
return libbpf_err(-EINVAL);
break;
+ case BPF_TRACE_FENTRY:
+ case BPF_TRACE_FEXIT:
+ attr.link_create.multi.btf_ids = (__u64) OPTS_GET(opts, multi.btf_ids, 0);
+ attr.link_create.multi.btf_ids_cnt = OPTS_GET(opts, multi.btf_ids_cnt, 0);
+ if (!OPTS_ZEROED(opts, multi))
+ return libbpf_err(-EINVAL);
+ break;
default:
if (!OPTS_ZEROED(opts, flags))
return libbpf_err(-EINVAL);
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 079cc81ac51e..e55abf3528b3 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -249,10 +249,14 @@ struct bpf_link_create_opts {
struct {
__u64 bpf_cookie;
} perf_event;
+ struct {
+ __u32 *btf_ids;
+ __u32 btf_ids_cnt;
+ } multi;
};
size_t :0;
};
-#define bpf_link_create_opts__last_field perf_event
+#define bpf_link_create_opts__last_field multi
LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index de7e09a6b5ec..4c11d38b1f92 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -246,6 +246,8 @@ enum sec_def_flags {
SEC_SLEEPABLE = 8,
/* allow non-strict prefix matching */
SEC_SLOPPY_PFX = 16,
+ /* BPF program type allows multiple functions attachment */
+ SEC_MULTI_FUNC = 32,
};
struct bpf_sec_def {
@@ -6723,6 +6725,9 @@ static int bpf_object_init_progs(struct bpf_object *obj, const struct bpf_object
continue;
}
+ if (prog->sec_def->cookie & SEC_MULTI_FUNC)
+ prog->prog_flags |= BPF_F_MULTI_FUNC;
+
bpf_program__set_type(prog, prog->sec_def->prog_type);
bpf_program__set_expected_attach_type(prog, prog->sec_def->expected_attach_type);
@@ -8318,6 +8323,7 @@ static struct bpf_link *attach_kprobe(const struct bpf_program *prog, long cooki
static struct bpf_link *attach_tp(const struct bpf_program *prog, long cookie);
static struct bpf_link *attach_raw_tp(const struct bpf_program *prog, long cookie);
static struct bpf_link *attach_trace(const struct bpf_program *prog, long cookie);
+static struct bpf_link *attach_trace_multi(const struct bpf_program *prog, long cookie);
static struct bpf_link *attach_lsm(const struct bpf_program *prog, long cookie);
static struct bpf_link *attach_iter(const struct bpf_program *prog, long cookie);
@@ -8345,6 +8351,8 @@ static const struct bpf_sec_def section_defs[] = {
SEC_DEF("fentry.s/", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fmod_ret.s/", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fexit.s/", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
+ SEC_DEF("fentry.multi/", TRACING, BPF_TRACE_FENTRY, SEC_MULTI_FUNC, attach_trace_multi),
+ SEC_DEF("fexit.multi/", TRACING, BPF_TRACE_FEXIT, SEC_MULTI_FUNC, attach_trace_multi),
SEC_DEF("freplace/", EXT, 0, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("lsm/", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
SEC_DEF("lsm.s/", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
@@ -8797,6 +8805,9 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attac
__u32 attach_prog_fd = prog->attach_prog_fd;
int err = 0;
+ if (prog->prog_flags & BPF_F_MULTI_FUNC)
+ return 0;
+
/* BPF program's BTF ID */
if (attach_prog_fd) {
err = libbpf_find_prog_btf_id(attach_name, attach_prog_fd);
@@ -10216,6 +10227,61 @@ static struct bpf_link *bpf_program__attach_btf_id(const struct bpf_program *pro
return (struct bpf_link *)link;
}
+static struct bpf_link *bpf_program__attach_multi(const struct bpf_program *prog)
+{
+ char *pattern = prog->sec_name + strlen(prog->sec_def->sec);
+ DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts);
+ enum bpf_attach_type attach_type;
+ int prog_fd, link_fd, cnt, err;
+ struct bpf_link *link = NULL;
+ __u32 *ids = NULL;
+
+ prog_fd = bpf_program__fd(prog);
+ if (prog_fd < 0) {
+ pr_warn("prog '%s': can't attach before loaded\n", prog->name);
+ return ERR_PTR(-EINVAL);
+ }
+
+ err = bpf_object__load_vmlinux_btf(prog->obj, true);
+ if (err)
+ return ERR_PTR(err);
+
+ cnt = btf__find_by_glob_kind(prog->obj->btf_vmlinux, BTF_KIND_FUNC,
+ pattern, NULL, &ids);
+ if (cnt <= 0)
+ return ERR_PTR(-EINVAL);
+
+ link = calloc(1, sizeof(*link));
+ if (!link) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+ link->detach = &bpf_link__detach_fd;
+
+ opts.multi.btf_ids = ids;
+ opts.multi.btf_ids_cnt = cnt;
+
+ attach_type = bpf_program__get_expected_attach_type(prog);
+ link_fd = bpf_link_create(prog_fd, 0, attach_type, &opts);
+ if (link_fd < 0) {
+ err = -errno;
+ goto out_err;
+ }
+ link->fd = link_fd;
+ free(ids);
+ return link;
+
+out_err:
+ free(link);
+ free(ids);
+ return ERR_PTR(err);
+}
+
+static struct bpf_link *attach_trace_multi(const struct bpf_program *prog, long cookie)
+{
+ return bpf_program__attach_multi(prog);
+}
+
struct bpf_link *bpf_program__attach_trace(const struct bpf_program *prog)
{
return bpf_program__attach_btf_id(prog);
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 23/29] selftests/bpf: Add bpf_arg/bpf_ret_value test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (21 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 22/29] libbpf: Add support to link multi func tracing program Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 24/29] selftests/bpf: Add fentry multi func test Jiri Olsa
` (5 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding tests for bpf_arg/bpf_ret_value helpers on
both fentry and fexit programs.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
.../selftests/bpf/prog_tests/args_test.c | 34 +++++++++++++++++++
tools/testing/selftests/bpf/progs/args_test.c | 30 ++++++++++++++++
2 files changed, 64 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/args_test.c
create mode 100644 tools/testing/selftests/bpf/progs/args_test.c
diff --git a/tools/testing/selftests/bpf/prog_tests/args_test.c b/tools/testing/selftests/bpf/prog_tests/args_test.c
new file mode 100644
index 000000000000..1938f2616ee9
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/args_test.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "args_test.skel.h"
+
+void test_args_test(void)
+{
+ struct args_test *skel = NULL;
+ __u32 duration = 0, retval;
+ int err, prog_fd;
+
+ skel = args_test__open();
+ if (!ASSERT_OK_PTR(skel, "args_test__open"))
+ return;
+
+ err = args_test__load(skel);
+ if (!ASSERT_OK(err, "args_test__load"))
+ goto cleanup;
+
+ err = args_test__attach(skel);
+ if (!ASSERT_OK(err, "args_test__attach"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.test1);
+ err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+
+ ASSERT_EQ(skel->bss->test1_result, 1, "test1_result");
+ ASSERT_EQ(skel->bss->test2_result, 1, "test2_result");
+
+cleanup:
+ args_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/args_test.c b/tools/testing/selftests/bpf/progs/args_test.c
new file mode 100644
index 000000000000..7fc8e9fb41bd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/args_test.c
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u64 test1_result = 0;
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(test1)
+{
+ __u64 a = bpf_arg(ctx, 0);
+ __u64 x = bpf_arg(ctx, 1);
+
+ test1_result = (int) a == 1 && x == 0;
+ return 0;
+}
+
+__u64 test2_result = 0;
+SEC("fexit/bpf_fentry_test2")
+int BPF_PROG(test2)
+{
+ __u64 ret = bpf_ret_value(ctx);
+ __u64 a = bpf_arg(ctx, 0);
+ __u64 b = bpf_arg(ctx, 1);
+ __u64 x = bpf_arg(ctx, 2);
+
+ test2_result = (int) a == 2 && b == 3 && ret == 5 && x == 0;
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 24/29] selftests/bpf: Add fentry multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (22 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 23/29] selftests/bpf: Add bpf_arg/bpf_ret_value test Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 25/29] selftests/bpf: Add fexit " Jiri Olsa
` (4 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding selftest for fentry multi func test that attaches
to bpf_fentry_test* functions and checks argument values
based on the processed function.
We need to cast to real arguments types in multi_arg_check,
because the checked value can be shorter than u64.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 4 +-
.../bpf/prog_tests/multi_fentry_test.c | 30 +++++++++
.../testing/selftests/bpf/progs/multi_check.c | 63 +++++++++++++++++++
.../selftests/bpf/progs/multi_fentry.c | 17 +++++
4 files changed, 113 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_check.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 35684d61aaeb..fa29ddd47cbe 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -322,7 +322,8 @@ endef
SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
- linked_vars.skel.h linked_maps.skel.h
+ linked_vars.skel.h linked_maps.skel.h \
+ multi_fentry_test.skel.h
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c
@@ -334,6 +335,7 @@ test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o
linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o
linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o
+multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o
LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c
new file mode 100644
index 000000000000..8dc08c3e715f
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "multi_fentry_test.skel.h"
+#include "trace_helpers.h"
+
+void test_multi_fentry_test(void)
+{
+ struct multi_fentry_test *skel = NULL;
+ __u32 duration = 0, retval;
+ int err, prog_fd;
+
+ skel = multi_fentry_test__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load"))
+ goto cleanup;
+
+ err = multi_fentry_test__attach(skel);
+ if (!ASSERT_OK(err, "fentry_attach"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.test);
+ err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+
+ ASSERT_EQ(skel->bss->test_result, 8, "test_result");
+
+cleanup:
+ multi_fentry_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c
new file mode 100644
index 000000000000..82acc9ee7715
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_check.c
@@ -0,0 +1,63 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+extern const void bpf_fentry_test1 __ksym;
+extern const void bpf_fentry_test2 __ksym;
+extern const void bpf_fentry_test3 __ksym;
+extern const void bpf_fentry_test4 __ksym;
+extern const void bpf_fentry_test5 __ksym;
+extern const void bpf_fentry_test6 __ksym;
+extern const void bpf_fentry_test7 __ksym;
+extern const void bpf_fentry_test8 __ksym;
+
+void multi_arg_check(__u64 *ctx, __u64 *test_result)
+{
+ void *ip = (void *) bpf_get_func_ip(ctx);
+
+ if (ip == &bpf_fentry_test1) {
+ int a = (int) ctx[0];
+
+ *test_result += a == 1;
+ } else if (ip == &bpf_fentry_test2) {
+ int a = (int) bpf_arg(ctx, 0);
+ __u64 b = bpf_arg(ctx, 1);
+
+ *test_result += a == 2 && b == 3;
+ } else if (ip == &bpf_fentry_test3) {
+ char a = (int) bpf_arg(ctx, 0);
+ int b = (int) bpf_arg(ctx, 1);
+ __u64 c = bpf_arg(ctx, 2);
+
+ *test_result += a == 4 && b == 5 && c == 6;
+ } else if (ip == &bpf_fentry_test4) {
+ void *a = (void *) bpf_arg(ctx, 0);
+ char b = (char) bpf_arg(ctx, 1);
+ int c = (int) bpf_arg(ctx, 2);
+ __u64 d = bpf_arg(ctx, 3);
+
+ *test_result += a == (void *) 7 && b == 8 && c == 9 && d == 10;
+ } else if (ip == &bpf_fentry_test5) {
+ __u64 a = bpf_arg(ctx, 0);
+ void *b = (void *) bpf_arg(ctx, 1);
+ short c = (short) bpf_arg(ctx, 2);
+ int d = (int) bpf_arg(ctx, 3);
+ __u64 e = bpf_arg(ctx, 4);
+
+ *test_result += a == 11 && b == (void *) 12 && c == 13 && d == 14 && e == 15;
+ } else if (ip == &bpf_fentry_test6) {
+ __u64 a = bpf_arg(ctx, 0);
+ void *b = (void *) bpf_arg(ctx, 1);
+ short c = (short) bpf_arg(ctx, 2);
+ int d = (int) bpf_arg(ctx, 3);
+ void *e = (void *) bpf_arg(ctx, 4);
+ __u64 f = bpf_arg(ctx, 5);
+
+ *test_result += a == 16 && b == (void *) 17 && c == 18 && d == 19 && e == (void *) 20 && f == 21;
+ } else if (ip == &bpf_fentry_test7) {
+ *test_result += 1;
+ } else if (ip == &bpf_fentry_test8) {
+ *test_result += 1;
+ }
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_fentry.c b/tools/testing/selftests/bpf/progs/multi_fentry.c
new file mode 100644
index 000000000000..b78d36772aa6
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_fentry.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u64 test_result = 0;
+
+__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result);
+
+SEC("fentry.multi/bpf_fentry_test*")
+int BPF_PROG(test, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ multi_arg_check(ctx, &test_result);
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 25/29] selftests/bpf: Add fexit multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (23 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 24/29] selftests/bpf: Add fentry multi func test Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 26/29] selftests/bpf: Add fentry/fexit " Jiri Olsa
` (3 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding selftest for fexit multi func test that attaches
to bpf_fentry_test* functions and checks argument values
based on the processed function.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 3 +-
.../bpf/prog_tests/multi_fexit_test.c | 31 +++++++++++++++++++
.../testing/selftests/bpf/progs/multi_check.c | 23 ++++++++++++++
.../testing/selftests/bpf/progs/multi_fexit.c | 20 ++++++++++++
4 files changed, 76 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_fexit.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index fa29ddd47cbe..42b67834d803 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -323,7 +323,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h \
- multi_fentry_test.skel.h
+ multi_fentry_test.skel.h multi_fexit_test.skel.h
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c
@@ -336,6 +336,7 @@ linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o
linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o
multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o
+multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o
LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c
new file mode 100644
index 000000000000..d9b0eedd9f45
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "multi_fexit_test.skel.h"
+#include "trace_helpers.h"
+
+void test_multi_fexit_test(void)
+{
+ struct multi_fexit_test *skel = NULL;
+ __u32 duration = 0, retval;
+ int err, prog_fd;
+
+ skel = multi_fexit_test__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "fexit_multi_skel_load"))
+ goto cleanup;
+
+ err = multi_fexit_test__attach(skel);
+ if (!ASSERT_OK(err, "fexit_attach"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.test);
+ err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+
+ ASSERT_EQ(skel->bss->test_arg_result, 8, "fexit_multi_arg_result");
+ ASSERT_EQ(skel->bss->test_ret_result, 8, "fexit_multi_ret_result");
+
+cleanup:
+ multi_fexit_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c
index 82acc9ee7715..2f14d232dd77 100644
--- a/tools/testing/selftests/bpf/progs/multi_check.c
+++ b/tools/testing/selftests/bpf/progs/multi_check.c
@@ -61,3 +61,26 @@ void multi_arg_check(__u64 *ctx, __u64 *test_result)
*test_result += 1;
}
}
+
+void multi_ret_check(void *ctx, __u64 *test_result)
+{
+ void *ip = (void *) bpf_get_func_ip(ctx);
+ int ret = (int) bpf_ret_value(ctx);
+
+ if (ip == &bpf_fentry_test1)
+ *test_result += ret == 2;
+ else if (ip == &bpf_fentry_test2)
+ *test_result += ret == 5;
+ else if (ip == &bpf_fentry_test3)
+ *test_result += ret == 15;
+ else if (ip == &bpf_fentry_test4)
+ *test_result += ret == 34;
+ else if (ip == &bpf_fentry_test5)
+ *test_result += ret == 65;
+ else if (ip == &bpf_fentry_test6)
+ *test_result += ret == 111;
+ else if (ip == &bpf_fentry_test7)
+ *test_result += ret == 0;
+ else if (ip == &bpf_fentry_test8)
+ *test_result += ret == 0;
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_fexit.c b/tools/testing/selftests/bpf/progs/multi_fexit.c
new file mode 100644
index 000000000000..54624acc7071
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_fexit.c
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result);
+__hidden extern void multi_ret_check(void *ctx, __u64 *test_result);
+
+__u64 test_arg_result = 0;
+__u64 test_ret_result = 0;
+
+SEC("fexit.multi/bpf_fentry_test*")
+int BPF_PROG(test)
+{
+ multi_arg_check(ctx, &test_arg_result);
+ multi_ret_check(ctx, &test_ret_result);
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 26/29] selftests/bpf: Add fentry/fexit multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (24 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 25/29] selftests/bpf: Add fexit " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 27/29] selftests/bpf: Add mixed " Jiri Olsa
` (2 subsequent siblings)
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding selftest for fentry/fexit multi func tests that attaches
to bpf_fentry_test* functions and checks argument values based
on the processed function.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 4 ++-
.../bpf/prog_tests/multi_fentry_fexit_test.c | 32 +++++++++++++++++++
.../selftests/bpf/progs/multi_fentry_fexit.c | 28 ++++++++++++++++
3 files changed, 63 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry_fexit.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 42b67834d803..236b6e0a36de 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -323,7 +323,8 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h \
- multi_fentry_test.skel.h multi_fexit_test.skel.h
+ multi_fentry_test.skel.h multi_fexit_test.skel.h \
+ multi_fentry_fexit_test.skel.h
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c
@@ -337,6 +338,7 @@ linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o
linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o
multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o
multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o
+multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o
LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c
new file mode 100644
index 000000000000..d54abf36ab2f
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "multi_fentry_fexit_test.skel.h"
+
+void test_multi_fentry_fexit_test(void)
+{
+ DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts);
+ struct multi_fentry_fexit_test *skel = NULL;
+ __u32 duration = 0, retval;
+ int err, prog_fd;
+
+ skel = multi_fentry_fexit_test__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load"))
+ goto cleanup;
+
+ err = multi_fentry_fexit_test__attach(skel);
+ if (!ASSERT_OK(err, "fentry_fexit_attach"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.test2);
+ err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+
+ ASSERT_EQ(skel->bss->test1_arg_result, 8, "test1_arg_result");
+ ASSERT_EQ(skel->bss->test2_arg_result, 8, "test2_arg_result");
+ ASSERT_EQ(skel->bss->test2_ret_result, 8, "test2_ret_result");
+
+cleanup:
+ multi_fentry_fexit_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c
new file mode 100644
index 000000000000..54ee94d060b2
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u64 test1_arg_result = 0;
+__u64 test2_arg_result = 0;
+__u64 test2_ret_result = 0;
+
+__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result);
+__hidden extern void multi_ret_check(void *ctx, __u64 *test_result);
+
+SEC("fentry.multi/bpf_fentry_test*")
+int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ multi_arg_check(ctx, &test1_arg_result);
+ return 0;
+}
+
+SEC("fexit.multi/bpf_fentry_test*")
+int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ multi_arg_check(ctx, &test2_arg_result);
+ multi_ret_check(ctx, &test2_ret_result);
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 27/29] selftests/bpf: Add mixed multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (25 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 26/29] selftests/bpf: Add fentry/fexit " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 28/29] selftests/bpf: Add ret_mod " Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 29/29] selftests/bpf: Add attach " Jiri Olsa
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding selftest for fentry/fexit multi func tests that attaches
to bpf_fentry_test* functions, where some of them have already
attached trampoline.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 3 +-
.../bpf/prog_tests/multi_mixed_test.c | 34 +++++++++++++++
.../testing/selftests/bpf/progs/multi_mixed.c | 43 +++++++++++++++++++
3 files changed, 79 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_mixed.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 236b6e0a36de..48970e983250 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -324,7 +324,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h \
multi_fentry_test.skel.h multi_fexit_test.skel.h \
- multi_fentry_fexit_test.skel.h
+ multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c
@@ -339,6 +339,7 @@ linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o
multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o
multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o
multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o
+multi_mixed_test.skel.h-deps := multi_mixed.o multi_check.o
LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
diff --git a/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c
new file mode 100644
index 000000000000..c9395b4eb5ac
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "multi_mixed_test.skel.h"
+
+void test_multi_mixed_test(void)
+{
+ DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts);
+ struct multi_mixed_test *skel = NULL;
+ __u32 duration = 0, retval;
+ int err, prog_fd;
+
+ skel = multi_mixed_test__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load"))
+ goto cleanup;
+
+ err = multi_mixed_test__attach(skel);
+ if (!ASSERT_OK(err, "fentry_attach"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(skel->progs.test1);
+ err = bpf_prog_test_run(prog_fd, 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+
+ ASSERT_EQ(skel->bss->test1_result, 1, "test1_result");
+ ASSERT_EQ(skel->bss->test2_result, 1, "test2_result");
+ ASSERT_EQ(skel->bss->test3_arg_result, 8, "test3_arg_result");
+ ASSERT_EQ(skel->bss->test4_arg_result, 8, "test4_arg_result");
+ ASSERT_EQ(skel->bss->test4_ret_result, 8, "test4_ret_result");
+
+cleanup:
+ multi_mixed_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_mixed.c b/tools/testing/selftests/bpf/progs/multi_mixed.c
new file mode 100644
index 000000000000..468a044753e9
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_mixed.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result);
+__hidden extern void multi_ret_check(void *ctx, __u64 *test_result);
+
+__u64 test1_result = 0;
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(test1, int a)
+{
+ test1_result += a == 1;
+ return 0;
+}
+
+__u64 test2_result = 0;
+SEC("fexit/bpf_fentry_test2")
+int BPF_PROG(test2, int a, __u64 b, int ret)
+{
+ test2_result += a == 2 && b == 3;
+ return 0;
+}
+
+__u64 test3_arg_result = 0;
+SEC("fentry.multi/bpf_fentry_test*")
+int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ multi_arg_check(ctx, &test3_arg_result);
+ return 0;
+}
+
+__u64 test4_arg_result = 0;
+__u64 test4_ret_result = 0;
+SEC("fexit.multi/bpf_fentry_test*")
+int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ multi_arg_check(ctx, &test4_arg_result);
+ multi_ret_check(ctx, &test4_ret_result);
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 28/29] selftests/bpf: Add ret_mod multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (26 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 27/29] selftests/bpf: Add mixed " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
2021-11-18 11:24 ` [PATCH bpf-next 29/29] selftests/bpf: Add attach " Jiri Olsa
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding extra test to existing modify_return test to
test this with multi func program attached on top
of the modify return program.
Because the supported wildcards do not allow us to
match both bpf_fentry_test* and bpf_modify_return_test,
adding extra code to look it up in kernel's BTF.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
.../selftests/bpf/prog_tests/modify_return.c | 114 +++++++++++++++++-
.../selftests/bpf/progs/multi_modify_return.c | 17 +++
2 files changed, 128 insertions(+), 3 deletions(-)
create mode 100644 tools/testing/selftests/bpf/progs/multi_modify_return.c
diff --git a/tools/testing/selftests/bpf/prog_tests/modify_return.c b/tools/testing/selftests/bpf/prog_tests/modify_return.c
index b772fe30ce9b..ffb0be7ea5a5 100644
--- a/tools/testing/selftests/bpf/prog_tests/modify_return.c
+++ b/tools/testing/selftests/bpf/prog_tests/modify_return.c
@@ -5,13 +5,100 @@
*/
#include <test_progs.h>
+#include <bpf/btf.h>
#include "modify_return.skel.h"
+#include "multi_modify_return.skel.h"
#define LOWER(x) ((x) & 0xffff)
#define UPPER(x) ((x) >> 16)
-static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret)
+struct multi_data {
+ struct multi_modify_return *skel;
+ int link_fentry;
+ int link_fexit;
+ __u32 btf_ids[9];
+};
+
+static int multi_btf_ids(struct multi_data *md)
+{
+ __u32 i, nr_types, ids_cnt;
+ struct btf *btf;
+
+ btf = btf__load_vmlinux_btf();
+ if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf"))
+ return -1;
+
+ nr_types = btf__get_nr_types(btf);
+
+ for (i = 1; i <= nr_types; i++) {
+ const struct btf_type *t = btf__type_by_id(btf, i);
+ const char *name;
+ bool match;
+
+ if (!btf_is_func(t))
+ continue;
+
+ name = btf__name_by_offset(btf, t->name_off);
+ if (!name)
+ continue;
+ match = strncmp(name, "bpf_modify_return_test",
+ sizeof("bpf_modify_return_test") - 1) == 0;
+ match |= strncmp(name, "bpf_fentry_test",
+ sizeof("bpf_fentry_test") - 1) == 0;
+ if (!match)
+ continue;
+
+ md->btf_ids[ids_cnt] = i;
+ ids_cnt++;
+ }
+
+ btf__free(btf);
+ return ASSERT_EQ(ids_cnt, 9, "multi_btf_ids") ? 0 : -1;
+}
+
+static int multi_attach(struct multi_data *md)
+{
+ DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts);
+ int prog_fd;
+
+ md->skel = multi_modify_return__open_and_load();
+ if (!ASSERT_OK_PTR(md->skel, "multi_attach_check__load"))
+ return -1;
+
+ opts.multi.btf_ids = md->btf_ids;
+ opts.multi.btf_ids_cnt = 9;
+
+ prog_fd = bpf_program__fd(md->skel->progs.test1);
+
+ md->link_fentry = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &opts);
+ if (!ASSERT_GE(md->link_fentry, 0, "bpf_link_create"))
+ goto cleanup;
+
+ prog_fd = bpf_program__fd(md->skel->progs.test2);
+
+ md->link_fexit = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &opts);
+ if (!ASSERT_GE(md->link_fexit, 0, "bpf_link_create"))
+ goto cleanup_close;
+
+ return 0;
+
+cleanup_close:
+ close(md->link_fentry);
+cleanup:
+ multi_modify_return__destroy(md->skel);
+ return -1;
+}
+
+static void multi_detach(struct multi_data *md)
+{
+ close(md->link_fentry);
+ close(md->link_fexit);
+ multi_modify_return__destroy(md->skel);
+}
+
+static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret,
+ struct multi_data *md)
{
struct modify_return *skel = NULL;
int err, prog_fd;
@@ -27,6 +114,9 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret)
if (CHECK(err, "modify_return", "attach failed: %d\n", err))
goto cleanup;
+ if (md && !ASSERT_OK(multi_attach(md), "multi_attach"))
+ goto cleanup;
+
skel->bss->input_retval = input_retval;
prog_fd = bpf_program__fd(skel->progs.fmod_ret_test);
err = bpf_prog_test_run(prog_fd, 1, NULL, 0, NULL, 0,
@@ -49,6 +139,8 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret)
CHECK(skel->bss->fmod_ret_result != 1, "modify_return",
"fmod_ret failed\n");
+ if (md)
+ multi_detach(md);
cleanup:
modify_return__destroy(skel);
}
@@ -56,11 +148,27 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret)
/* TODO: conflict with get_func_ip_test */
void serial_test_modify_return(void)
{
+ struct multi_data data = {};
+
+ run_test(0 /* input_retval */,
+ 1 /* want_side_effect */,
+ 4 /* want_ret */,
+ NULL /* no multi func test */);
+ run_test(-EINVAL /* input_retval */,
+ 0 /* want_side_effect */,
+ -EINVAL /* want_ret */,
+ NULL /* no multi func test */);
+
+ if (!ASSERT_OK(multi_btf_ids(&data), "multi_attach"))
+ return;
+
run_test(0 /* input_retval */,
1 /* want_side_effect */,
- 4 /* want_ret */);
+ 4 /* want_ret */,
+ &data);
run_test(-EINVAL /* input_retval */,
0 /* want_side_effect */,
- -EINVAL /* want_ret */);
+ -EINVAL /* want_ret */,
+ &data);
}
diff --git a/tools/testing/selftests/bpf/progs/multi_modify_return.c b/tools/testing/selftests/bpf/progs/multi_modify_return.c
new file mode 100644
index 000000000000..34754e438c96
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_modify_return.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+SEC("fentry.multi/bpf_fentry_test*")
+int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ return 0;
+}
+
+SEC("fexit.multi/bpf_fentry_test*")
+int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread* [PATCH bpf-next 29/29] selftests/bpf: Add attach multi func test
2021-11-18 11:24 [RFC bpf-next v5 00/29] bpf: Add batch support for attaching trampolines Jiri Olsa
` (27 preceding siblings ...)
2021-11-18 11:24 ` [PATCH bpf-next 28/29] selftests/bpf: Add ret_mod " Jiri Olsa
@ 2021-11-18 11:24 ` Jiri Olsa
28 siblings, 0 replies; 49+ messages in thread
From: Jiri Olsa @ 2021-11-18 11:24 UTC (permalink / raw)
To: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Cc: netdev, bpf, Martin KaFai Lau, Song Liu, Yonghong Song,
John Fastabend, KP Singh
Adding test code to check on trampolines spliting.
The tests attached various bpf_fetry_* functions in a way
so there's always non trivial IDs intersection, that leads
to trampoline splitting in kenrel code.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 4 +-
.../bpf/prog_tests/multi_attach_test.c | 176 ++++++++++++++++++
.../selftests/bpf/progs/multi_attach.c | 105 +++++++++++
3 files changed, 284 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_attach_test.c
create mode 100644 tools/testing/selftests/bpf/progs/multi_attach.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 48970e983250..095d966a747a 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -324,7 +324,8 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h \
multi_fentry_test.skel.h multi_fexit_test.skel.h \
- multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h
+ multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h \
+ multi_attach_test.skel.h
LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \
test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c
@@ -340,6 +341,7 @@ multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o
multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o
multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o
multi_mixed_test.skel.h-deps := multi_mixed.o multi_check.o
+multi_attach_test.skel.h-deps := multi_attach.o multi_check.o
LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps)))
diff --git a/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c b/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c
new file mode 100644
index 000000000000..c183941215a6
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c
@@ -0,0 +1,176 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <linux/btf_ids.h>
+#include "multi_attach_test.skel.h"
+#include <bpf/btf.h>
+
+static __u32 btf_ids[8];
+
+static int load_btf_ids(void)
+{
+ __u32 i, nr_types, cnt;
+ struct btf *btf;
+
+ btf = btf__load_vmlinux_btf();
+ if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf"))
+ return -1;
+
+ nr_types = btf__get_nr_types(btf);
+
+ for (i = 1, cnt = 0; i <= nr_types && cnt < 8; i++) {
+ const struct btf_type *t = btf__type_by_id(btf, i);
+ const char *name;
+
+ if (!btf_is_func(t))
+ continue;
+
+ name = btf__name_by_offset(btf, t->name_off);
+ if (!name)
+ continue;
+ if (strncmp(name, "bpf_fentry_test", sizeof("bpf_fentry_test") - 1))
+ continue;
+
+ btf_ids[cnt] = i;
+ cnt++;
+ }
+
+ btf__free(btf);
+ return ASSERT_EQ(cnt, 8, "bpf_fentry_test_cnt") ? 0 : -1;
+}
+
+static int link_prog_from_cnt(const struct bpf_program *prog, int from, int cnt)
+{
+ DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts);
+ enum bpf_attach_type attach_type;
+ int prog_fd, link_fd;
+
+ opts.multi.btf_ids = btf_ids + (from - 1);
+ opts.multi.btf_ids_cnt = cnt;
+
+ prog_fd = bpf_program__fd(prog);
+ if (!ASSERT_GE(prog_fd, 0, "link_from_to_prog_fd"))
+ return -1;
+ attach_type = bpf_program__get_expected_attach_type(prog);
+ link_fd = bpf_link_create(prog_fd, 0, attach_type, &opts);
+ if (!ASSERT_GE(link_fd, 0, "link_from_to_link_fd"))
+ return -1;
+ return link_fd;
+}
+
+static int prog_from_cnt(const struct bpf_program *prog, int *from, int *cnt)
+{
+ const char *sec;
+ int err, to;
+
+ sec = bpf_program__section_name(prog);
+ sec = strchr(sec, '/');
+ if (!sec)
+ return -1;
+ sec++;
+ err = sscanf(sec, "bpf_fentry_test%d-%d", from, &to);
+ if (err != 2)
+ return -1;
+ *cnt = to - *from + 1;
+ return 0;
+}
+
+static int link_test(const struct bpf_program *prog1,
+ const struct bpf_program *prog2,
+ __u64 *test_result1, __u64 *test_result2,
+ bool do_close, int link_fd[2])
+{
+ int from1, cnt1, from2, cnt2, err;
+ __u32 duration = 0, retval;
+
+ if (!ASSERT_OK(prog_from_cnt(prog1, &from1, &cnt1), "prog_from_cnt__prog1"))
+ return -1;
+
+ if (!ASSERT_OK(prog_from_cnt(prog2, &from2, &cnt2), "prog_from_cnt__prog2"))
+ return -1;
+
+ link_fd[0] = link_prog_from_cnt(prog1, from1, cnt1);
+ if (link_fd[0] < 0)
+ return -1;
+
+ link_fd[1] = link_prog_from_cnt(prog2, from2, cnt2);
+ if (link_fd[1] < 0)
+ return -1;
+
+ *test_result1 = 0;
+ *test_result2 = 0;
+
+ err = bpf_prog_test_run(bpf_program__fd(prog1), 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+ ASSERT_EQ(*test_result1, cnt1, "test_result");
+ ASSERT_EQ(*test_result2, cnt2, "test_result");
+
+ if (do_close) {
+ close(link_fd[0]);
+ close(link_fd[1]);
+ }
+ return err;
+}
+
+void test_multi_attach_test(void)
+{
+ struct bpf_link *link7 = NULL, *link8 = NULL;
+ int link_fd[6] = { -1 }, i, err;
+ struct multi_attach_test *skel;
+ __u32 duration = 0, retval;
+
+ for (i = 0; i < 6; i++)
+ link_fd[i] = -1;
+
+ skel = multi_attach_test__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "multi_attach__load"))
+ return;
+
+ if (!ASSERT_OK(load_btf_ids(), "load_btf_ids"))
+ goto cleanup;
+
+#define LINK_TEST(__prog1, __prog2, __close) \
+ err = link_test(skel->progs.test ## __prog1, \
+ skel->progs.test ## __prog2, \
+ &skel->bss->test_result ## __prog1, \
+ &skel->bss->test_result ## __prog2, \
+ __close, link_fd + __prog1 - 1); \
+ if (err) \
+ goto cleanup;
+
+ LINK_TEST(1, 2, true);
+ LINK_TEST(3, 4, true);
+ LINK_TEST(1, 3, true);
+ LINK_TEST(2, 4, true);
+
+ LINK_TEST(1, 2, false);
+ LINK_TEST(3, 4, false);
+ LINK_TEST(5, 6, false);
+
+#undef LINK_TEST
+
+ link7 = bpf_program__attach(skel->progs.test7);
+ if (!ASSERT_OK_PTR(link7, "multi_attach_check__test1_attach"))
+ goto cleanup;
+
+ link8 = bpf_program__attach(skel->progs.test8);
+ if (!ASSERT_OK_PTR(link7, "multi_attach_check__test2_attach"))
+ goto cleanup;
+
+ err = bpf_prog_test_run(bpf_program__fd(skel->progs.test7), 1, NULL, 0,
+ NULL, NULL, &retval, &duration);
+
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(retval, 0, "test_run");
+ ASSERT_EQ(skel->bss->test_result7, 1, "test_result7");
+ ASSERT_EQ(skel->bss->test_result8, 1, "test_result8");
+
+cleanup:
+ bpf_link__destroy(link8);
+ bpf_link__destroy(link7);
+ for (i = 0; i < 6; i++)
+ close(link_fd[i]);
+ multi_attach_test__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/multi_attach.c b/tools/testing/selftests/bpf/progs/multi_attach.c
new file mode 100644
index 000000000000..d403f5f1f27e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/multi_attach.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+char _license[] SEC("license") = "GPL";
+
+__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result);
+__hidden extern void multi_ret_check(void *ctx, __u64 *test_result);
+
+__u64 test_result1 = 0;
+
+SEC("fentry.multi/bpf_fentry_test1-5")
+int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ multi_arg_check(ctx, &test_result1);
+ return 0;
+}
+
+__u64 test_result2 = 0;
+
+SEC("fentry.multi/bpf_fentry_test4-8")
+int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ multi_arg_check(ctx, &test_result2);
+ return 0;
+}
+
+__u64 test_result3 = 0;
+
+SEC("fexit.multi/bpf_fentry_test1-5")
+int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ __u64 arg_result = 0, ret_result = 0;
+
+ multi_arg_check(ctx, &arg_result);
+ multi_ret_check(ctx, &ret_result);
+
+ if (arg_result && ret_result)
+ test_result3 += 1;
+ return 0;
+}
+
+__u64 test_result4 = 0;
+
+SEC("fexit.multi/bpf_fentry_test4-8")
+int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret)
+{
+ __u64 arg_result = 0, ret_result = 0;
+
+ multi_arg_check(ctx, &arg_result);
+ multi_ret_check(ctx, &ret_result);
+
+ if (arg_result && ret_result)
+ test_result4 += 1;
+ return 0;
+}
+
+__u64 test_result5 = 0;
+
+SEC("fentry.multi/bpf_fentry_test1-8")
+int BPF_PROG(test5, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ multi_arg_check(ctx, &test_result5);
+ return 0;
+}
+
+__u64 test_result6 = 0;
+
+SEC("fexit.multi/bpf_fentry_test1-8")
+int BPF_PROG(test6, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f)
+{
+ __u64 arg_result = 0, ret_result = 0;
+
+ multi_arg_check(ctx, &arg_result);
+ multi_ret_check(ctx, &ret_result);
+
+ if (arg_result && ret_result)
+ test_result6 += 1;
+ return 0;
+}
+
+__u64 test_result7 = 0;
+
+SEC("fentry/bpf_fentry_test1")
+int BPF_PROG(test7, int a)
+{
+ multi_arg_check(ctx, &test_result7);
+ return 0;
+}
+
+__u64 test_result8 = 0;
+
+SEC("fexit/bpf_fentry_test2")
+int BPF_PROG(test8, int a, __u64 b, int ret)
+{
+ __u64 arg_result = 0, ret_result = 0;
+
+ multi_arg_check(ctx, &arg_result);
+ multi_ret_check(ctx, &ret_result);
+
+ if (arg_result && ret_result)
+ test_result8 += 1;
+ return 0;
+}
--
2.31.1
^ permalink raw reply related [flat|nested] 49+ messages in thread