* [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing
@ 2022-11-03 12:00 Peter Zijlstra
2022-11-03 12:00 ` [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") Peter Zijlstra
` (4 more replies)
0 siblings, 5 replies; 7+ messages in thread
From: Peter Zijlstra @ 2022-11-03 12:00 UTC (permalink / raw)
To: olsajiri, ast, daniel
Cc: linux-kernel, netdev, bpf, peterz, bjorn, toke, David.Laight, rostedt
Hi!
Even thought the __attribute__((patchable_function_entry())) solution to the
BPF dispatcher woes works, it turns out to not be supported by the whole range
of ageing compilers we support. Specifically this attribute seems to be GCC-8
and later.
This is another approach -- using static_call() to rewrite the dispatcher
function. I've compile tested this on:
x86_64 (inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)
A previous version was tested and found working by Bjorn.
It is split in two patches; first reverting the current approach and then
introducing the new for ease of review.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop")
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
@ 2022-11-03 12:00 ` Peter Zijlstra
2022-11-03 12:00 ` [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace) Peter Zijlstra
` (3 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Peter Zijlstra @ 2022-11-03 12:00 UTC (permalink / raw)
To: olsajiri, ast, daniel
Cc: linux-kernel, netdev, bpf, peterz, bjorn, toke, David.Laight, rostedt
Because __attribute__((patchable_function_entry)) is only available
since GCC-8 this solution fails to build on the minimum required GCC
version.
Undo these changes so we might try again -- without cluttering up the
patches with too many changes.
This is an almost complete revert of:
dbe69b299884 ("bpf: Fix dispatcher patchable function entry to 5 bytes nop")
ceea991a019c ("bpf: Move bpf_dispatcher function out of ftrace locations")
(notably the arch/x86/Kconfig hunk is kept).
Reported-by: David Laight <David.Laight@aculab.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/439d8dc735bb4858875377df67f1b29a@AcuMS.aculab.com
---
arch/x86/net/bpf_jit_comp.c | 13 -------------
include/linux/bpf.h | 21 +--------------------
kernel/bpf/dispatcher.c | 6 ------
3 files changed, 1 insertion(+), 39 deletions(-)
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -11,7 +11,6 @@
#include <linux/bpf.h>
#include <linux/memory.h>
#include <linux/sort.h>
-#include <linux/init.h>
#include <asm/extable.h>
#include <asm/set_memory.h>
#include <asm/nospec-branch.h>
@@ -389,18 +388,6 @@ static int __bpf_arch_text_poke(void *ip
return ret;
}
-int __init bpf_arch_init_dispatcher_early(void *ip)
-{
- const u8 *nop_insn = x86_nops[5];
-
- if (is_endbr(*(u32 *)ip))
- ip += ENDBR_INSN_SIZE;
-
- if (memcmp(ip, nop_insn, X86_PATCH_SIZE))
- text_poke_early(ip, nop_insn, X86_PATCH_SIZE);
- return 0;
-}
-
int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
void *old_addr, void *new_addr)
{
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -27,7 +27,6 @@
#include <linux/bpfptr.h>
#include <linux/btf.h>
#include <linux/rcupdate_trace.h>
-#include <linux/init.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
@@ -971,8 +970,6 @@ struct bpf_trampoline *bpf_trampoline_ge
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
-int __init bpf_arch_init_dispatcher_early(void *ip);
-
#define BPF_DISPATCHER_INIT(_name) { \
.mutex = __MUTEX_INITIALIZER(_name.mutex), \
.func = &_name##_func, \
@@ -986,21 +983,7 @@ int __init bpf_arch_init_dispatcher_earl
}, \
}
-#define BPF_DISPATCHER_INIT_CALL(_name) \
- static int __init _name##_init(void) \
- { \
- return bpf_arch_init_dispatcher_early(_name##_func); \
- } \
- early_initcall(_name##_init)
-
-#ifdef CONFIG_X86_64
-#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
-#else
-#define BPF_DISPATCHER_ATTRIBUTES
-#endif
-
#define DEFINE_BPF_DISPATCHER(name) \
- notrace BPF_DISPATCHER_ATTRIBUTES \
noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
@@ -1010,9 +993,7 @@ int __init bpf_arch_init_dispatcher_earl
} \
EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \
struct bpf_dispatcher bpf_dispatcher_##name = \
- BPF_DISPATCHER_INIT(bpf_dispatcher_##name); \
- BPF_DISPATCHER_INIT_CALL(bpf_dispatcher_##name);
-
+ BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
#define DECLARE_BPF_DISPATCHER(name) \
unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
--- a/kernel/bpf/dispatcher.c
+++ b/kernel/bpf/dispatcher.c
@@ -4,7 +4,6 @@
#include <linux/hash.h>
#include <linux/bpf.h>
#include <linux/filter.h>
-#include <linux/init.h>
/* The BPF dispatcher is a multiway branch code generator. The
* dispatcher is a mechanism to avoid the performance penalty of an
@@ -90,11 +89,6 @@ int __weak arch_prepare_bpf_dispatcher(v
{
return -ENOTSUPP;
}
-
-int __weak __init bpf_arch_init_dispatcher_early(void *ip)
-{
- return -ENOTSUPP;
-}
static int bpf_dispatcher_prepare(struct bpf_dispatcher *d, void *image, void *buf)
{
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
2022-11-03 12:00 ` [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") Peter Zijlstra
@ 2022-11-03 12:00 ` Peter Zijlstra
2022-11-14 17:36 ` Steven Rostedt
2022-11-03 13:42 ` [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Björn Töpel
` (2 subsequent siblings)
4 siblings, 1 reply; 7+ messages in thread
From: Peter Zijlstra @ 2022-11-03 12:00 UTC (permalink / raw)
To: olsajiri, ast, daniel
Cc: linux-kernel, netdev, bpf, peterz, bjorn, toke, David.Laight, rostedt
The dispatcher function is currently abusing the ftrace __fentry__
call location for its own purposes -- this obviously gives trouble
when the dispatcher and ftrace are both in use.
A previous solution tried using __attribute__((patchable_function_entry()))
which works, except it is GCC-8+ only, breaking the build on the
earlier still supported compilers. Instead use static_call() -- which
has its own annotations and does not conflict with ftrace -- to
rewrite the dispatch function.
By using: return static_call()(ctx, insni, bpf_func) you get a perfect
forwarding tail call as function body (iow a single jmp instruction).
By having the default static_call() target be
bpf_dispatcher_nop_func() it retains the default behaviour (an
indirect call to the argument function). Only
once a dispatcher program is attached is the target rewritten to
directly call the JIT'ed image.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/Y1/oBlK0yFk5c/Im@hirez.programming.kicks-ass.net
---
include/linux/bpf.h | 39 ++++++++++++++++++++++++++++++++++++++-
kernel/bpf/dispatcher.c | 22 ++++++++--------------
2 files changed, 46 insertions(+), 15 deletions(-)
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -27,6 +27,7 @@
#include <linux/bpfptr.h>
#include <linux/btf.h>
#include <linux/rcupdate_trace.h>
+#include <linux/static_call.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
@@ -953,6 +954,10 @@ struct bpf_dispatcher {
void *rw_image;
u32 image_off;
struct bpf_ksym ksym;
+#ifdef CONFIG_HAVE_STATIC_CALL
+ struct static_call_key *sc_key;
+ void *sc_tramp;
+#endif
};
static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
@@ -970,6 +975,34 @@ struct bpf_trampoline *bpf_trampoline_ge
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
+
+/*
+ * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
+ * indirection with a direct call to the bpf program. If the architecture does
+ * not have STATIC_CALL, avoid a double-indirection.
+ */
+#ifdef CONFIG_HAVE_STATIC_CALL
+
+#define __BPF_DISPATCHER_SC_INIT(_name) \
+ .sc_key = &STATIC_CALL_KEY(_name), \
+ .sc_tramp = STATIC_CALL_TRAMP_ADDR(_name),
+
+#define __BPF_DISPATCHER_SC(name) \
+ DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func)
+
+#define __BPF_DISPATCHER_CALL(name) \
+ static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func)
+
+#define __BPF_DISPATCHER_UPDATE(_d, _new) \
+ __static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new))
+
+#else
+#define __BPF_DISPATCHER_SC_INIT(name)
+#define __BPF_DISPATCHER_SC(name)
+#define __BPF_DISPATCHER_CALL(name) bpf_func(ctx, insnsi)
+#define __BPF_DISPATCHER_UPDATE(_d, _new)
+#endif
+
#define BPF_DISPATCHER_INIT(_name) { \
.mutex = __MUTEX_INITIALIZER(_name.mutex), \
.func = &_name##_func, \
@@ -981,25 +1014,29 @@ int arch_prepare_bpf_dispatcher(void *im
.name = #_name, \
.lnode = LIST_HEAD_INIT(_name.ksym.lnode), \
}, \
+ __BPF_DISPATCHER_SC_INIT(_name##_call) \
}
#define DEFINE_BPF_DISPATCHER(name) \
+ __BPF_DISPATCHER_SC(name); \
noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func) \
{ \
- return bpf_func(ctx, insnsi); \
+ return __BPF_DISPATCHER_CALL(name); \
} \
EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \
struct bpf_dispatcher bpf_dispatcher_##name = \
BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
+
#define DECLARE_BPF_DISPATCHER(name) \
unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func); \
extern struct bpf_dispatcher bpf_dispatcher_##name;
+
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func
#define BPF_DISPATCHER_PTR(name) (&bpf_dispatcher_##name)
void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
--- a/kernel/bpf/dispatcher.c
+++ b/kernel/bpf/dispatcher.c
@@ -4,6 +4,7 @@
#include <linux/hash.h>
#include <linux/bpf.h>
#include <linux/filter.h>
+#include <linux/static_call.h>
/* The BPF dispatcher is a multiway branch code generator. The
* dispatcher is a mechanism to avoid the performance penalty of an
@@ -104,17 +105,11 @@ static int bpf_dispatcher_prepare(struct
static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs)
{
- void *old, *new, *tmp;
- u32 noff;
- int err;
-
- if (!prev_num_progs) {
- old = NULL;
- noff = 0;
- } else {
- old = d->image + d->image_off;
+ void *new, *tmp;
+ u32 noff = 0;
+
+ if (prev_num_progs)
noff = d->image_off ^ (PAGE_SIZE / 2);
- }
new = d->num_progs ? d->image + noff : NULL;
tmp = d->num_progs ? d->rw_image + noff : NULL;
@@ -128,11 +123,10 @@ static void bpf_dispatcher_update(struct
return;
}
- err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP, old, new);
- if (err || !new)
- return;
+ __BPF_DISPATCHER_UPDATE(d, new ?: &bpf_dispatcher_nop_func);
- d->image_off = noff;
+ if (new)
+ d->image_off = noff;
}
void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
2022-11-03 12:00 ` [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") Peter Zijlstra
2022-11-03 12:00 ` [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace) Peter Zijlstra
@ 2022-11-03 13:42 ` Björn Töpel
2022-11-03 14:48 ` Jiri Olsa
2022-11-04 22:20 ` patchwork-bot+netdevbpf
4 siblings, 0 replies; 7+ messages in thread
From: Björn Töpel @ 2022-11-03 13:42 UTC (permalink / raw)
To: Peter Zijlstra, olsajiri, ast, daniel
Cc: linux-kernel, netdev, bpf, peterz, toke, David.Laight, rostedt
Peter Zijlstra <peterz@infradead.org> writes:
> Hi!
>
> Even thought the __attribute__((patchable_function_entry())) solution to the
> BPF dispatcher woes works, it turns out to not be supported by the whole range
> of ageing compilers we support. Specifically this attribute seems to be GCC-8
> and later.
>
> This is another approach -- using static_call() to rewrite the dispatcher
> function. I've compile tested this on:
>
> x86_64 (inline static-call support)
> i386 (out-of-line static-call support)
> aargh64 (no static-call support)
>
> A previous version was tested and found working by Bjorn.
>
> It is split in two patches; first reverting the current approach and then
> introducing the new for ease of review.
Took it for a spin on x86_64/KVM. For the series:
Acked-by: Björn Töpel <bjorn@kernel.org>
Tested-by: Björn Töpel <bjorn@kernel.org>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
` (2 preceding siblings ...)
2022-11-03 13:42 ` [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Björn Töpel
@ 2022-11-03 14:48 ` Jiri Olsa
2022-11-04 22:20 ` patchwork-bot+netdevbpf
4 siblings, 0 replies; 7+ messages in thread
From: Jiri Olsa @ 2022-11-03 14:48 UTC (permalink / raw)
To: Peter Zijlstra
Cc: olsajiri, ast, daniel, linux-kernel, netdev, bpf, bjorn, toke,
David.Laight, rostedt
On Thu, Nov 03, 2022 at 01:00:12PM +0100, Peter Zijlstra wrote:
> Hi!
>
> Even thought the __attribute__((patchable_function_entry())) solution to the
> BPF dispatcher woes works, it turns out to not be supported by the whole range
> of ageing compilers we support. Specifically this attribute seems to be GCC-8
> and later.
>
> This is another approach -- using static_call() to rewrite the dispatcher
> function. I've compile tested this on:
>
> x86_64 (inline static-call support)
> i386 (out-of-line static-call support)
> aargh64 (no static-call support)
>
> A previous version was tested and found working by Bjorn.
>
> It is split in two patches; first reverting the current approach and then
> introducing the new for ease of review.
>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Jiri Olsa <jolsa@kernel.org>
thanks,
jirka
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
` (3 preceding siblings ...)
2022-11-03 14:48 ` Jiri Olsa
@ 2022-11-04 22:20 ` patchwork-bot+netdevbpf
4 siblings, 0 replies; 7+ messages in thread
From: patchwork-bot+netdevbpf @ 2022-11-04 22:20 UTC (permalink / raw)
To: Peter Zijlstra
Cc: olsajiri, ast, daniel, linux-kernel, netdev, bpf, bjorn, toke,
David.Laight, rostedt
Hello:
This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <daniel@iogearbox.net>:
On Thu, 03 Nov 2022 13:00:12 +0100 you wrote:
> Hi!
>
> Even thought the __attribute__((patchable_function_entry())) solution to the
> BPF dispatcher woes works, it turns out to not be supported by the whole range
> of ageing compilers we support. Specifically this attribute seems to be GCC-8
> and later.
>
> [...]
Here is the summary with links:
- [1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop")
(no matching commit)
- [2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
https://git.kernel.org/bpf/bpf/c/c86df29d11df
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
2022-11-03 12:00 ` [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace) Peter Zijlstra
@ 2022-11-14 17:36 ` Steven Rostedt
0 siblings, 0 replies; 7+ messages in thread
From: Steven Rostedt @ 2022-11-14 17:36 UTC (permalink / raw)
To: Peter Zijlstra
Cc: olsajiri, ast, daniel, linux-kernel, netdev, bpf, bjorn, toke,
David.Laight
On Thu, 03 Nov 2022 13:00:14 +0100
Peter Zijlstra <peterz@infradead.org> wrote:
> The dispatcher function is currently abusing the ftrace __fentry__
> call location for its own purposes -- this obviously gives trouble
> when the dispatcher and ftrace are both in use.
Awesome! Thanks for doing this Peter!
Acked-by: Steven Rostedt (Google) <rostedt@goodmis.org>
-- Steve
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-11-14 17:35 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-03 12:00 [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Peter Zijlstra
2022-11-03 12:00 ` [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") Peter Zijlstra
2022-11-03 12:00 ` [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace) Peter Zijlstra
2022-11-14 17:36 ` Steven Rostedt
2022-11-03 13:42 ` [PATCH 0/2] bpf: Yet another approach to fix the BPF dispatcher thing Björn Töpel
2022-11-03 14:48 ` Jiri Olsa
2022-11-04 22:20 ` patchwork-bot+netdevbpf
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.