* [PATCH bpf-next] bpf: avoid retpoline for bpf_for_each_map_elem
@ 2021-10-06 0:18 Andrey Ignatov
2021-10-06 2:28 ` Alexei Starovoitov
0 siblings, 1 reply; 2+ messages in thread
From: Andrey Ignatov @ 2021-10-06 0:18 UTC (permalink / raw)
To: bpf; +Cc: Andrey Ignatov, ast, daniel, andrii, kernel-team
Similarly to 09772d92cd5a ("bpf: avoid retpoline for
lookup/update/delete calls on maps") and 84430d4232c3 ("bpf, verifier:
avoid retpoline for map push/pop/peek operation") avoid indirect call
while calling bpf_for_each_map_elem.
Before (a program fragment):
; if (rules_map) {
142: (15) if r4 == 0x0 goto pc+8
143: (bf) r3 = r10
; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
144: (07) r3 += -24
145: (bf) r1 = r4
146: (18) r2 = subprog[+5]
148: (b7) r4 = 0
149: (85) call bpf_for_each_map_elem#143680 <-- indirect call via
helper
After (same program fragment):
; if (rules_map) {
142: (15) if r4 == 0x0 goto pc+8
143: (bf) r3 = r10
; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
144: (07) r3 += -24
145: (bf) r1 = r4
146: (18) r2 = subprog[+5]
148: (b7) r4 = 0
149: (85) call bpf_for_each_array_elem#170336 <-- direct call
On a benchmark that calls bpf_for_each_map_elem() once and does many
other things (mostly checking fields in skb) with CONFIG_RETPOLINE=y it
makes program faster.
Before:
============================================================================
Benchmark.cpp time/iter iters/s
============================================================================
IngressMatchByRemoteEndpoint 80.78ns 12.38M
IngressMatchByRemoteIP 80.66ns 12.40M
IngressMatchByRemotePort 80.87ns 12.37M
After:
============================================================================
Benchmark.cpp time/iter iters/s
============================================================================
IngressMatchByRemoteEndpoint 73.49ns 13.61M
IngressMatchByRemoteIP 71.48ns 13.99M
IngressMatchByRemotePort 70.39ns 14.21M
Signed-off-by: Andrey Ignatov <rdna@fb.com>
---
kernel/bpf/verifier.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1433752db740..68948f1ed443 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -12946,7 +12946,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
insn->imm == BPF_FUNC_map_push_elem ||
insn->imm == BPF_FUNC_map_pop_elem ||
insn->imm == BPF_FUNC_map_peek_elem ||
- insn->imm == BPF_FUNC_redirect_map)) {
+ insn->imm == BPF_FUNC_redirect_map ||
+ insn->imm == BPF_FUNC_for_each_map_elem)) {
aux = &env->insn_aux_data[i + delta];
if (bpf_map_ptr_poisoned(aux))
goto patch_call_imm;
@@ -12990,6 +12991,11 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
(int (*)(struct bpf_map *map, void *value))NULL));
BUILD_BUG_ON(!__same_type(ops->map_redirect,
(int (*)(struct bpf_map *map, u32 ifindex, u64 flags))NULL));
+ BUILD_BUG_ON(!__same_type(ops->map_for_each_callback,
+ (int (*)(struct bpf_map *map,
+ bpf_callback_t callback_fn,
+ void *callback_ctx,
+ u64 flags))NULL));
patch_map_ops_generic:
switch (insn->imm) {
@@ -13014,6 +13020,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
case BPF_FUNC_redirect_map:
insn->imm = BPF_CALL_IMM(ops->map_redirect);
continue;
+ case BPF_FUNC_for_each_map_elem:
+ insn->imm = BPF_CALL_IMM(ops->map_for_each_callback);
+ continue;
}
goto patch_call_imm;
--
2.30.2
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH bpf-next] bpf: avoid retpoline for bpf_for_each_map_elem
2021-10-06 0:18 [PATCH bpf-next] bpf: avoid retpoline for bpf_for_each_map_elem Andrey Ignatov
@ 2021-10-06 2:28 ` Alexei Starovoitov
0 siblings, 0 replies; 2+ messages in thread
From: Alexei Starovoitov @ 2021-10-06 2:28 UTC (permalink / raw)
To: Andrey Ignatov
Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Kernel Team
On Tue, Oct 5, 2021 at 5:19 PM Andrey Ignatov <rdna@fb.com> wrote:
>
> Before:
>
> IngressMatchByRemoteEndpoint 80.78ns 12.38M
> IngressMatchByRemoteIP 80.66ns 12.40M
> IngressMatchByRemotePort 80.87ns 12.37M
>
> After:
>
> IngressMatchByRemoteEndpoint 73.49ns 13.61M
> IngressMatchByRemoteIP 71.48ns 13.99M
> IngressMatchByRemotePort 70.39ns 14.21M
Nice gains :)
Applied.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-10-06 2:28 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-06 0:18 [PATCH bpf-next] bpf: avoid retpoline for bpf_for_each_map_elem Andrey Ignatov
2021-10-06 2:28 ` Alexei Starovoitov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).