From: Andrii Nakryiko <andrii@kernel.org>
To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net,
martin.lau@kernel.org
Cc: andrii@kernel.org, kernel-team@meta.com
Subject: [PATCH bpf-next] bpf: mark kprobe_multi_link_prog_run as always inlined function
Date: Wed, 20 Mar 2024 13:06:10 -0700 [thread overview]
Message-ID: <20240320200610.2556049-1-andrii@kernel.org> (raw)
kprobe_multi_link_prog_run() is called both for multi-kprobe and
multi-kretprobe BPF programs from kprobe_multi_link_handler() and
kprobe_multi_link_exit_handler(), respectively.
kprobe_multi_link_prog_run() is doing all the relevant work, with those
wrappers just satisfying ftrace's interfaces (kprobe callback is
supposed to return int, while kretprobe returns void).
With this structure compile performs tail-call optimization:
Dump of assembler code for function kprobe_multi_link_exit_handler:
0xffffffff8122f1e0 <+0>: add $0xffffffffffffffc0,%rdi
0xffffffff8122f1e4 <+4>: mov %rcx,%rdx
0xffffffff8122f1e7 <+7>: jmp 0xffffffff81230080 <kprobe_multi_link_prog_run>
This means that when trying to capture LBR that traces all indirect branches
we are wasting an entry just to record that kprobe_multi_link_exit_handler
called/jumped into kprobe_multi_link_prog_run.
LBR entries are especially sparse on AMD CPUs (just 16 entries on latest CPUs
vs typically 32 on latest Intel CPUs), and every entry counts (and we already
have a bunch of other LBR entries spent getting to a BPF program), so it would
be great to not waste any more than necessary.
Marking it as just `static inline` doesn't change anything, compiler
still performs tail call optimization only. But by marking
kprobe_multi_link_prog_run() as __always_inline we ensure that compiler
fully inlines it, avoiding jumps:
Dump of assembler code for function kprobe_multi_link_exit_handler:
0xffffffff8122f4e0 <+0>: push %r15
0xffffffff8122f4e2 <+2>: push %r14
0xffffffff8122f4e4 <+4>: push %r13
0xffffffff8122f4e6 <+6>: push %r12
0xffffffff8122f4e8 <+8>: push %rbx
0xffffffff8122f4e9 <+9>: sub $0x10,%rsp
0xffffffff8122f4ed <+13>: mov %rdi,%r14
0xffffffff8122f4f0 <+16>: lea -0x40(%rdi),%rax
...
0xffffffff8122f590 <+176>: call 0xffffffff8108e420 <sched_clock>
0xffffffff8122f595 <+181>: sub %r14,%rax
0xffffffff8122f598 <+184>: add %rax,0x8(%rbx,%r13,1)
0xffffffff8122f59d <+189>: jmp 0xffffffff8122f541 <kprobe_multi_link_exit_handler+97>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
---
kernel/trace/bpf_trace.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 434e3ece6688..0bebd6f02e17 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -2796,7 +2796,7 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
return run_ctx->entry_ip;
}
-static int
+static __always_inline int
kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,
unsigned long entry_ip, struct pt_regs *regs)
{
--
2.43.0
next reply other threads:[~2024-03-20 20:06 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-20 20:06 Andrii Nakryiko [this message]
2024-03-21 6:55 ` [PATCH bpf-next] bpf: mark kprobe_multi_link_prog_run as always inlined function Alexei Starovoitov
2024-03-21 7:02 ` Alexei Starovoitov
2024-03-21 16:04 ` Andrii Nakryiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240320200610.2556049-1-andrii@kernel.org \
--to=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=kernel-team@meta.com \
--cc=martin.lau@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.