From: Yonghong Song <yhs@fb.com>
To: Jiri Olsa <jolsa@redhat.com>,
Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andriin@fb.com>,
"Steven Rostedt (VMware)" <rostedt@goodmis.org>,
Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@chromium.org>, Daniel Xu <dxu@dxuuu.xyz>,
Viktor Malik <vmalik@redhat.com>
Subject: Re: [RFCv3 00/19] x86/ftrace/bpf: Add batch support for direct/tracing attach
Date: Sat, 19 Jun 2021 09:19:57 -0700 [thread overview]
Message-ID: <4af931a5-3c43-9571-22ac-63e5d299fa42@fb.com> (raw)
In-Reply-To: <YM2r139rHuXialVG@krava>
On 6/19/21 1:33 AM, Jiri Olsa wrote:
> On Thu, Jun 17, 2021 at 01:29:45PM -0700, Andrii Nakryiko wrote:
>> On Sat, Jun 5, 2021 at 4:12 AM Jiri Olsa <jolsa@kernel.org> wrote:
>>>
>>> hi,
>>> saga continues.. ;-) previous post is in here [1]
>>>
>>> After another discussion with Steven, he mentioned that if we fix
>>> the ftrace graph problem with direct functions, he'd be open to
>>> add batch interface for direct ftrace functions.
>>>
>>> He already had prove of concept fix for that, which I took and broke
>>> up into several changes. I added the ftrace direct batch interface
>>> and bpf new interface on top of that.
>>>
>>> It's not so many patches after all, so I thought having them all
>>> together will help the review, because they are all connected.
>>> However I can break this up into separate patchsets if necessary.
>>>
>>> This patchset contains:
>>>
>>> 1) patches (1-4) that fix the ftrace graph tracing over the function
>>> with direct trampolines attached
>>> 2) patches (5-8) that add batch interface for ftrace direct function
>>> register/unregister/modify
>>> 3) patches (9-19) that add support to attach BPF program to multiple
>>> functions
>>>
>>> In nutshell:
>>>
>>> Ad 1) moves the graph tracing setup before the direct trampoline
>>> prepares the stack, so they don't clash
>>>
>>> Ad 2) uses ftrace_ops interface to register direct function with
>>> all functions in ftrace_ops filter.
>>>
>>> Ad 3) creates special program and trampoline type to allow attachment
>>> of multiple functions to single program.
>>>
>>> There're more detailed desriptions in related changelogs.
>>>
>>> I have working bpftrace multi attachment code on top this. I briefly
>>> checked retsnoop and I think it could use the new API as well.
>>
>> Ok, so I had a bit of time and enthusiasm to try that with retsnoop.
>> The ugly code is at [0] if you'd like to see what kind of changes I
>> needed to make to use this (it won't work if you check it out because
>> it needs your libbpf changes synced into submodule, which I only did
>> locally). But here are some learnings from that experiment both to
>> emphasize how important it is to make this work and how restrictive
>> are some of the current limitations.
>>
>> First, good news. Using this mass-attach API to attach to almost 1000
>> kernel functions goes from
>>
>> Plain fentry/fexit:
>> ===================
>> real 0m27.321s
>> user 0m0.352s
>> sys 0m20.919s
>>
>> to
>>
>> Mass-attach fentry/fexit:
>> =========================
>> real 0m2.728s
>> user 0m0.329s
>> sys 0m2.380s
>
> I did not meassured the bpftrace speedup, because the new code
> attached instantly ;-)
>
>>
>> It's a 10x speed up. And a good chunk of those 2.7 seconds is in some
>> preparatory steps not related to fentry/fexit stuff.
>>
>> It's not exactly apples-to-apples, though, because the limitations you
>> have right now prevents attaching both fentry and fexit programs to
>> the same set of kernel functions. This makes it pretty useless for a
>
> hum, you could do link_update with fexit program on the link fd,
> like in the selftest, right?
>
>> lot of cases, in particular for retsnoop. So I haven't really tested
>> retsnoop end-to-end, I only verified that I do see fentries triggered,
>> but can't have matching fexits. So the speed-up might be smaller due
>> to additional fexit mass-attach (once that is allowed), but it's still
>> a massive difference. So we absolutely need to get this optimization
>> in.
>>
>> Few more thoughts, if you'd like to plan some more work ahead ;)
>>
>> 1. We need similar mass-attach functionality for kprobe/kretprobe, as
>> there are use cases where kprobe are more useful than fentry (e.g., >6
>> args funcs, or funcs with input arguments that are not supported by
>> BPF verifier, like struct-by-value). It's not clear how to best
>> represent this, given currently we attach kprobe through perf_event,
>> but we'll need to think about this for sure.
>
> I'm fighting with the '2 trampolines concept' at the moment, but the
> mass attach for kprobes seems interesting ;-) will check
>
>>
>> 2. To make mass-attach fentry/fexit useful for practical purposes, it
>> would be really great to have an ability to fetch traced function's
>> IP. I.e., if we fentry/fexit func kern_func_abc, bpf_get_func_ip()
>> would return IP of that functions that matches the one in
>> /proc/kallsyms. Right now I do very brittle hacks to do that.
>
> so I hoped that we could store ip always in ctx-8 and have
> the bpf_get_func_ip helper to access that, but the BPF_PROG
> macro does not pass ctx value to the program, just args
ctx does pass to the bpf program. You can check BPF_PROG
macro definition.
>
> we could perhaps somehow store the ctx in BPF_PROG before calling
> the bpf program, but I did not get to try that yet
>
>>
>> So all-in-all, super excited about this, but I hope all those issues
>> are addressed to make retsnoop possible and fast.
>>
>> [0] https://github.com/anakryiko/retsnoop/commit/8a07bc4d8c47d025f755c108f92f0583e3fda6d8
>
> thanks for checking on this,
> jirka
>
next prev parent reply other threads:[~2021-06-19 16:20 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-05 11:10 [RFCv3 00/19] x86/ftrace/bpf: Add batch support for direct/tracing attach Jiri Olsa
2021-06-05 11:10 ` [PATCH 01/19] x86/ftrace: Remove extra orig rax move Jiri Olsa
2021-06-05 11:10 ` [PATCH 02/19] x86/ftrace: Remove fault protection code in prepare_ftrace_return Jiri Olsa
2021-06-05 11:10 ` [PATCH 03/19] x86/ftrace: Make function graph use ftrace directly Jiri Olsa
2021-06-08 18:35 ` Andrii Nakryiko
2021-06-08 18:51 ` Jiri Olsa
2021-06-08 19:11 ` Steven Rostedt
2021-06-05 11:10 ` [PATCH 04/19] tracing: Add trampoline/graph selftest Jiri Olsa
2021-06-05 11:10 ` [PATCH 05/19] ftrace: Add ftrace_add_rec_direct function Jiri Olsa
2021-06-05 11:10 ` [PATCH 06/19] ftrace: Add multi direct register/unregister interface Jiri Olsa
2021-06-05 11:10 ` [PATCH 07/19] ftrace: Add multi direct modify interface Jiri Olsa
2021-06-05 11:10 ` [PATCH 08/19] ftrace/samples: Add multi direct interface test module Jiri Olsa
2021-06-05 11:10 ` [PATCH 09/19] bpf, x64: Allow to use caller address from stack Jiri Olsa
2021-06-07 3:07 ` Yonghong Song
2021-06-07 18:13 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 10/19] bpf: Allow to store caller's ip as argument Jiri Olsa
2021-06-07 3:21 ` Yonghong Song
2021-06-07 18:15 ` Jiri Olsa
2021-06-08 18:49 ` Andrii Nakryiko
2021-06-08 20:58 ` Jiri Olsa
2021-06-08 21:02 ` Andrii Nakryiko
2021-06-08 21:11 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 11/19] bpf: Add support to load multi func tracing program Jiri Olsa
2021-06-07 3:56 ` Yonghong Song
2021-06-07 18:18 ` Jiri Olsa
2021-06-07 19:35 ` Yonghong Song
2021-06-05 11:10 ` [PATCH 12/19] bpf: Add bpf_trampoline_alloc function Jiri Olsa
2021-06-05 11:10 ` [PATCH 13/19] bpf: Add support to link multi func tracing program Jiri Olsa
2021-06-07 5:36 ` Yonghong Song
2021-06-07 18:25 ` Jiri Olsa
2021-06-07 19:39 ` Yonghong Song
2021-06-08 15:42 ` Alexei Starovoitov
2021-06-08 18:17 ` Jiri Olsa
2021-06-08 18:49 ` Alexei Starovoitov
2021-06-08 21:07 ` Jiri Olsa
2021-06-08 23:05 ` Alexei Starovoitov
2021-06-09 5:08 ` Andrii Nakryiko
2021-06-09 13:42 ` Jiri Olsa
2021-06-09 13:33 ` Jiri Olsa
2021-06-09 5:18 ` Andrii Nakryiko
2021-06-09 13:53 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 14/19] libbpf: Add btf__find_by_pattern_kind function Jiri Olsa
2021-06-09 5:29 ` Andrii Nakryiko
2021-06-09 13:59 ` Jiri Olsa
2021-06-09 14:19 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 15/19] libbpf: Add support to link multi func tracing program Jiri Olsa
2021-06-07 5:49 ` Yonghong Song
2021-06-07 18:28 ` Jiri Olsa
2021-06-07 19:42 ` Yonghong Song
2021-06-07 20:11 ` Jiri Olsa
2021-06-09 5:34 ` Andrii Nakryiko
2021-06-09 14:17 ` Jiri Olsa
2021-06-10 17:05 ` Andrii Nakryiko
2021-06-10 20:35 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 16/19] selftests/bpf: Add fentry multi func test Jiri Olsa
2021-06-07 6:06 ` Yonghong Song
2021-06-07 18:42 ` Jiri Olsa
2021-06-09 5:40 ` Andrii Nakryiko
2021-06-09 14:29 ` Jiri Olsa
2021-06-10 17:00 ` Andrii Nakryiko
2021-06-10 20:28 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 17/19] selftests/bpf: Add fexit " Jiri Olsa
2021-06-05 11:10 ` [PATCH 18/19] selftests/bpf: Add fentry/fexit " Jiri Olsa
2021-06-09 5:41 ` Andrii Nakryiko
2021-06-09 14:29 ` Jiri Olsa
2021-06-05 11:10 ` [PATCH 19/19] selftests/bpf: Temporary fix for fentry_fexit_multi_test Jiri Olsa
2021-06-17 20:29 ` [RFCv3 00/19] x86/ftrace/bpf: Add batch support for direct/tracing attach Andrii Nakryiko
2021-06-19 8:33 ` Jiri Olsa
2021-06-19 16:19 ` Yonghong Song [this message]
2021-06-19 17:09 ` Jiri Olsa
2021-06-20 16:56 ` Yonghong Song
2021-06-20 17:47 ` Alexei Starovoitov
2021-06-21 6:46 ` Andrii Nakryiko
2021-06-21 6:50 ` Andrii Nakryiko
2021-07-06 20:26 ` Andrii Nakryiko
2021-07-07 15:19 ` Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4af931a5-3c43-9571-22ac-63e5d299fa42@fb.com \
--to=yhs@fb.com \
--cc=andrii.nakryiko@gmail.com \
--cc=andriin@fb.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dxu@dxuuu.xyz \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=kafai@fb.com \
--cc=kpsingh@chromium.org \
--cc=netdev@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=songliubraving@fb.com \
--cc=vmalik@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).