Re: [PATCHv2 RFC bpf-next 0/7] bpf: Add support for ftrace probe

[Steven Rostedt <rostedt@goodmis.org>]

On Tue, 20 Apr 2021 08:33:43 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:


> I don't see how you can do it without BTF.

I agree. 

> The mass-attach feature should prepare generic 6 or so arguments
> from all functions it attached to.
> On x86-64 it's trivial because 6 regs are the same.
> On arm64 is now more challenging since return value regs overlaps with
> first argument, so bpf trampoline (when it's ready for arm64) will look
> a bit different than bpf trampoline on x86-64 to preserve arg0, arg1,
> ..arg6, ret
> 64-bit values that bpf prog expects to see.
> On x86-32 it's even more trickier, since the same 6 args need to be copied
> from a combination of regs and stack.
> This is not some hypothetical case. We already use BTF in x86-32 JIT
> and btf_func_model was introduced specifically to handle such cases.
> So I really don't see how ftrace can do that just yet. It has to understand BTF

ftrace doesn't need to understand BTF, but the call back does. Ftrace will
give the callback all the information it needs to get the arguments from
the regs that hold the arguments and a pointer to the stack.

It's a limited set of regs that produce the arguments. ftrace only needs to
supply it. How those regs are converted to arguments requires more
understanding of those arguments, which BTF can give you.


> of all of the funcs it attaches to otherwise it's just saving all regs.
> That approach was a pain to deal with.
> Just look at bpf code samples with ugly per architecture macros to access regs.
> BPF trampoline solved it and I don't think going back to per-arch macros
> is an option at this point.

How does it solve it besides a one to one mapped trampoline to function?
Once you attach more than one function to the trampoline, it needs BTF to
translate it, and you need to save enough regs to have access to the args
that are needed.

For a direct call, which attaches to only one function, you could do short
cuts. If the function you hook to, only has one argument, you may be able
to safely call the callback code and only save that one argument. But as
soon as you want to attach to more than one function to the same
trampoline, you will need to save the regs of the args with the most
arguments.

What ftrace can give you is what I have called "ftrace_regs", which is
really just pt_regs with only the arguments saved. This is much less than
the ftrace_regs_caller that is used by kprobes. Because kprobes expects to
be called by an int3. ftrace_regs_caller() is much faster than an int3, but
still needs to save all regs that an interrupt would save (including
flags!) and makes it slower than a normal ftrace_caller() (and why I
created the two). But the ftrace_caller() saves only the regs needed to
create arguments (as ftrace must handle all functions and all their args,
so that the callback does not corrupt them).

For x86_64, this is rdi, rsi, rdx, rcx, r8, r9. That's all that is saved.
And this is passed to the callback as well as the stack pointer. With this
information, you could get any arguments of any function on x86_64. And it
is trivial to do the same for other architectures.

For every function attached to ftrace, you would have a BPF program created
from the BTF of the function prototype to parse the arguments. The
ftrace_regs holds only the information needed to retrieve those arguments,
Have a hash that maps the traced function with its bpf program and then
simply execute that, feeding it the ftrace_regs. Then the bpf program could
parse out the arguments.

That is, you could have BPF hook to the ftrace callback (and it lets you
hook to a subset of functions), have a hash table that maps the function to
the BTF prototype of that function. Then you can have a generic BPF program
look up the BTF prototype of the function it is passed (via the hash), and
then retrieve the arguments.

The ftrace_regs could also be stored on the shadow stack to parse on the
return exit entry too (as Jiri wants to do).

Yes, the BPF trampoline is fine for a 1 to 1 mapping of a function to its
bpf program, but once you want to attach more than 1 function to a
trampoline, you'll need a way to parse its arguments for each function that
the trampoline is attached to. The difference is, you'll need a lookup table
to find the BTF of the function that is called.

I think this is doable.

-- Steve