Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs using the BPF_PROG_TEST_RUN API

[Masami Hiramatsu <mhiramat@kernel.org>]

On Tue, 5 May 2020 18:25:38 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Thu, Apr 30, 2020 at 7:44 PM Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > On Tue, 28 Apr 2020 09:19:47 -0300
> > Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >
> > > Em Tue, Apr 28, 2020 at 12:47:53PM +0200, Eelco Chaudron escreveu:
> > > > On 28 Apr 2020, at 6:04, Alexei Starovoitov wrote:
> > > > > On Fri, Apr 24, 2020 at 02:29:56PM +0200, Eelco Chaudron wrote:
> > >
> > > > > > > But in reality I think few kprobes in the prog will be enough to
> > > > > > > debug the program and XDP prog may still process millions of
> > > > > > > packets because your kprobe could be in error path and the user
> > > > > > > may want to capture only specific things when it triggers.
> > >
> > > > > > > kprobe bpf prog will execute in such case and it can capture
> > > > > > > necessary state from xdp prog, from packet or from maps that xdp
> > > > > > > prog is using.
> > >
> > > > > > > Some sort of bpf-gdb would be needed in user space.  Obviously
> > > > > > > people shouldn't be writing such kprob-bpf progs that debug
> > > > > > > other bpf progs by hand. bpf-gdb should be able to generate them
> > > > > > > automatically.
> > >
> > > > > > See my opening comment. What you're describing here is more when
> > > > > > the right developer has access to the specific system. But this
> > > > > > might not even be possible in some environments.
> > >
> > > > > All I'm saying that kprobe is a way to trace kernel.
> > > > > The same facility should be used to trace bpf progs.
> > >
> > > > perf doesn’t support tracing bpf programs, do you know of any tools that
> > > > can, or you have any examples that would do this?
> > >
> > > I'm discussing with Yonghong and Masami what would be needed for 'perf
> > > probe' to be able to add kprobes to BPF jitted areas in addition to
> > > vmlinux and modules.
> >
> > At a grance, at first we need a debuginfo which maps the source code and
> > BPF binaries. We also need to get a map from the kernel indicating
> > which instructions the bpf code was jited to.
> > Are there any such information?
> 
> it's already there. Try 'bpftool prog dump jited id N'
> It will show something like this:
> ; data = ({typeof(errors.leaf) *leaf =
> bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &type_key); if (!leaf) {
> bpf_map_update_elem_(bpf_pseudo_fd(1, -11), &type_key, &zero,
> BPF_NOEXIST); leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &t;
>  81d:    movabs $0xffff8881a0679000,%rdi
> ; return bpf_map_lookup_elem((void *)map, key);
>  827:    mov    %rbx,%rsi
>  82a:    callq  0xffffffffe0f7f448
>  82f:    test   %rax,%rax
>  832:    je     0x0000000000000838
>  834:    add    $0x40,%rax
> ; if (!data)
>  838:    test   %rax,%rax
>  83b:    je     0x0000000000000846
>  83d:    mov    $0x1,%edi
> ; lock_xadd(data, 1);
>  842:    lock add %edi,0x0(%rax)

Hm, so bpftool or libbpf has some source-address (offset) mapping
APIs, that will help for me.

BTW, I would like to confirm that if the kernel has jited code,
it will not fall back to xlated code. Is it correct?

> > Also, I would like to know the target BPF (XDP) is running in kprobes
> > context or not. BPF tracer sometimes use the kprobes to hook the event
> > and run in the kprobe (INT3) context. That will be need more work to
> > probe it.
> > For the BPF code which just runs in tracepoint context, it will be easy
> > to probe it. (we may need to break a limitation of notrace, which we
> > already has a kconfig)
> 
> yeah. this mechanism won't be able to debug bpf progs that are
> attached to kprobes via int3. But that is rare case.
> Most kprobe+bpf are for function entry and adding int3 to jited bpf code
> will work just like for normal kernel functions.

OK, anyway, I've done the nested kprobe support on x86/arm/arm64 :)
That will make it easy.

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>