Measuring performance of BPF programs

Measuring performance of BPF programs

[Matheus Salgueiro Castanho]

Hi XDP-Newbies,

I've been working with BPF programs on the XDP and TC hooks and wanted
to understand their overhead on the overall packet processing stack.
Mainly I'd like to know how many times those programs have run and
also how long each execution took.

I know that BPF programs can also be used for tracing inside the
kernel, so I looked up into bpftrace [1] to help me with this task, so
I could start monitoring my programs faster.

After browsing through the list of available trace hooks (bpftrace -l)
I found a series of kprobes for xdp- and bpf-related functions. At
first, however, none of them seem to map directly to an individual
program execution.

For the TC case, I found the kprobe for cls_bpf_classify(), which
seems to be the function that actually runs the BPF program on TC and
handles the output. Although it has extra processing compared to the
BPF program per se[[, this seemed close enough.

However, I didn't find a similar kprobe for the XDP case. I imagine
this is so because XDP native runs inside the NIC driver, right?

Is there a kprobe that maps to a single XDP program execution, or at
least close enough? Or is there a better way to measure and trace TC
and XDP programs?

Thanks in advance,
Matheus Castanho

[1] https://github.com/iovisor/bpftrace

Re: Measuring performance of BPF programs

[Sebastiano Miano]

Hello Matheus,
starting from kernel 5.1 it is possible to get some statistics of a
specific eBPF program running in the kernel, thanks to this [1] patch.
You can enable it by setting 'kernel.bpf_stats_enabled=1' and show the
result using bpftool [2] ('bpftool prof show id <bpf_prog_id>).
This will print the total time spent on a given eBPF program and the
total number of runs.

If you want to get more information on a running eBPF program you
could use 'perf report + report' as any other kernel function.
In this case, you should see a symbol like
'bpf_prog_<prog_tag>_<func_name>' that refers to a single eBPF
program.

Hope this helps.

[1] https://patchwork.ozlabs.org/patch/1047415/
[2] https://github.com/torvalds/linux/tree/master/tools/bpf/bpftool

Cheers,
--
Sebastiano Miano,
Politecnico di Torino (DAUIN),
https://sebymiano.github.io/


On Tue, Jun 11, 2019 at 2:37 PM Matheus Salgueiro Castanho
<matheus.castanho@dcc.ufmg.br> wrote:
>
> Hi XDP-Newbies,
>
> I've been working with BPF programs on the XDP and TC hooks and wanted
> to understand their overhead on the overall packet processing stack.
> Mainly I'd like to know how many times those programs have run and
> also how long each execution took.
>
> I know that BPF programs can also be used for tracing inside the
> kernel, so I looked up into bpftrace [1] to help me with this task, so
> I could start monitoring my programs faster.
>
> After browsing through the list of available trace hooks (bpftrace -l)
> I found a series of kprobes for xdp- and bpf-related functions. At
> first, however, none of them seem to map directly to an individual
> program execution.
>
> For the TC case, I found the kprobe for cls_bpf_classify(), which
> seems to be the function that actually runs the BPF program on TC and
> handles the output. Although it has extra processing compared to the
> BPF program per se[[, this seemed close enough.
>
> However, I didn't find a similar kprobe for the XDP case. I imagine
> this is so because XDP native runs inside the NIC driver, right?
>
> Is there a kprobe that maps to a single XDP program execution, or at
> least close enough? Or is there a better way to measure and trace TC
> and XDP programs?
>
> Thanks in advance,
> Matheus Castanho
>
> [1] https://github.com/iovisor/bpftrace

Re: Measuring performance of BPF programs

[Matheus Salgueiro Castanho]

Hi Sebastiano,

Thanks for the tip. Right now I'm using kernel 5.0, so I tried the
second option with 'perf record' + 'perf report', and it gave me
exactly what I needed.

After installing the kernel symbols I'm able to see all function
calls, including calls to cls_bpf_classify() and
bpf_xdp_adjust_head(), but I don't see any symbols with the format you
said.

Am I missing some symbol packages?

Thanks,
Matheus Castanho

On Tue, Jun 11, 2019 at 11:43 AM Sebastiano Miano
<sebastiano.miano@polito.it> wrote:
>
> Hello Matheus,
> starting from kernel 5.1 it is possible to get some statistics of a
> specific eBPF program running in the kernel, thanks to this [1] patch.
> You can enable it by setting 'kernel.bpf_stats_enabled=1' and show the
> result using bpftool [2] ('bpftool prof show id <bpf_prog_id>).
> This will print the total time spent on a given eBPF program and the
> total number of runs.
>
> If you want to get more information on a running eBPF program you
> could use 'perf report + report' as any other kernel function.
> In this case, you should see a symbol like
> 'bpf_prog_<prog_tag>_<func_name>' that refers to a single eBPF
> program.
>
> Hope this helps.
>
> [1] https://patchwork.ozlabs.org/patch/1047415/
> [2] https://github.com/torvalds/linux/tree/master/tools/bpf/bpftool
>
> Cheers,
> --
> Sebastiano Miano,
> Politecnico di Torino (DAUIN),
> https://sebymiano.github.io/
>
>
> On Tue, Jun 11, 2019 at 2:37 PM Matheus Salgueiro Castanho
> <matheus.castanho@dcc.ufmg.br> wrote:
> >
> > Hi XDP-Newbies,
> >
> > I've been working with BPF programs on the XDP and TC hooks and wanted
> > to understand their overhead on the overall packet processing stack.
> > Mainly I'd like to know how many times those programs have run and
> > also how long each execution took.
> >
> > I know that BPF programs can also be used for tracing inside the
> > kernel, so I looked up into bpftrace [1] to help me with this task, so
> > I could start monitoring my programs faster.
> >
> > After browsing through the list of available trace hooks (bpftrace -l)
> > I found a series of kprobes for xdp- and bpf-related functions. At
> > first, however, none of them seem to map directly to an individual
> > program execution.
> >
> > For the TC case, I found the kprobe for cls_bpf_classify(), which
> > seems to be the function that actually runs the BPF program on TC and
> > handles the output. Although it has extra processing compared to the
> > BPF program per se[[, this seemed close enough.
> >
> > However, I didn't find a similar kprobe for the XDP case. I imagine
> > this is so because XDP native runs inside the NIC driver, right?
> >
> > Is there a kprobe that maps to a single XDP program execution, or at
> > least close enough? Or is there a better way to measure and trace TC
> > and XDP programs?
> >
> > Thanks in advance,
> > Matheus Castanho
> >
> > [1] https://github.com/iovisor/bpftrace

Re: Measuring performance of BPF programs

[Sebastiano Miano]

On Thu, Jun 13, 2019 at 1:56 PM Matheus Salgueiro Castanho
<matheus.castanho@dcc.ufmg.br> wrote:
>
> Hi Sebastiano,
>
> Thanks for the tip. Right now I'm using kernel 5.0, so I tried the
> second option with 'perf record' + 'perf report', and it gave me
> exactly what I needed.
>
> After installing the kernel symbols I'm able to see all function
> calls, including calls to cls_bpf_classify() and
> bpf_xdp_adjust_head(), but I don't see any symbols with the format you
> said.
>
> Am I missing some symbol packages?

Could you please try with a newer kernel (e.g., v5.1)? I guess this
has been added recently, although I am not able to find the
corresponding patch.
Moreover, make sure you first run 'perf record -ag' and then load the
corresponding eBPF program.

Hope this helps.

Cheers,
Sebastiano

>
> Thanks,
> Matheus Castanho