Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty <vamsi@araalinetworks.com>]

First of all thank you very much for your quick response. And helpful pointers.
It seems like you also think what I am attempting to do should work.

Please see inline [VAMSI-2].

On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> <vamsi@araalinetworks.com> wrote:
> >
> > Had a few questions on CO-RE dependencies and usage. From what I read
> > CO-RE needs a supported kernel version and be compiled with
> > `CONFIG_DEBUG_INFO_BTF=y`.
> >
> > I also understand there are three pieces to enable CO-RE
> > functionality. (1) The BTF format. For efficient/compressed kernel
> > symbol table. (2) clang changes to emit the BTF relocations. (3)
>
> BTF is not really a symbol table, rather a type information. Like
> simpler and more compact DWARF.
>
> > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > these 3 steps are done the resulting byte code is no different from
> > non-CO-RE byte code.
> >
> > Given this I am hoping the knowledgeable folks on this mailer correct
> > and guide me if I am stating something incorrectly.
> >
> > (1) Is the kernel support requirement ONLY for the purposes of
> > generating and exposing the BTF file information on
> > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > `libbpf` can find the BTF information at a standard location?.
>
> /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> try to search for vmlinux image (and BTF info within it) in a few
> standard locations, see [0]. Early versions of in-kernel BTF didn't
> even expose /sys/kernel/btf/vmlinux.
>
>   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
>
> >
> > (2) If the answer to the above question is YES. Could the below
> > mechanism be used so that it works on all kernels whether they support
> > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> >        (a) Extract BTF generation process outside of the kernel build.
> > Use this to generate the equivalent BTF file for it.
>
> Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> info, but it's also possible to just embed BTF manually with a direct
> invocation of pahole -J, see [1] on how it's done for
> CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> matter the version, and it will work with CO-RE relocations.
>
>   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
>

[VAMSI-2] Yes, this is exactly what I did. I extracted out the
`gen_btf` from the
`link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
file for the
4.14.0 kernel.

> >        (b) Make changes to `libbpf` to look for BTF not only at the
> > standard locations but also at a user specified location. The BTF file
> > generated in (a) can be presented here.
>
> You can already do that, actually, though it's not very obvious. You
> can specify (or override) kernel BTF location by using
> bpf_object__load_xattr() and passing target_btf_path pointing to your
> BTF location (see [2]). I've been meaning to add it instead to a
> bpf_object_open_opts, btw, to make its use possible with a BPF
> skeleton. Also keep in mind that currently libbpf expects that custom
> BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> we can improve that, of course.
>
>   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141

[VAMSI-2] I took a look at this and what you suggested above does not
work as is.
Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
`bpf_object__load_vmlinux_btf` is not yet modified to use the
`target_btf_path` attribute.
Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
you suggested enabling
use from the BPF skeleton seems useful and I can possibly help with that.

For now, just for proof of concept I modified the search options in
`libbpf_find_kernel_btf` to
include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.

libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: kernel doesn't support global data
libbpf: failed to load object 'tcpconnect_bpf'
libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
failed to load BPF object: -95

This is the reason I had posted on the mailer. If the CO-RE executable
has relocations
resolved by the time of the BPF load. Why do we need to check for
kernel support?. Also,
does this mean what I am attempting to do will not work?.

Best Regards. And again thanks a lot for your precious time.
- Vamsi.

> >
> > This should provide us a way to enable CO-RE functionality on older
> > kernel versions as well. I tried to make the above changes and tried
> > against a 4.14 kernel and it did not work. Either I am not doing
> > something right or my assumptions are wrong.
> >
> > Thanks in advance for your time. And I hope someone here can guide me
> > in the right direction.
> >
> > Regards
> > Vamsi.