Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Tue, Mar 9, 2021 at 9:58 PM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
>
>
> On 5 Mar 2021, at 03:32, Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote:
>
>
> Specially the attach_kprobe_legacy() function:
>
> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
>
> I wanted to reply here in case others also face this.
>
>
> Great, glad it worked out. It would be great if you could contribute
> legacy kprobe support for libbpf as a proper patch, since it probably
> would be useful for a bunch of other people stuck with old kernels.
>
>
>
> I’m sorry to come back to this but I’d like to clarify something, if you allow me.
>
> If I recompile old kernels (4.x.y) with the “scripts/link-vmlinux.sh" patch (setting $btf_vmlinux_bin_o and gen_btf()) I’m able to use "pahole -J" to generate the .BTF ELF section from a vmlinux file (out of the debugging package, for example) using its DWARF data.
>
> Using objcopy, I’m also able to extract only the .BTF ELF section from it and use the generated file (smaller) as a base BTF file for libbpf (since old kernels don’t have /sys/kernel/btf/vmlinux interface).
>
> So, in my case, with this, I can get an ~30MB ELF file (from a an almost 600MB vmlinux) with BTF data that can feed libbpf to do needed relocations for my BPF object. Execution works perfectly and I can have the same libbpf based code to run in a 4.15 and a 5.8 kernel, smooth.

Surprised that .BTF is so big at 30MB. It depends on kernel config you
are using, but that's still few times bigger than what I normally see.

Otherwise, yeah, that's how it should work (except see the patch that
adds core_btf_path and discussion around it).

>
> What is not entirely clear to me yet is … why can’t I use a “vmlinux” file from a previous compiled kernel (that has not been compiled with a changed link-vmlinux.sh file) and do the same: generate the BTF section from its DWARF data with pahole and use generated file (or the BTF section extract only) as input to libbpf.
>
> I mean, I can do, but it does not work… Assumption: it only works for the ones I build with patched link-vmlinux.sh (not the ones already built and provided as packages). The code execution output (debug=1 on libbpf) is at : https://pastebin.ubuntu.com/p/bx6tygY8p2/
>

From what I see all the CO-RE relocations applied successfully (even
though all the offsets stayed the same, so presumably you compiled
your BPF program with vmlinux.h from the exact same kernel you are
running it on?). Are you sure that vmlinux image you are providing
corresponds to the actual kernel you are running on?

I'd start by comparing libbpf logs for vmlinux you get with modified
link-vmlinux.sh script and with just explicit pahole -J. If all the
CO-RE parts are identical, the problem is somewhere else most
probably.

I see "libbpf: load bpf program failed: Invalid argument" in that log,
which means that CO-RE was done and successful and only when trying to
load your BPF program into the kernel it failed.


> The difference for a new 4.x.y kernel and the existing ones (older packaged kernels) is the vmlinux_link() function linking the BTF object file in each of the 3 tmp_kallsyms steps.
>
> Is there a way I can get the already existing kernels to work with only pahole DWARF to BTF conversion data ?

Yes and you've found it, I think. There is no difference to libbpf and
.BTF itself whether it's run in link-vmlinux.sh or with explicit
pahole -J. Look for the problem somewhere else.

>
> Thank you!
>
> -rafaeldtinoco
>