Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
>
> > From what I see all the CO-RE relocations applied successfully (even
> > though all the offsets stayed the same, so presumably you compiled
> > your BPF program with vmlinux.h from the exact same kernel you are
> > running it on?). Are you sure that vmlinux image you are providing
> > corresponds to the actual kernel you are running on?
>
> Yep, I have created the following:
>
> https://pastebin.ubuntu.com/p/h58YyNr4HR/
>

Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux
is some way (but then why would kernel start and only fail to
load/verify your BPF program?). It's also clear that CO-RE is doing
exactly the same instruction patching, so shouldn't be some invalid
CO-RE relocation applied, I think. So no idea and not sure how to
investigate this.

But I think I'd never do `pahole -J` on actual vmlinux image you are
going to run. It's much safer and more convenient to make a copy,
generate .BTF and then extract just .BTF section into a small binary,
which can be provided separately.


> to make this easier.
>
> It is a 4.15.0-1080 kernel and a 4.15.18+. They are pretty close

Also this. Seems like it is two different kernels still, however small
the difference between them is. Have you tried building *exactly* the
same kernel with exactly the same config, but just with pahole -J
during the build on vmlinux.o (which is later linked into a final
vmlinux) vs running pahole -J on final vmlinux after the build. Notice
that two approach differ in terms of which object file is being
modified. With link-vmlinux.sh, kernel linker scripts dictate final
layout, while if you run `pahole -J` on final vmlinux, it can modify
the layout and potentially break something sensitive. pahole is using
llvm-objcopy internally, and llvm-objcopy isn't a very trivial tool.


> despite the versioning (second was generated with make deb-dpkg).
>
> I’m using same .config file for both, only difference is that the
> 4.15.18+ was compiled with the changed link-vmlinux.sh file.
>
> The /usr/lib/debug/boot/vmlinux files are generated by the same
> build and I have tried 2 or 3 of the existing packaged kernels. The
> only thing I did was “pahole -J” to /usr/lib/debug/boot/vmlinux-XXX
> files (adding the BTF section to them).
>
> Running same binary in a 5.8.0-43 kernel works out-of-the-box:
>
> https://pastebin.ubuntu.com/p/VKTjMcp6Xs/
>
> >
> > I'd start by comparing libbpf logs for vmlinux you get with modified
> > link-vmlinux.sh script and with just explicit pahole -J. If all the
> > CO-RE parts are identical, the problem is somewhere else most
> > probably.
>
> The difference between (1) and (2) from the paste (error and success):
>
> libbpf: CO-RE relocating [0] struct task_struct: found target
> candidate [17361] struct task_struct in [vmlinux]
> libbpf: CO-RE relocating [0] struct task_struct: found target
> candidate [17360] struct task_struct in [vmlinux]
>
> libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17361]
> struct task_struct.comm (0:90 @ offset 2640)
> libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17360]
> struct task_struct.comm (0:90 @ offset 2640)
>
> Code is at:
>
> https://github.com/rafaeldtinoco/portablebpf
>
> and it is not much different than any other libbpf example.
>
> thanks again for verifying this!
>
> -rafaeldtinoco
>
>
>