Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco <rafaeldtinoco@ubuntu.com>]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco
> <rafaeldtinoco@ubuntu.com> wrote:
>>> From what I see all the CO-RE relocations applied successfully (even
>>> though all the offsets stayed the same, so presumably you compiled
>>> your BPF program with vmlinux.h from the exact same kernel you are
>>> running it on?). Are you sure that vmlinux image you are providing
>>> corresponds to the actual kernel you are running on?
>>
>> Yep, I have created the following:
>>
>> https://pastebin.ubuntu.com/p/h58YyNr4HR/
>
> Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux
> is some way (but then why would kernel start and only fail to
> load/verify your BPF program?). It's also clear that CO-RE is doing
> exactly the same instruction patching, so shouldn't be some invalid
> CO-RE relocation applied, I think. So no idea and not sure how to
> investigate this.
>
> But I think I'd never do `pahole -J` on actual vmlinux image you are
> going to run. It's much safer and more convenient to make a copy,
> generate .BTF and then extract just .BTF section into a small binary,
> which can be provided separately.
>

FOUND the cause of the issue...

Compiling the EXACT same kernel with different building scripts
(deb-pkg vs debian/rules in my case) resulted in a very similar
kernel (same .config, same autoconf.h, no visible changes).

Unfortunately one of the kernels worked fined (reading the BTF
extracted section OR same section within a vmlinux entire
file).. but the other did not.

Instrumenting this bad 4.15 kernel (out of debian/rules build)
I found that the following sanity checks took place in kernel:

bpf_check():

if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level  
|| !log->ubuf)

and a simple change in libbpf (mitigation of course):

   attr.log_buf = 0;
   attr.log_level = 0;
   attr.log_size = 0;

before

fd = sys_bpf_prog_load(&attr, sizeof(attr));

made the code to also run in this second kernel (built with the
debian/rules building scripts):

https://pastebin.ubuntu.com/p/scJDM3D9Zr/

Now I still have to discover why log_buf is miss-behaving in this
kernel being built with debian/rules* scripts and not with the
vanilla building scripts (despite config file being the same).

(FYIO, documenting it here for others also)...