[BPF CO-RE clarification] Use CO-RE on older kernel versions.

[BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty]

Had a few questions on CO-RE dependencies and usage. From what I read
CO-RE needs a supported kernel version and be compiled with
`CONFIG_DEBUG_INFO_BTF=y`.

I also understand there are three pieces to enable CO-RE
functionality. (1) The BTF format. For efficient/compressed kernel
symbol table. (2) clang changes to emit the BTF relocations. (3)
`libbpf` changes to locate a BTF file and fix-up relocations. Once
these 3 steps are done the resulting byte code is no different from
non-CO-RE byte code.

Given this I am hoping the knowledgeable folks on this mailer correct
and guide me if I am stating something incorrectly.

(1) Is the kernel support requirement ONLY for the purposes of
generating and exposing the BTF file information on
`/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
`libbpf` can find the BTF information at a standard location?.

(2) If the answer to the above question is YES. Could the below
mechanism be used so that it works on all kernels whether they support
the `CONFIG_DEBUG_INFO_BTF` flag or not?.
       (a) Extract BTF generation process outside of the kernel build.
Use this to generate the equivalent BTF file for it.
       (b) Make changes to `libbpf` to look for BTF not only at the
standard locations but also at a user specified location. The BTF file
generated in (a) can be presented here.

This should provide us a way to enable CO-RE functionality on older
kernel versions as well. I tried to make the above changes and tried
against a 4.14 kernel and it did not work. Either I am not doing
something right or my assumptions are wrong.

Thanks in advance for your time. And I hope someone here can guide me
in the right direction.

Regards
Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
<vamsi@araalinetworks.com> wrote:
>
> Had a few questions on CO-RE dependencies and usage. From what I read
> CO-RE needs a supported kernel version and be compiled with
> `CONFIG_DEBUG_INFO_BTF=y`.
>
> I also understand there are three pieces to enable CO-RE
> functionality. (1) The BTF format. For efficient/compressed kernel
> symbol table. (2) clang changes to emit the BTF relocations. (3)

BTF is not really a symbol table, rather a type information. Like
simpler and more compact DWARF.

> `libbpf` changes to locate a BTF file and fix-up relocations. Once
> these 3 steps are done the resulting byte code is no different from
> non-CO-RE byte code.
>
> Given this I am hoping the knowledgeable folks on this mailer correct
> and guide me if I am stating something incorrectly.
>
> (1) Is the kernel support requirement ONLY for the purposes of
> generating and exposing the BTF file information on
> `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> `libbpf` can find the BTF information at a standard location?.

/sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
try to search for vmlinux image (and BTF info within it) in a few
standard locations, see [0]. Early versions of in-kernel BTF didn't
even expose /sys/kernel/btf/vmlinux.

  [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580

>
> (2) If the answer to the above question is YES. Could the below
> mechanism be used so that it works on all kernels whether they support
> the `CONFIG_DEBUG_INFO_BTF` flag or not?.
>        (a) Extract BTF generation process outside of the kernel build.
> Use this to generate the equivalent BTF file for it.

Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
info, but it's also possible to just embed BTF manually with a direct
invocation of pahole -J, see [1] on how it's done for
CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
matter the version, and it will work with CO-RE relocations.

  [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170

>        (b) Make changes to `libbpf` to look for BTF not only at the
> standard locations but also at a user specified location. The BTF file
> generated in (a) can be presented here.

You can already do that, actually, though it's not very obvious. You
can specify (or override) kernel BTF location by using
bpf_object__load_xattr() and passing target_btf_path pointing to your
BTF location (see [2]). I've been meaning to add it instead to a
bpf_object_open_opts, btw, to make its use possible with a BPF
skeleton. Also keep in mind that currently libbpf expects that custom
BTF to be an ELF file with .BTF section, not just a raw BTF data. But
we can improve that, of course.

  [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
>
> This should provide us a way to enable CO-RE functionality on older
> kernel versions as well. I tried to make the above changes and tried
> against a 4.14 kernel and it did not work. Either I am not doing
> something right or my assumptions are wrong.
>
> Thanks in advance for your time. And I hope someone here can guide me
> in the right direction.
>
> Regards
> Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty]

First of all thank you very much for your quick response. And helpful pointers.
It seems like you also think what I am attempting to do should work.

Please see inline [VAMSI-2].

On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> <vamsi@araalinetworks.com> wrote:
> >
> > Had a few questions on CO-RE dependencies and usage. From what I read
> > CO-RE needs a supported kernel version and be compiled with
> > `CONFIG_DEBUG_INFO_BTF=y`.
> >
> > I also understand there are three pieces to enable CO-RE
> > functionality. (1) The BTF format. For efficient/compressed kernel
> > symbol table. (2) clang changes to emit the BTF relocations. (3)
>
> BTF is not really a symbol table, rather a type information. Like
> simpler and more compact DWARF.
>
> > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > these 3 steps are done the resulting byte code is no different from
> > non-CO-RE byte code.
> >
> > Given this I am hoping the knowledgeable folks on this mailer correct
> > and guide me if I am stating something incorrectly.
> >
> > (1) Is the kernel support requirement ONLY for the purposes of
> > generating and exposing the BTF file information on
> > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > `libbpf` can find the BTF information at a standard location?.
>
> /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> try to search for vmlinux image (and BTF info within it) in a few
> standard locations, see [0]. Early versions of in-kernel BTF didn't
> even expose /sys/kernel/btf/vmlinux.
>
>   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
>
> >
> > (2) If the answer to the above question is YES. Could the below
> > mechanism be used so that it works on all kernels whether they support
> > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> >        (a) Extract BTF generation process outside of the kernel build.
> > Use this to generate the equivalent BTF file for it.
>
> Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> info, but it's also possible to just embed BTF manually with a direct
> invocation of pahole -J, see [1] on how it's done for
> CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> matter the version, and it will work with CO-RE relocations.
>
>   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
>

[VAMSI-2] Yes, this is exactly what I did. I extracted out the
`gen_btf` from the
`link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
file for the
4.14.0 kernel.

> >        (b) Make changes to `libbpf` to look for BTF not only at the
> > standard locations but also at a user specified location. The BTF file
> > generated in (a) can be presented here.
>
> You can already do that, actually, though it's not very obvious. You
> can specify (or override) kernel BTF location by using
> bpf_object__load_xattr() and passing target_btf_path pointing to your
> BTF location (see [2]). I've been meaning to add it instead to a
> bpf_object_open_opts, btw, to make its use possible with a BPF
> skeleton. Also keep in mind that currently libbpf expects that custom
> BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> we can improve that, of course.
>
>   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141

[VAMSI-2] I took a look at this and what you suggested above does not
work as is.
Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
`bpf_object__load_vmlinux_btf` is not yet modified to use the
`target_btf_path` attribute.
Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
you suggested enabling
use from the BPF skeleton seems useful and I can possibly help with that.

For now, just for proof of concept I modified the search options in
`libbpf_find_kernel_btf` to
include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.

libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
libbpf: Kernel doesn't support BTF, skipping uploading it.
libbpf: kernel doesn't support global data
libbpf: failed to load object 'tcpconnect_bpf'
libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
failed to load BPF object: -95

This is the reason I had posted on the mailer. If the CO-RE executable
has relocations
resolved by the time of the BPF load. Why do we need to check for
kernel support?. Also,
does this mean what I am attempting to do will not work?.

Best Regards. And again thanks a lot for your precious time.
- Vamsi.

> >
> > This should provide us a way to enable CO-RE functionality on older
> > kernel versions as well. I tried to make the above changes and tried
> > against a 4.14 kernel and it did not work. Either I am not doing
> > something right or my assumptions are wrong.
> >
> > Thanks in advance for your time. And I hope someone here can guide me
> > in the right direction.
> >
> > Regards
> > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty
<vamsi@araalinetworks.com> wrote:
>
> First of all thank you very much for your quick response. And helpful pointers.
> It seems like you also think what I am attempting to do should work.
>
> Please see inline [VAMSI-2].
>
> On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> > <vamsi@araalinetworks.com> wrote:
> > >
> > > Had a few questions on CO-RE dependencies and usage. From what I read
> > > CO-RE needs a supported kernel version and be compiled with
> > > `CONFIG_DEBUG_INFO_BTF=y`.
> > >
> > > I also understand there are three pieces to enable CO-RE
> > > functionality. (1) The BTF format. For efficient/compressed kernel
> > > symbol table. (2) clang changes to emit the BTF relocations. (3)
> >
> > BTF is not really a symbol table, rather a type information. Like
> > simpler and more compact DWARF.
> >
> > > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > > these 3 steps are done the resulting byte code is no different from
> > > non-CO-RE byte code.
> > >
> > > Given this I am hoping the knowledgeable folks on this mailer correct
> > > and guide me if I am stating something incorrectly.
> > >
> > > (1) Is the kernel support requirement ONLY for the purposes of
> > > generating and exposing the BTF file information on
> > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > > `libbpf` can find the BTF information at a standard location?.
> >
> > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> > try to search for vmlinux image (and BTF info within it) in a few
> > standard locations, see [0]. Early versions of in-kernel BTF didn't
> > even expose /sys/kernel/btf/vmlinux.
> >
> >   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
> >
> > >
> > > (2) If the answer to the above question is YES. Could the below
> > > mechanism be used so that it works on all kernels whether they support
> > > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> > >        (a) Extract BTF generation process outside of the kernel build.
> > > Use this to generate the equivalent BTF file for it.
> >
> > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> > info, but it's also possible to just embed BTF manually with a direct
> > invocation of pahole -J, see [1] on how it's done for
> > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> > matter the version, and it will work with CO-RE relocations.
> >
> >   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
> >
>
> [VAMSI-2] Yes, this is exactly what I did. I extracted out the
> `gen_btf` from the
> `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
> file for the
> 4.14.0 kernel.
>
> > >        (b) Make changes to `libbpf` to look for BTF not only at the
> > > standard locations but also at a user specified location. The BTF file
> > > generated in (a) can be presented here.
> >
> > You can already do that, actually, though it's not very obvious. You
> > can specify (or override) kernel BTF location by using
> > bpf_object__load_xattr() and passing target_btf_path pointing to your
> > BTF location (see [2]). I've been meaning to add it instead to a
> > bpf_object_open_opts, btw, to make its use possible with a BPF
> > skeleton. Also keep in mind that currently libbpf expects that custom
> > BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> > we can improve that, of course.
> >
> >   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
>
> [VAMSI-2] I took a look at this and what you suggested above does not
> work as is.
> Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
> `bpf_object__load_vmlinux_btf` is not yet modified to use the
> `target_btf_path` attribute.

Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but
since then added a bunch more use cases. So some libbpf changes are
needed to make this work. But it should still work for CO-RE to have a
custom BTF.

I'm not sure about making bpf_object__load_vmlinux_btf() load custom
BTF as the real kernel BTF, because that will never work for
fentry/fexit, struct_ops, etc. I think it is better to teach
bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF
if we need it only for CO-RE relocations *and* we have it overloaded
with target_btf_path

> Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
> you suggested enabling
> use from the BPF skeleton seems useful and I can possibly help with that.

Yeah, adding something like core_btf_path option to
bpf_object_open_opts would go nicely with this change.

>
> For now, just for proof of concept I modified the search options in
> `libbpf_find_kernel_btf` to
> include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.
>
> libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0

so here you successfully loaded custom BTF, which is good.

> libbpf: Kernel doesn't support BTF, skipping uploading it.

this just means that your BPF object's BTF won't be loaded into the
kernel. That's no big deal, ignore this.

> libbpf: kernel doesn't support global data

But this means that your BPF programs rely on global variables, which
are not supported by the kernel. So you need to change the code to not
use global variables to make this work on very old kernels.


> libbpf: failed to load object 'tcpconnect_bpf'
> libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
> failed to load BPF object: -95

This is probably OPNOTSUPP from the global data above

>
> This is the reason I had posted on the mailer. If the CO-RE executable
> has relocations
> resolved by the time of the BPF load. Why do we need to check for
> kernel support?. Also,
> does this mean what I am attempting to do will not work?.
>

it will work with minimal libbpf logic changes. Nothing in principle
prevents this.


> Best Regards. And again thanks a lot for your precious time.
> - Vamsi.
>
> > >
> > > This should provide us a way to enable CO-RE functionality on older
> > > kernel versions as well. I tried to make the above changes and tried
> > > against a 4.14 kernel and it did not work. Either I am not doing
> > > something right or my assumptions are wrong.
> > >
> > > Thanks in advance for your time. And I hope someone here can guide me
> > > in the right direction.
> > >
> > > Regards
> > > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty]

Andrii,
    Thank you so much for the quick response. You were right. I was
trying to CO-RE the `tcpconnect.py` program. It had some `.rodata`
which I removed. Now the program goes much further. It now finishes
the relocations successfully. But, fails at the next step
`bpf_object__load_progs`.

===
libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50
(ALU/ALU64) imm 56 -> 56
libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No
such file or directory
libbpf: failed to determine kprobe perf type: No such file or directory
libbpf: prog 'tcp_v4_connect': failed to create kprobe
'tcp_v4_connect' perf event: No such file or directory
libbpf: failed to auto-attach program 'tcp_v4_connect': -2
===

This host has a 4.14 kernel AmazonLinux2 and does not have the above
file. It instead has this

===
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:kprobes/p_sys_execve_bcc_2566 sys_execve
===

I am guessing this is a backward compatibility issue?. I will try to
look at an earlier version of libbpf to see how this was handled.
Meanwhile, if you have further comments they are appreciated.

And again I can't thank you enough for how helpful you have been and
your time. Cheers!.




On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty
> <vamsi@araalinetworks.com> wrote:
> >
> > First of all thank you very much for your quick response. And helpful pointers.
> > It seems like you also think what I am attempting to do should work.
> >
> > Please see inline [VAMSI-2].
> >
> > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> > > <vamsi@araalinetworks.com> wrote:
> > > >
> > > > Had a few questions on CO-RE dependencies and usage. From what I read
> > > > CO-RE needs a supported kernel version and be compiled with
> > > > `CONFIG_DEBUG_INFO_BTF=y`.
> > > >
> > > > I also understand there are three pieces to enable CO-RE
> > > > functionality. (1) The BTF format. For efficient/compressed kernel
> > > > symbol table. (2) clang changes to emit the BTF relocations. (3)
> > >
> > > BTF is not really a symbol table, rather a type information. Like
> > > simpler and more compact DWARF.
> > >
> > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > > > these 3 steps are done the resulting byte code is no different from
> > > > non-CO-RE byte code.
> > > >
> > > > Given this I am hoping the knowledgeable folks on this mailer correct
> > > > and guide me if I am stating something incorrectly.
> > > >
> > > > (1) Is the kernel support requirement ONLY for the purposes of
> > > > generating and exposing the BTF file information on
> > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > > > `libbpf` can find the BTF information at a standard location?.
> > >
> > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> > > try to search for vmlinux image (and BTF info within it) in a few
> > > standard locations, see [0]. Early versions of in-kernel BTF didn't
> > > even expose /sys/kernel/btf/vmlinux.
> > >
> > >   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
> > >
> > > >
> > > > (2) If the answer to the above question is YES. Could the below
> > > > mechanism be used so that it works on all kernels whether they support
> > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> > > >        (a) Extract BTF generation process outside of the kernel build.
> > > > Use this to generate the equivalent BTF file for it.
> > >
> > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> > > info, but it's also possible to just embed BTF manually with a direct
> > > invocation of pahole -J, see [1] on how it's done for
> > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> > > matter the version, and it will work with CO-RE relocations.
> > >
> > >   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
> > >
> >
> > [VAMSI-2] Yes, this is exactly what I did. I extracted out the
> > `gen_btf` from the
> > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
> > file for the
> > 4.14.0 kernel.
> >
> > > >        (b) Make changes to `libbpf` to look for BTF not only at the
> > > > standard locations but also at a user specified location. The BTF file
> > > > generated in (a) can be presented here.
> > >
> > > You can already do that, actually, though it's not very obvious. You
> > > can specify (or override) kernel BTF location by using
> > > bpf_object__load_xattr() and passing target_btf_path pointing to your
> > > BTF location (see [2]). I've been meaning to add it instead to a
> > > bpf_object_open_opts, btw, to make its use possible with a BPF
> > > skeleton. Also keep in mind that currently libbpf expects that custom
> > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> > > we can improve that, of course.
> > >
> > >   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
> >
> > [VAMSI-2] I took a look at this and what you suggested above does not
> > work as is.
> > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
> > `bpf_object__load_vmlinux_btf` is not yet modified to use the
> > `target_btf_path` attribute.
>
> Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but
> since then added a bunch more use cases. So some libbpf changes are
> needed to make this work. But it should still work for CO-RE to have a
> custom BTF.
>
> I'm not sure about making bpf_object__load_vmlinux_btf() load custom
> BTF as the real kernel BTF, because that will never work for
> fentry/fexit, struct_ops, etc. I think it is better to teach
> bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF
> if we need it only for CO-RE relocations *and* we have it overloaded
> with target_btf_path
>
> > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
> > you suggested enabling
> > use from the BPF skeleton seems useful and I can possibly help with that.
>
> Yeah, adding something like core_btf_path option to
> bpf_object_open_opts would go nicely with this change.
>
> >
> > For now, just for proof of concept I modified the search options in
> > `libbpf_find_kernel_btf` to
> > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.
> >
> > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
>
> so here you successfully loaded custom BTF, which is good.
>
> > libbpf: Kernel doesn't support BTF, skipping uploading it.
>
> this just means that your BPF object's BTF won't be loaded into the
> kernel. That's no big deal, ignore this.
>
> > libbpf: kernel doesn't support global data
>
> But this means that your BPF programs rely on global variables, which
> are not supported by the kernel. So you need to change the code to not
> use global variables to make this work on very old kernels.
>
>
> > libbpf: failed to load object 'tcpconnect_bpf'
> > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
> > failed to load BPF object: -95
>
> This is probably OPNOTSUPP from the global data above
>
> >
> > This is the reason I had posted on the mailer. If the CO-RE executable
> > has relocations
> > resolved by the time of the BPF load. Why do we need to check for
> > kernel support?. Also,
> > does this mean what I am attempting to do will not work?.
> >
>
> it will work with minimal libbpf logic changes. Nothing in principle
> prevents this.
>
>
> > Best Regards. And again thanks a lot for your precious time.
> > - Vamsi.
> >
> > > >
> > > > This should provide us a way to enable CO-RE functionality on older
> > > > kernel versions as well. I tried to make the above changes and tried
> > > > against a 4.14 kernel and it did not work. Either I am not doing
> > > > something right or my assumptions are wrong.
> > > >
> > > > Thanks in advance for your time. And I hope someone here can guide me
> > > > in the right direction.
> > > >
> > > > Regards
> > > > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote:
>
> Andrii,
>     Thank you so much for the quick response. You were right. I was
> trying to CO-RE the `tcpconnect.py` program. It had some `.rodata`
> which I removed. Now the program goes much further. It now finishes
> the relocations successfully. But, fails at the next step
> `bpf_object__load_progs`.

It's discouraged to use top posting on the mailing list. So for the
future please reply inline.


>
> ===
> libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50
> (ALU/ALU64) imm 56 -> 56
> libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No
> such file or directory
> libbpf: failed to determine kprobe perf type: No such file or directory
> libbpf: prog 'tcp_v4_connect': failed to create kprobe
> 'tcp_v4_connect' perf event: No such file or directory
> libbpf: failed to auto-attach program 'tcp_v4_connect': -2
> ===
>
> This host has a 4.14 kernel AmazonLinux2 and does not have the above
> file. It instead has this
>
> ===
> $ sudo cat /sys/kernel/debug/tracing/kprobe_events
> p:kprobes/p_sys_execve_bcc_2566 sys_execve
> ===
>

Right. Libbpf only supports a newer and safer way to attach to
kprobes. For your experiments, try to stick to tracepoints and you'll
have a better time.

But it's another thing I've been meaning to add to libbpf for
supporting older kernels. I even have code written to do legacy kprobe
attachment, just need to find time to send a patch to add it as a
fallback for kernels that don't support new kprobe interface.


> I am guessing this is a backward compatibility issue?. I will try to
> look at an earlier version of libbpf to see how this was handled.

it was never supported by libbpf. BCC support both old and new APIs,
but old APIs is more dangerous, as it's easy to leave attached kprobe
BPF program active in the kernel unintentionally. E.g., if the process
crashes. But with bpf_link and its expected use of
bpf_link__destroy(), we can support the legacy API (it won't be any
safer, but still).

> Meanwhile, if you have further comments they are appreciated.
>
> And again I can't thank you enough for how helpful you have been and
> your time. Cheers!.
>
>
>
>
> On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty
> > <vamsi@araalinetworks.com> wrote:
> > >
> > > First of all thank you very much for your quick response. And helpful pointers.
> > > It seems like you also think what I am attempting to do should work.
> > >
> > > Please see inline [VAMSI-2].
> > >
> > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> > > > <vamsi@araalinetworks.com> wrote:
> > > > >
> > > > > Had a few questions on CO-RE dependencies and usage. From what I read
> > > > > CO-RE needs a supported kernel version and be compiled with
> > > > > `CONFIG_DEBUG_INFO_BTF=y`.
> > > > >
> > > > > I also understand there are three pieces to enable CO-RE
> > > > > functionality. (1) The BTF format. For efficient/compressed kernel
> > > > > symbol table. (2) clang changes to emit the BTF relocations. (3)
> > > >
> > > > BTF is not really a symbol table, rather a type information. Like
> > > > simpler and more compact DWARF.
> > > >
> > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > > > > these 3 steps are done the resulting byte code is no different from
> > > > > non-CO-RE byte code.
> > > > >
> > > > > Given this I am hoping the knowledgeable folks on this mailer correct
> > > > > and guide me if I am stating something incorrectly.
> > > > >
> > > > > (1) Is the kernel support requirement ONLY for the purposes of
> > > > > generating and exposing the BTF file information on
> > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > > > > `libbpf` can find the BTF information at a standard location?.
> > > >
> > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> > > > try to search for vmlinux image (and BTF info within it) in a few
> > > > standard locations, see [0]. Early versions of in-kernel BTF didn't
> > > > even expose /sys/kernel/btf/vmlinux.
> > > >
> > > >   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
> > > >
> > > > >
> > > > > (2) If the answer to the above question is YES. Could the below
> > > > > mechanism be used so that it works on all kernels whether they support
> > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> > > > >        (a) Extract BTF generation process outside of the kernel build.
> > > > > Use this to generate the equivalent BTF file for it.
> > > >
> > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> > > > info, but it's also possible to just embed BTF manually with a direct
> > > > invocation of pahole -J, see [1] on how it's done for
> > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> > > > matter the version, and it will work with CO-RE relocations.
> > > >
> > > >   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
> > > >
> > >
> > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the
> > > `gen_btf` from the
> > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
> > > file for the
> > > 4.14.0 kernel.
> > >
> > > > >        (b) Make changes to `libbpf` to look for BTF not only at the
> > > > > standard locations but also at a user specified location. The BTF file
> > > > > generated in (a) can be presented here.
> > > >
> > > > You can already do that, actually, though it's not very obvious. You
> > > > can specify (or override) kernel BTF location by using
> > > > bpf_object__load_xattr() and passing target_btf_path pointing to your
> > > > BTF location (see [2]). I've been meaning to add it instead to a
> > > > bpf_object_open_opts, btw, to make its use possible with a BPF
> > > > skeleton. Also keep in mind that currently libbpf expects that custom
> > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> > > > we can improve that, of course.
> > > >
> > > >   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
> > >
> > > [VAMSI-2] I took a look at this and what you suggested above does not
> > > work as is.
> > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
> > > `bpf_object__load_vmlinux_btf` is not yet modified to use the
> > > `target_btf_path` attribute.
> >
> > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but
> > since then added a bunch more use cases. So some libbpf changes are
> > needed to make this work. But it should still work for CO-RE to have a
> > custom BTF.
> >
> > I'm not sure about making bpf_object__load_vmlinux_btf() load custom
> > BTF as the real kernel BTF, because that will never work for
> > fentry/fexit, struct_ops, etc. I think it is better to teach
> > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF
> > if we need it only for CO-RE relocations *and* we have it overloaded
> > with target_btf_path
> >
> > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
> > > you suggested enabling
> > > use from the BPF skeleton seems useful and I can possibly help with that.
> >
> > Yeah, adding something like core_btf_path option to
> > bpf_object_open_opts would go nicely with this change.
> >
> > >
> > > For now, just for proof of concept I modified the search options in
> > > `libbpf_find_kernel_btf` to
> > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.
> > >
> > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
> >
> > so here you successfully loaded custom BTF, which is good.
> >
> > > libbpf: Kernel doesn't support BTF, skipping uploading it.
> >
> > this just means that your BPF object's BTF won't be loaded into the
> > kernel. That's no big deal, ignore this.
> >
> > > libbpf: kernel doesn't support global data
> >
> > But this means that your BPF programs rely on global variables, which
> > are not supported by the kernel. So you need to change the code to not
> > use global variables to make this work on very old kernels.
> >
> >
> > > libbpf: failed to load object 'tcpconnect_bpf'
> > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
> > > failed to load BPF object: -95
> >
> > This is probably OPNOTSUPP from the global data above
> >
> > >
> > > This is the reason I had posted on the mailer. If the CO-RE executable
> > > has relocations
> > > resolved by the time of the BPF load. Why do we need to check for
> > > kernel support?. Also,
> > > does this mean what I am attempting to do will not work?.
> > >
> >
> > it will work with minimal libbpf logic changes. Nothing in principle
> > prevents this.
> >
> >
> > > Best Regards. And again thanks a lot for your precious time.
> > > - Vamsi.
> > >
> > > > >
> > > > > This should provide us a way to enable CO-RE functionality on older
> > > > > kernel versions as well. I tried to make the above changes and tried
> > > > > against a 4.14 kernel and it did not work. Either I am not doing
> > > > > something right or my assumptions are wrong.
> > > > >
> > > > > Thanks in advance for your time. And I hope someone here can guide me
> > > > > in the right direction.
> > > > >
> > > > > Regards
> > > > > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty]

On Thu, Jan 7, 2021 at 3:32 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote:
> >
> > Andrii,
> >     Thank you so much for the quick response. You were right. I was
> > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata`
> > which I removed. Now the program goes much further. It now finishes
> > the relocations successfully. But, fails at the next step
> > `bpf_object__load_progs`.
>
> It's discouraged to use top posting on the mailing list. So for the
> future please reply inline.
>

[VAMSI-3] Apologies. Will be mindful next time on.

>
> >
> > ===
> > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50
> > (ALU/ALU64) imm 56 -> 56
> > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No
> > such file or directory
> > libbpf: failed to determine kprobe perf type: No such file or directory
> > libbpf: prog 'tcp_v4_connect': failed to create kprobe
> > 'tcp_v4_connect' perf event: No such file or directory
> > libbpf: failed to auto-attach program 'tcp_v4_connect': -2
> > ===
> >
> > This host has a 4.14 kernel AmazonLinux2 and does not have the above
> > file. It instead has this
> >
> > ===
> > $ sudo cat /sys/kernel/debug/tracing/kprobe_events
> > p:kprobes/p_sys_execve_bcc_2566 sys_execve
> > ===
> >
>
> Right. Libbpf only supports a newer and safer way to attach to
> kprobes. For your experiments, try to stick to tracepoints and you'll
> have a better time.
>
> But it's another thing I've been meaning to add to libbpf for
> supporting older kernels. I even have code written to do legacy kprobe
> attachment, just need to find time to send a patch to add it as a
> fallback for kernels that don't support new kprobe interface.
>

[VAMSI-3] I think that will be very helpful, as there are bound to be hosts
with older kernels for the foreseeable future. Good to know it's being
considered.

>
> > I am guessing this is a backward compatibility issue?. I will try to
> > look at an earlier version of libbpf to see how this was handled.
>
> it was never supported by libbpf. BCC support both old and new APIs,
> but old APIs is more dangerous, as it's easy to leave attached kprobe
> BPF program active in the kernel unintentionally. E.g., if the process
> crashes. But with bpf_link and its expected use of
> bpf_link__destroy(), we can support the legacy API (it won't be any
> safer, but still).
>

[VAMSI-3] Will use tracepoints to validate the process as you suggested. I
will then try to see what I can do to use this on kprobe/kretprobe's. Will also
check how BCC does this. I will start a new thread if needed.

Again appreciate the inputs and direction very much.

Best Regards
Vamsi.

> > Meanwhile, if you have further comments they are appreciated.
> >
> > And again I can't thank you enough for how helpful you have been and
> > your time. Cheers!.
> >
> >
> >
> >
> > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty
> > > <vamsi@araalinetworks.com> wrote:
> > > >
> > > > First of all thank you very much for your quick response. And helpful pointers.
> > > > It seems like you also think what I am attempting to do should work.
> > > >
> > > > Please see inline [VAMSI-2].
> > > >
> > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
> > > > <andrii.nakryiko@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> > > > > <vamsi@araalinetworks.com> wrote:
> > > > > >
> > > > > > Had a few questions on CO-RE dependencies and usage. From what I read
> > > > > > CO-RE needs a supported kernel version and be compiled with
> > > > > > `CONFIG_DEBUG_INFO_BTF=y`.
> > > > > >
> > > > > > I also understand there are three pieces to enable CO-RE
> > > > > > functionality. (1) The BTF format. For efficient/compressed kernel
> > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3)
> > > > >
> > > > > BTF is not really a symbol table, rather a type information. Like
> > > > > simpler and more compact DWARF.
> > > > >
> > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > > > > > these 3 steps are done the resulting byte code is no different from
> > > > > > non-CO-RE byte code.
> > > > > >
> > > > > > Given this I am hoping the knowledgeable folks on this mailer correct
> > > > > > and guide me if I am stating something incorrectly.
> > > > > >
> > > > > > (1) Is the kernel support requirement ONLY for the purposes of
> > > > > > generating and exposing the BTF file information on
> > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > > > > > `libbpf` can find the BTF information at a standard location?.
> > > > >
> > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> > > > > try to search for vmlinux image (and BTF info within it) in a few
> > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't
> > > > > even expose /sys/kernel/btf/vmlinux.
> > > > >
> > > > >   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
> > > > >
> > > > > >
> > > > > > (2) If the answer to the above question is YES. Could the below
> > > > > > mechanism be used so that it works on all kernels whether they support
> > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> > > > > >        (a) Extract BTF generation process outside of the kernel build.
> > > > > > Use this to generate the equivalent BTF file for it.
> > > > >
> > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> > > > > info, but it's also possible to just embed BTF manually with a direct
> > > > > invocation of pahole -J, see [1] on how it's done for
> > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> > > > > matter the version, and it will work with CO-RE relocations.
> > > > >
> > > > >   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
> > > > >
> > > >
> > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the
> > > > `gen_btf` from the
> > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
> > > > file for the
> > > > 4.14.0 kernel.
> > > >
> > > > > >        (b) Make changes to `libbpf` to look for BTF not only at the
> > > > > > standard locations but also at a user specified location. The BTF file
> > > > > > generated in (a) can be presented here.
> > > > >
> > > > > You can already do that, actually, though it's not very obvious. You
> > > > > can specify (or override) kernel BTF location by using
> > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your
> > > > > BTF location (see [2]). I've been meaning to add it instead to a
> > > > > bpf_object_open_opts, btw, to make its use possible with a BPF
> > > > > skeleton. Also keep in mind that currently libbpf expects that custom
> > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> > > > > we can improve that, of course.
> > > > >
> > > > >   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
> > > >
> > > > [VAMSI-2] I took a look at this and what you suggested above does not
> > > > work as is.
> > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
> > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the
> > > > `target_btf_path` attribute.
> > >
> > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but
> > > since then added a bunch more use cases. So some libbpf changes are
> > > needed to make this work. But it should still work for CO-RE to have a
> > > custom BTF.
> > >
> > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom
> > > BTF as the real kernel BTF, because that will never work for
> > > fentry/fexit, struct_ops, etc. I think it is better to teach
> > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF
> > > if we need it only for CO-RE relocations *and* we have it overloaded
> > > with target_btf_path
> > >
> > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
> > > > you suggested enabling
> > > > use from the BPF skeleton seems useful and I can possibly help with that.
> > >
> > > Yeah, adding something like core_btf_path option to
> > > bpf_object_open_opts would go nicely with this change.
> > >
> > > >
> > > > For now, just for proof of concept I modified the search options in
> > > > `libbpf_find_kernel_btf` to
> > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.
> > > >
> > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
> > >
> > > so here you successfully loaded custom BTF, which is good.
> > >
> > > > libbpf: Kernel doesn't support BTF, skipping uploading it.
> > >
> > > this just means that your BPF object's BTF won't be loaded into the
> > > kernel. That's no big deal, ignore this.
> > >
> > > > libbpf: kernel doesn't support global data
> > >
> > > But this means that your BPF programs rely on global variables, which
> > > are not supported by the kernel. So you need to change the code to not
> > > use global variables to make this work on very old kernels.
> > >
> > >
> > > > libbpf: failed to load object 'tcpconnect_bpf'
> > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
> > > > failed to load BPF object: -95
> > >
> > > This is probably OPNOTSUPP from the global data above
> > >
> > > >
> > > > This is the reason I had posted on the mailer. If the CO-RE executable
> > > > has relocations
> > > > resolved by the time of the BPF load. Why do we need to check for
> > > > kernel support?. Also,
> > > > does this mean what I am attempting to do will not work?.
> > > >
> > >
> > > it will work with minimal libbpf logic changes. Nothing in principle
> > > prevents this.
> > >
> > >
> > > > Best Regards. And again thanks a lot for your precious time.
> > > > - Vamsi.
> > > >
> > > > > >
> > > > > > This should provide us a way to enable CO-RE functionality on older
> > > > > > kernel versions as well. I tried to make the above changes and tried
> > > > > > against a 4.14 kernel and it did not work. Either I am not doing
> > > > > > something right or my assumptions are wrong.
> > > > > >
> > > > > > Thanks in advance for your time. And I hope someone here can guide me
> > > > > > in the right direction.
> > > > > >
> > > > > > Regards
> > > > > > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Vamsi Kodavanty]

On Thu, Jan 7, 2021 at 4:16 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote:
>
> On Thu, Jan 7, 2021 at 3:32 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote:
> > >
> > > Andrii,
> > >     Thank you so much for the quick response. You were right. I was
> > > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata`
> > > which I removed. Now the program goes much further. It now finishes
> > > the relocations successfully. But, fails at the next step
> > > `bpf_object__load_progs`.
> >
> > It's discouraged to use top posting on the mailing list. So for the
> > future please reply inline.
> >
>
> [VAMSI-3] Apologies. Will be mindful next time on.
>
> >
> > >
> > > ===
> > > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50
> > > (ALU/ALU64) imm 56 -> 56
> > > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No
> > > such file or directory
> > > libbpf: failed to determine kprobe perf type: No such file or directory
> > > libbpf: prog 'tcp_v4_connect': failed to create kprobe
> > > 'tcp_v4_connect' perf event: No such file or directory
> > > libbpf: failed to auto-attach program 'tcp_v4_connect': -2
> > > ===
> > >
> > > This host has a 4.14 kernel AmazonLinux2 and does not have the above
> > > file. It instead has this
> > >
> > > ===
> > > $ sudo cat /sys/kernel/debug/tracing/kprobe_events
> > > p:kprobes/p_sys_execve_bcc_2566 sys_execve
> > > ===
> > >
> >
> > Right. Libbpf only supports a newer and safer way to attach to
> > kprobes. For your experiments, try to stick to tracepoints and you'll
> > have a better time.
> >
> > But it's another thing I've been meaning to add to libbpf for
> > supporting older kernels. I even have code written to do legacy kprobe
> > attachment, just need to find time to send a patch to add it as a
> > fallback for kernels that don't support new kprobe interface.
> >
>
> [VAMSI-3] I think that will be very helpful, as there are bound to be hosts
> with older kernels for the foreseeable future. Good to know it's being
> considered.
>
> >
> > > I am guessing this is a backward compatibility issue?. I will try to
> > > look at an earlier version of libbpf to see how this was handled.
> >
> > it was never supported by libbpf. BCC support both old and new APIs,
> > but old APIs is more dangerous, as it's easy to leave attached kprobe
> > BPF program active in the kernel unintentionally. E.g., if the process
> > crashes. But with bpf_link and its expected use of
> > bpf_link__destroy(), we can support the legacy API (it won't be any
> > safer, but still).
> >
>
> [VAMSI-3] Will use tracepoints to validate the process as you suggested. I
> will then try to see what I can do to use this on kprobe/kretprobe's. Will also
> check how BCC does this. I will start a new thread if needed.
>

[VAMSI-4] Just to round up this thread. The tracepoints did work out
of the box.
I tried the `execsnoop` tool and it worked fine.

Thank you again.
Vamsi.

> Again appreciate the inputs and direction very much.
>
> Best Regards
> Vamsi.
>
> > > Meanwhile, if you have further comments they are appreciated.
> > >
> > > And again I can't thank you enough for how helpful you have been and
> > > your time. Cheers!.
> > >
> > >
> > >
> > >
> > > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty
> > > > <vamsi@araalinetworks.com> wrote:
> > > > >
> > > > > First of all thank you very much for your quick response. And helpful pointers.
> > > > > It seems like you also think what I am attempting to do should work.
> > > > >
> > > > > Please see inline [VAMSI-2].
> > > > >
> > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko
> > > > > <andrii.nakryiko@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty
> > > > > > <vamsi@araalinetworks.com> wrote:
> > > > > > >
> > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read
> > > > > > > CO-RE needs a supported kernel version and be compiled with
> > > > > > > `CONFIG_DEBUG_INFO_BTF=y`.
> > > > > > >
> > > > > > > I also understand there are three pieces to enable CO-RE
> > > > > > > functionality. (1) The BTF format. For efficient/compressed kernel
> > > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3)
> > > > > >
> > > > > > BTF is not really a symbol table, rather a type information. Like
> > > > > > simpler and more compact DWARF.
> > > > > >
> > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once
> > > > > > > these 3 steps are done the resulting byte code is no different from
> > > > > > > non-CO-RE byte code.
> > > > > > >
> > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct
> > > > > > > and guide me if I am stating something incorrectly.
> > > > > > >
> > > > > > > (1) Is the kernel support requirement ONLY for the purposes of
> > > > > > > generating and exposing the BTF file information on
> > > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications
> > > > > > > `libbpf` can find the BTF information at a standard location?.
> > > > > >
> > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also
> > > > > > try to search for vmlinux image (and BTF info within it) in a few
> > > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't
> > > > > > even expose /sys/kernel/btf/vmlinux.
> > > > > >
> > > > > >   [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580
> > > > > >
> > > > > > >
> > > > > > > (2) If the answer to the above question is YES. Could the below
> > > > > > > mechanism be used so that it works on all kernels whether they support
> > > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?.
> > > > > > >        (a) Extract BTF generation process outside of the kernel build.
> > > > > > > Use this to generate the equivalent BTF file for it.
> > > > > >
> > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF
> > > > > > info, but it's also possible to just embed BTF manually with a direct
> > > > > > invocation of pahole -J, see [1] on how it's done for
> > > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no
> > > > > > matter the version, and it will work with CO-RE relocations.
> > > > > >
> > > > > >   [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170
> > > > > >
> > > > >
> > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the
> > > > > `gen_btf` from the
> > > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF
> > > > > file for the
> > > > > 4.14.0 kernel.
> > > > >
> > > > > > >        (b) Make changes to `libbpf` to look for BTF not only at the
> > > > > > > standard locations but also at a user specified location. The BTF file
> > > > > > > generated in (a) can be presented here.
> > > > > >
> > > > > > You can already do that, actually, though it's not very obvious. You
> > > > > > can specify (or override) kernel BTF location by using
> > > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your
> > > > > > BTF location (see [2]). I've been meaning to add it instead to a
> > > > > > bpf_object_open_opts, btw, to make its use possible with a BPF
> > > > > > skeleton. Also keep in mind that currently libbpf expects that custom
> > > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But
> > > > > > we can improve that, of course.
> > > > > >
> > > > > >   [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141
> > > > >
> > > > > [VAMSI-2] I took a look at this and what you suggested above does not
> > > > > work as is.
> > > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like
> > > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the
> > > > > `target_btf_path` attribute.
> > > >
> > > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but
> > > > since then added a bunch more use cases. So some libbpf changes are
> > > > needed to make this work. But it should still work for CO-RE to have a
> > > > custom BTF.
> > > >
> > > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom
> > > > BTF as the real kernel BTF, because that will never work for
> > > > fentry/fexit, struct_ops, etc. I think it is better to teach
> > > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF
> > > > if we need it only for CO-RE relocations *and* we have it overloaded
> > > > with target_btf_path
> > > >
> > > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As
> > > > > you suggested enabling
> > > > > use from the BPF skeleton seems useful and I can possibly help with that.
> > > >
> > > > Yeah, adding something like core_btf_path option to
> > > > bpf_object_open_opts would go nicely with this change.
> > > >
> > > > >
> > > > > For now, just for proof of concept I modified the search options in
> > > > > `libbpf_find_kernel_btf` to
> > > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures.
> > > > >
> > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0
> > > >
> > > > so here you successfully loaded custom BTF, which is good.
> > > >
> > > > > libbpf: Kernel doesn't support BTF, skipping uploading it.
> > > >
> > > > this just means that your BPF object's BTF won't be loaded into the
> > > > kernel. That's no big deal, ignore this.
> > > >
> > > > > libbpf: kernel doesn't support global data
> > > >
> > > > But this means that your BPF programs rely on global variables, which
> > > > are not supported by the kernel. So you need to change the code to not
> > > > use global variables to make this work on very old kernels.
> > > >
> > > >
> > > > > libbpf: failed to load object 'tcpconnect_bpf'
> > > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95
> > > > > failed to load BPF object: -95
> > > >
> > > > This is probably OPNOTSUPP from the global data above
> > > >
> > > > >
> > > > > This is the reason I had posted on the mailer. If the CO-RE executable
> > > > > has relocations
> > > > > resolved by the time of the BPF load. Why do we need to check for
> > > > > kernel support?. Also,
> > > > > does this mean what I am attempting to do will not work?.
> > > > >
> > > >
> > > > it will work with minimal libbpf logic changes. Nothing in principle
> > > > prevents this.
> > > >
> > > >
> > > > > Best Regards. And again thanks a lot for your precious time.
> > > > > - Vamsi.
> > > > >
> > > > > > >
> > > > > > > This should provide us a way to enable CO-RE functionality on older
> > > > > > > kernel versions as well. I tried to make the above changes and tried
> > > > > > > against a 4.14 kernel and it did not work. Either I am not doing
> > > > > > > something right or my assumptions are wrong.
> > > > > > >
> > > > > > > Thanks in advance for your time. And I hope someone here can guide me
> > > > > > > in the right direction.
> > > > > > >
> > > > > > > Regards
> > > > > > > Vamsi.

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

> > From:   Vamsi Kodavanty <vamsi@araalinetworks.com>
> > Date:   Thu, 7 Jan 2021 17:31:11 -0800
> > To:     Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > Cc:     bpf <bpf@vger.kernel.org>
> >
> >
> > Right. Libbpf only supports a newer and safer way to attach to
> > kprobes. For your experiments, try to stick to tracepoints and you'll
> > have a better time.
> >
> > But it's another thing I've been meaning to add to libbpf for
> > supporting older kernels. I even have code written to do legacy kprobe
> > attachment, just need to find time to send a patch to add it as a
> > fallback for kernels that don't support new kprobe interface.

Initially I'd like to thank you *a lot* for this thread, it helped me
creating:

https://github.com/rafaeldtinoco/portablebpf

showing up exactly what was discussed here AND I could run the same
binary in v4.15 and v.5.8 kernels as long as BTF was generated with:

https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch

Specially the attach_kprobe_legacy() function:

https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31

I wanted to reply here in case others also face this.

Only bad thing was kernel v4.15 missed global data support as showed in:

https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

But using perf event was good enough for an example.

- rafaeldtinoco

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Wed, Mar 3, 2021 at 10:15 AM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
> > > From:   Vamsi Kodavanty <vamsi@araalinetworks.com>
> > > Date:   Thu, 7 Jan 2021 17:31:11 -0800
> > > To:     Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > > Cc:     bpf <bpf@vger.kernel.org>
> > >
> > >
> > > Right. Libbpf only supports a newer and safer way to attach to
> > > kprobes. For your experiments, try to stick to tracepoints and you'll
> > > have a better time.
> > >
> > > But it's another thing I've been meaning to add to libbpf for
> > > supporting older kernels. I even have code written to do legacy kprobe
> > > attachment, just need to find time to send a patch to add it as a
> > > fallback for kernels that don't support new kprobe interface.
>
> Initially I'd like to thank you *a lot* for this thread, it helped me
> creating:
>
> https://github.com/rafaeldtinoco/portablebpf
>
> showing up exactly what was discussed here AND I could run the same
> binary in v4.15 and v.5.8 kernels as long as BTF was generated with:
>
> https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch

I was wondering if it might be useful to have a script that would use
pahole to do DWARF to BTF conversion for existing vmlinux image (e.g.,
from /boot/vmlinux-$(uname -r)), assuming DWARF is in that vmlinux (or
could be found somewhere nearby), and then would spit out only .BTF
contents as a binary file, which can be passed to libbpf on
bpf_object__open(). That seems useful and there have been at least a
few cases where people tried to use CO-RE on old kernels
pre-CONFIG_DEBUG_INFO_BTF, but were always confused by how to get that
BTF data.

[cc Arnaldo]
It would also simplify things a bunch if pahole had an option to emit
.BTF into a separate non-ELF file, instead of modifying vmlinux
in-place. WDYT?

>
> Specially the attach_kprobe_legacy() function:
>
> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
>
> I wanted to reply here in case others also face this.

Great, glad it worked out. It would be great if you could contribute
legacy kprobe support for libbpf as a proper patch, since it probably
would be useful for a bunch of other people stuck with old kernels.

>
> Only bad thing was kernel v4.15 missed global data support as showed in:
>
> https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
>
> But using perf event was good enough for an example.
>
> - rafaeldtinoco

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Arnaldo Carvalho de Melo]

Em Wed, Mar 03, 2021 at 11:05:44PM -0800, Andrii Nakryiko escreveu:
> On Wed, Mar 3, 2021 at 10:15 AM Rafael David Tinoco
> <rafaeldtinoco@ubuntu.com> wrote:
> >
> > > > From:   Vamsi Kodavanty <vamsi@araalinetworks.com>
> > > > Date:   Thu, 7 Jan 2021 17:31:11 -0800
> > > > To:     Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > > > Cc:     bpf <bpf@vger.kernel.org>
> > > >
> > > >
> > > > Right. Libbpf only supports a newer and safer way to attach to
> > > > kprobes. For your experiments, try to stick to tracepoints and you'll
> > > > have a better time.
> > > >
> > > > But it's another thing I've been meaning to add to libbpf for
> > > > supporting older kernels. I even have code written to do legacy kprobe
> > > > attachment, just need to find time to send a patch to add it as a
> > > > fallback for kernels that don't support new kprobe interface.
> >
> > Initially I'd like to thank you *a lot* for this thread, it helped me
> > creating:
> >
> > https://github.com/rafaeldtinoco/portablebpf
> >
> > showing up exactly what was discussed here AND I could run the same
> > binary in v4.15 and v.5.8 kernels as long as BTF was generated with:
> >
> > https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch
> 
> I was wondering if it might be useful to have a script that would use
> pahole to do DWARF to BTF conversion for existing vmlinux image (e.g.,
> from /boot/vmlinux-$(uname -r)), assuming DWARF is in that vmlinux (or
> could be found somewhere nearby), and then would spit out only .BTF
> contents as a binary file, which can be passed to libbpf on
> bpf_object__open(). That seems useful and there have been at least a
> few cases where people tried to use CO-RE on old kernels
> pre-CONFIG_DEBUG_INFO_BTF, but were always confused by how to get that
> BTF data.
> 
> [cc Arnaldo]
> It would also simplify things a bunch if pahole had an option to emit
> .BTF into a separate non-ELF file, instead of modifying vmlinux
> in-place. WDYT?

Sure, that is a nice addition, makes it more flexible to cover this
usecase.

- Arnaldo
 
> >
> > Specially the attach_kprobe_legacy() function:
> >
> > https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
> >
> > I wanted to reply here in case others also face this.
> 
> Great, glad it worked out. It would be great if you could contribute
> legacy kprobe support for libbpf as a proper patch, since it probably
> would be useful for a bunch of other people stuck with old kernels.
> 
> >
> > Only bad thing was kernel v4.15 missed global data support as showed in:
> >
> > https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
> >
> > But using perf event was good enough for an example.
> >
> > - rafaeldtinoco

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

>> Specially the attach_kprobe_legacy() function:
>> 
>> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
>> 
>> I wanted to reply here in case others also face this.
> 
> Great, glad it worked out. It would be great if you could contribute
> legacy kprobe support for libbpf as a proper patch, since it probably
> would be useful for a bunch of other people stuck with old kernels.

Definitely! Will suggest a patch for this soon.

Thanks for the feedback.

-rafaeldtinoco

Fwd: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]


> On 5 Mar 2021, at 03:32, Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote:
> 
> 
>>> Specially the attach_kprobe_legacy() function:
>>> 
>>> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
>>> 
>>> I wanted to reply here in case others also face this.
>> 
>> Great, glad it worked out. It would be great if you could contribute
>> legacy kprobe support for libbpf as a proper patch, since it probably
>> would be useful for a bunch of other people stuck with old kernels.
> 

(sending this again in proper format for the list)

I’m sorry to come back to this but I’d like to clarify something, if you allow me.

If I recompile old kernels (4.x.y) with the “scripts/link-vmlinux.sh" patch (setting $btf_vmlinux_bin_o and gen_btf()) I’m able to use "pahole -J" to generate the .BTF ELF section from a vmlinux file (out of the debugging package, for example) using its DWARF data.

Using objcopy, I’m also able to extract only the .BTF ELF section from it and use the generated file (smaller) as a base BTF file for libbpf (since old kernels don’t have /sys/kernel/btf/vmlinux interface).

So, in my case, with this, I can get an ~30MB ELF file (from a an almost 600MB vmlinux) with BTF data that can feed libbpf to do needed relocations for my BPF object. Execution works perfectly and I can have the same libbpf based code to run in a 4.15 and a 5.8 kernel, smooth.

What is not entirely clear to me yet is … why can’t I use a “vmlinux” file from a previous compiled kernel (that has not been compiled with a changed link-vmlinux.sh file) and do the same: generate the BTF section from its DWARF data with pahole and use generated file (or the BTF section extract only) as input to libbpf.

I mean, I can do, but it does not work… Assumption: it only works for the ones I build with patched link-vmlinux.sh (not the ones already built and provided as packages). The code execution output (debug=1 on libbpf) is at : https://pastebin.ubuntu.com/p/bx6tygY8p2/

The difference for a new 4.x.y kernel and the existing ones (older packaged kernels) is the vmlinux_link() function linking the BTF object file in each of the 3 tmp_kallsyms steps.

Is there a way I can get the already existing kernels to work with only pahole DWARF to BTF conversion data ?

Thank you!

-rafaeldtinoco
> 


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Tue, Mar 9, 2021 at 9:58 PM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
>
>
> On 5 Mar 2021, at 03:32, Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote:
>
>
> Specially the attach_kprobe_legacy() function:
>
> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31
>
> I wanted to reply here in case others also face this.
>
>
> Great, glad it worked out. It would be great if you could contribute
> legacy kprobe support for libbpf as a proper patch, since it probably
> would be useful for a bunch of other people stuck with old kernels.
>
>
>
> I’m sorry to come back to this but I’d like to clarify something, if you allow me.
>
> If I recompile old kernels (4.x.y) with the “scripts/link-vmlinux.sh" patch (setting $btf_vmlinux_bin_o and gen_btf()) I’m able to use "pahole -J" to generate the .BTF ELF section from a vmlinux file (out of the debugging package, for example) using its DWARF data.
>
> Using objcopy, I’m also able to extract only the .BTF ELF section from it and use the generated file (smaller) as a base BTF file for libbpf (since old kernels don’t have /sys/kernel/btf/vmlinux interface).
>
> So, in my case, with this, I can get an ~30MB ELF file (from a an almost 600MB vmlinux) with BTF data that can feed libbpf to do needed relocations for my BPF object. Execution works perfectly and I can have the same libbpf based code to run in a 4.15 and a 5.8 kernel, smooth.

Surprised that .BTF is so big at 30MB. It depends on kernel config you
are using, but that's still few times bigger than what I normally see.

Otherwise, yeah, that's how it should work (except see the patch that
adds core_btf_path and discussion around it).

>
> What is not entirely clear to me yet is … why can’t I use a “vmlinux” file from a previous compiled kernel (that has not been compiled with a changed link-vmlinux.sh file) and do the same: generate the BTF section from its DWARF data with pahole and use generated file (or the BTF section extract only) as input to libbpf.
>
> I mean, I can do, but it does not work… Assumption: it only works for the ones I build with patched link-vmlinux.sh (not the ones already built and provided as packages). The code execution output (debug=1 on libbpf) is at : https://pastebin.ubuntu.com/p/bx6tygY8p2/
>

From what I see all the CO-RE relocations applied successfully (even
though all the offsets stayed the same, so presumably you compiled
your BPF program with vmlinux.h from the exact same kernel you are
running it on?). Are you sure that vmlinux image you are providing
corresponds to the actual kernel you are running on?

I'd start by comparing libbpf logs for vmlinux you get with modified
link-vmlinux.sh script and with just explicit pahole -J. If all the
CO-RE parts are identical, the problem is somewhere else most
probably.

I see "libbpf: load bpf program failed: Invalid argument" in that log,
which means that CO-RE was done and successful and only when trying to
load your BPF program into the kernel it failed.


> The difference for a new 4.x.y kernel and the existing ones (older packaged kernels) is the vmlinux_link() function linking the BTF object file in each of the 3 tmp_kallsyms steps.
>
> Is there a way I can get the already existing kernels to work with only pahole DWARF to BTF conversion data ?

Yes and you've found it, I think. There is no difference to libbpf and
.BTF itself whether it's run in link-vmlinux.sh or with explicit
pahole -J. Look for the problem somewhere else.

>
> Thank you!
>
> -rafaeldtinoco
>

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

> From what I see all the CO-RE relocations applied successfully (even
> though all the offsets stayed the same, so presumably you compiled
> your BPF program with vmlinux.h from the exact same kernel you are
> running it on?). Are you sure that vmlinux image you are providing
> corresponds to the actual kernel you are running on?

Yep, I have created the following:

https://pastebin.ubuntu.com/p/h58YyNr4HR/

to make this easier.

It is a 4.15.0-1080 kernel and a 4.15.18+. They are pretty close
despite the versioning (second was generated with make deb-dpkg).

I’m using same .config file for both, only difference is that the
4.15.18+ was compiled with the changed link-vmlinux.sh file.

The /usr/lib/debug/boot/vmlinux files are generated by the same
build and I have tried 2 or 3 of the existing packaged kernels. The
only thing I did was “pahole -J” to /usr/lib/debug/boot/vmlinux-XXX
files (adding the BTF section to them).

Running same binary in a 5.8.0-43 kernel works out-of-the-box:

https://pastebin.ubuntu.com/p/VKTjMcp6Xs/

>
> I'd start by comparing libbpf logs for vmlinux you get with modified
> link-vmlinux.sh script and with just explicit pahole -J. If all the
> CO-RE parts are identical, the problem is somewhere else most
> probably.

The difference between (1) and (2) from the paste (error and success):

libbpf: CO-RE relocating [0] struct task_struct: found target
candidate [17361] struct task_struct in [vmlinux]
libbpf: CO-RE relocating [0] struct task_struct: found target
candidate [17360] struct task_struct in [vmlinux]

libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17361]
struct task_struct.comm (0:90 @ offset 2640)
libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17360]
struct task_struct.comm (0:90 @ offset 2640)

Code is at:

https://github.com/rafaeldtinoco/portablebpf

and it is not much different than any other libbpf example.

thanks again for verifying this!

-rafaeldtinoco

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
>
> > From what I see all the CO-RE relocations applied successfully (even
> > though all the offsets stayed the same, so presumably you compiled
> > your BPF program with vmlinux.h from the exact same kernel you are
> > running it on?). Are you sure that vmlinux image you are providing
> > corresponds to the actual kernel you are running on?
>
> Yep, I have created the following:
>
> https://pastebin.ubuntu.com/p/h58YyNr4HR/
>

Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux
is some way (but then why would kernel start and only fail to
load/verify your BPF program?). It's also clear that CO-RE is doing
exactly the same instruction patching, so shouldn't be some invalid
CO-RE relocation applied, I think. So no idea and not sure how to
investigate this.

But I think I'd never do `pahole -J` on actual vmlinux image you are
going to run. It's much safer and more convenient to make a copy,
generate .BTF and then extract just .BTF section into a small binary,
which can be provided separately.


> to make this easier.
>
> It is a 4.15.0-1080 kernel and a 4.15.18+. They are pretty close

Also this. Seems like it is two different kernels still, however small
the difference between them is. Have you tried building *exactly* the
same kernel with exactly the same config, but just with pahole -J
during the build on vmlinux.o (which is later linked into a final
vmlinux) vs running pahole -J on final vmlinux after the build. Notice
that two approach differ in terms of which object file is being
modified. With link-vmlinux.sh, kernel linker scripts dictate final
layout, while if you run `pahole -J` on final vmlinux, it can modify
the layout and potentially break something sensitive. pahole is using
llvm-objcopy internally, and llvm-objcopy isn't a very trivial tool.


> despite the versioning (second was generated with make deb-dpkg).
>
> I’m using same .config file for both, only difference is that the
> 4.15.18+ was compiled with the changed link-vmlinux.sh file.
>
> The /usr/lib/debug/boot/vmlinux files are generated by the same
> build and I have tried 2 or 3 of the existing packaged kernels. The
> only thing I did was “pahole -J” to /usr/lib/debug/boot/vmlinux-XXX
> files (adding the BTF section to them).
>
> Running same binary in a 5.8.0-43 kernel works out-of-the-box:
>
> https://pastebin.ubuntu.com/p/VKTjMcp6Xs/
>
> >
> > I'd start by comparing libbpf logs for vmlinux you get with modified
> > link-vmlinux.sh script and with just explicit pahole -J. If all the
> > CO-RE parts are identical, the problem is somewhere else most
> > probably.
>
> The difference between (1) and (2) from the paste (error and success):
>
> libbpf: CO-RE relocating [0] struct task_struct: found target
> candidate [17361] struct task_struct in [vmlinux]
> libbpf: CO-RE relocating [0] struct task_struct: found target
> candidate [17360] struct task_struct in [vmlinux]
>
> libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17361]
> struct task_struct.comm (0:90 @ offset 2640)
> libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17360]
> struct task_struct.comm (0:90 @ offset 2640)
>
> Code is at:
>
> https://github.com/rafaeldtinoco/portablebpf
>
> and it is not much different than any other libbpf example.
>
> thanks again for verifying this!
>
> -rafaeldtinoco
>
>
>

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco
> <rafaeldtinoco@ubuntu.com> wrote:
>>> From what I see all the CO-RE relocations applied successfully (even
>>> though all the offsets stayed the same, so presumably you compiled
>>> your BPF program with vmlinux.h from the exact same kernel you are
>>> running it on?). Are you sure that vmlinux image you are providing
>>> corresponds to the actual kernel you are running on?
>>
>> Yep, I have created the following:
>>
>> https://pastebin.ubuntu.com/p/h58YyNr4HR/
>
> Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux
> is some way (but then why would kernel start and only fail to
> load/verify your BPF program?). It's also clear that CO-RE is doing
> exactly the same instruction patching, so shouldn't be some invalid
> CO-RE relocation applied, I think. So no idea and not sure how to
> investigate this.
>
> But I think I'd never do `pahole -J` on actual vmlinux image you are
> going to run. It's much safer and more convenient to make a copy,
> generate .BTF and then extract just .BTF section into a small binary,
> which can be provided separately.
>

FOUND the cause of the issue...

Compiling the EXACT same kernel with different building scripts
(deb-pkg vs debian/rules in my case) resulted in a very similar
kernel (same .config, same autoconf.h, no visible changes).

Unfortunately one of the kernels worked fined (reading the BTF
extracted section OR same section within a vmlinux entire
file).. but the other did not.

Instrumenting this bad 4.15 kernel (out of debian/rules build)
I found that the following sanity checks took place in kernel:

bpf_check():

if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level  
|| !log->ubuf)

and a simple change in libbpf (mitigation of course):

   attr.log_buf = 0;
   attr.log_level = 0;
   attr.log_size = 0;

before

fd = sys_bpf_prog_load(&attr, sizeof(attr));

made the code to also run in this second kernel (built with the
debian/rules building scripts):

https://pastebin.ubuntu.com/p/scJDM3D9Zr/

Now I still have to discover why log_buf is miss-behaving in this
kernel being built with debian/rules* scripts and not with the
vanilla building scripts (despite config file being the same).

(FYIO, documenting it here for others also)...

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

>>
>>>> From what I see all the CO-RE relocations applied successfully (even
>>>> though all the offsets stayed the same, so presumably you compiled
>>>> your BPF program with vmlinux.h from the exact same kernel you are
>>>> running it on?). Are you sure that vmlinux image you are providing
>>>> corresponds to the actual kernel you are running on?
>
> FOUND the cause of the issue…
>

...

>
> bpf_check():
>
> if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level  
> || !log->ubuf)
>
> and a simple change in libbpf (mitigation of course):
>
>   attr.log_buf = 0;
>   attr.log_level = 0;
>   attr.log_size = 0;
>
> before
>
> fd = sys_bpf_prog_load(&attr, sizeof(attr));

With instrumented kernel… no changes to libbpf or code:

attr.log_buf = (nil)
attr.log_level = 0
attr.log_size = 0
load_attr.log_buf = 0x7fd1c0a92010
load_attr.log_level = 0
load_attr.log_size = 16777215
libbpf: load bpf program failed: Invalid argument
libbpf: failed to load program 'ip_set_create'
libbpf: failed to load object 'mine_bpf'
libbpf: failed to load BPF skeleton 'mine_bpf': -22
failed to load BPF object: -22

[   27.857858] MINE: BPF_PROG_TYPE_KPROBE KERNEL VERSION ISSUE
[   27.857993] MINE: LINUX_VERSION_CODE: 266002
[   27.858131] MINE: YOUR VERSION: 265984

2 problems here:

0) attr.kern_version

- looks like for some reason attr.kern_version is different from
the one running

- bypassing kernel BPF_KPROBE version check, I get:

1) load_attr has log_buf and log_size but not log_level for some reason.

- this triggers an issue in bpf_check() within kernel IF I bypass
the BPF_KPROBE version check (kerne 4.15).

NOW, If I hard-code attr.kern_version in bpf.c to:

attr.kern_version = 266002;
fd = sys_bpf_prog_load(&attr, sizeof(attr));

then

attr.log_buf = (nil)
attr.log_level = 0
attr.log_size = 0
load_attr.log_buf = (nil)
load_attr.log_level = 0
load_attr.log_size = 0
Tracing... Hit Ctrl-C to end.

I don’t have the 2 problems, as log_level is zeroed, together with
log_buf and log_size.

Any clue on why this happens ?

Thank you!

-rafaeldtinoco

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Andrii Nakryiko]

On Wed, Mar 17, 2021 at 7:31 AM Rafael David Tinoco
<rafaeldtinoco@ubuntu.com> wrote:
>
>
> >>
> >>>> From what I see all the CO-RE relocations applied successfully (even
> >>>> though all the offsets stayed the same, so presumably you compiled
> >>>> your BPF program with vmlinux.h from the exact same kernel you are
> >>>> running it on?). Are you sure that vmlinux image you are providing
> >>>> corresponds to the actual kernel you are running on?
> >
> > FOUND the cause of the issue…
> >
>
> ...
>
> >
> > bpf_check():
> >
> > if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level
> > || !log->ubuf)
> >
> > and a simple change in libbpf (mitigation of course):
> >
> >   attr.log_buf = 0;
> >   attr.log_level = 0;
> >   attr.log_size = 0;
> >
> > before
> >
> > fd = sys_bpf_prog_load(&attr, sizeof(attr));
>
> With instrumented kernel… no changes to libbpf or code:
>
> attr.log_buf = (nil)
> attr.log_level = 0
> attr.log_size = 0
> load_attr.log_buf = 0x7fd1c0a92010
> load_attr.log_level = 0
> load_attr.log_size = 16777215
> libbpf: load bpf program failed: Invalid argument
> libbpf: failed to load program 'ip_set_create'
> libbpf: failed to load object 'mine_bpf'
> libbpf: failed to load BPF skeleton 'mine_bpf': -22
> failed to load BPF object: -22
>
> [   27.857858] MINE: BPF_PROG_TYPE_KPROBE KERNEL VERSION ISSUE
> [   27.857993] MINE: LINUX_VERSION_CODE: 266002
> [   27.858131] MINE: YOUR VERSION: 265984
>
> 2 problems here:
>
> 0) attr.kern_version
>
> - looks like for some reason attr.kern_version is different from
> the one running
>
> - bypassing kernel BPF_KPROBE version check, I get:
>
> 1) load_attr has log_buf and log_size but not log_level for some reason.
>
> - this triggers an issue in bpf_check() within kernel IF I bypass
> the BPF_KPROBE version check (kerne 4.15).
>
> NOW, If I hard-code attr.kern_version in bpf.c to:
>
> attr.kern_version = 266002;
> fd = sys_bpf_prog_load(&attr, sizeof(attr));
>
> then
>
> attr.log_buf = (nil)
> attr.log_level = 0
> attr.log_size = 0
> load_attr.log_buf = (nil)
> load_attr.log_level = 0
> load_attr.log_size = 0
> Tracing... Hit Ctrl-C to end.
>
> I don’t have the 2 problems, as log_level is zeroed, together with
> log_buf and log_size.
>
> Any clue on why this happens ?

So seems like you figured out kernel_version check, right? And it
seems like log_buf is not really a problem as well? Or it still
causing issues?

>
> Thank you!
>
> -rafaeldtinoco
>

Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions.

[Rafael David Tinoco]

>> Any clue on why this happens ?
>
> So seems like you figured out kernel_version check, right? And it
> seems like log_buf is not really a problem as well? Or it still
> causing issues?
>

All good here Andrii. Working on the legacy kprobe and kern_version
patches and will address anything else there.

tks, cheers