* [BPF CO-RE clarification] Use CO-RE on older kernel versions. @ 2021-01-06 18:02 Vamsi Kodavanty 2021-01-06 23:55 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Vamsi Kodavanty @ 2021-01-06 18:02 UTC (permalink / raw) To: bpf Had a few questions on CO-RE dependencies and usage. From what I read CO-RE needs a supported kernel version and be compiled with `CONFIG_DEBUG_INFO_BTF=y`. I also understand there are three pieces to enable CO-RE functionality. (1) The BTF format. For efficient/compressed kernel symbol table. (2) clang changes to emit the BTF relocations. (3) `libbpf` changes to locate a BTF file and fix-up relocations. Once these 3 steps are done the resulting byte code is no different from non-CO-RE byte code. Given this I am hoping the knowledgeable folks on this mailer correct and guide me if I am stating something incorrectly. (1) Is the kernel support requirement ONLY for the purposes of generating and exposing the BTF file information on `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications `libbpf` can find the BTF information at a standard location?. (2) If the answer to the above question is YES. Could the below mechanism be used so that it works on all kernels whether they support the `CONFIG_DEBUG_INFO_BTF` flag or not?. (a) Extract BTF generation process outside of the kernel build. Use this to generate the equivalent BTF file for it. (b) Make changes to `libbpf` to look for BTF not only at the standard locations but also at a user specified location. The BTF file generated in (a) can be presented here. This should provide us a way to enable CO-RE functionality on older kernel versions as well. I tried to make the above changes and tried against a 4.14 kernel and it did not work. Either I am not doing something right or my assumptions are wrong. Thanks in advance for your time. And I hope someone here can guide me in the right direction. Regards Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-06 18:02 [BPF CO-RE clarification] Use CO-RE on older kernel versions Vamsi Kodavanty @ 2021-01-06 23:55 ` Andrii Nakryiko 2021-01-07 18:12 ` Vamsi Kodavanty 0 siblings, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-01-06 23:55 UTC (permalink / raw) To: Vamsi Kodavanty; +Cc: bpf On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > Had a few questions on CO-RE dependencies and usage. From what I read > CO-RE needs a supported kernel version and be compiled with > `CONFIG_DEBUG_INFO_BTF=y`. > > I also understand there are three pieces to enable CO-RE > functionality. (1) The BTF format. For efficient/compressed kernel > symbol table. (2) clang changes to emit the BTF relocations. (3) BTF is not really a symbol table, rather a type information. Like simpler and more compact DWARF. > `libbpf` changes to locate a BTF file and fix-up relocations. Once > these 3 steps are done the resulting byte code is no different from > non-CO-RE byte code. > > Given this I am hoping the knowledgeable folks on this mailer correct > and guide me if I am stating something incorrectly. > > (1) Is the kernel support requirement ONLY for the purposes of > generating and exposing the BTF file information on > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > `libbpf` can find the BTF information at a standard location?. /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also try to search for vmlinux image (and BTF info within it) in a few standard locations, see [0]. Early versions of in-kernel BTF didn't even expose /sys/kernel/btf/vmlinux. [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > (2) If the answer to the above question is YES. Could the below > mechanism be used so that it works on all kernels whether they support > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > (a) Extract BTF generation process outside of the kernel build. > Use this to generate the equivalent BTF file for it. Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF info, but it's also possible to just embed BTF manually with a direct invocation of pahole -J, see [1] on how it's done for CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no matter the version, and it will work with CO-RE relocations. [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > (b) Make changes to `libbpf` to look for BTF not only at the > standard locations but also at a user specified location. The BTF file > generated in (a) can be presented here. You can already do that, actually, though it's not very obvious. You can specify (or override) kernel BTF location by using bpf_object__load_xattr() and passing target_btf_path pointing to your BTF location (see [2]). I've been meaning to add it instead to a bpf_object_open_opts, btw, to make its use possible with a BPF skeleton. Also keep in mind that currently libbpf expects that custom BTF to be an ELF file with .BTF section, not just a raw BTF data. But we can improve that, of course. [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > This should provide us a way to enable CO-RE functionality on older > kernel versions as well. I tried to make the above changes and tried > against a 4.14 kernel and it did not work. Either I am not doing > something right or my assumptions are wrong. > > Thanks in advance for your time. And I hope someone here can guide me > in the right direction. > > Regards > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-06 23:55 ` Andrii Nakryiko @ 2021-01-07 18:12 ` Vamsi Kodavanty 2021-01-07 18:52 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Vamsi Kodavanty @ 2021-01-07 18:12 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf First of all thank you very much for your quick response. And helpful pointers. It seems like you also think what I am attempting to do should work. Please see inline [VAMSI-2]. On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > <vamsi@araalinetworks.com> wrote: > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > CO-RE needs a supported kernel version and be compiled with > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > I also understand there are three pieces to enable CO-RE > > functionality. (1) The BTF format. For efficient/compressed kernel > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > BTF is not really a symbol table, rather a type information. Like > simpler and more compact DWARF. > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > these 3 steps are done the resulting byte code is no different from > > non-CO-RE byte code. > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > and guide me if I am stating something incorrectly. > > > > (1) Is the kernel support requirement ONLY for the purposes of > > generating and exposing the BTF file information on > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > `libbpf` can find the BTF information at a standard location?. > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > try to search for vmlinux image (and BTF info within it) in a few > standard locations, see [0]. Early versions of in-kernel BTF didn't > even expose /sys/kernel/btf/vmlinux. > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > (2) If the answer to the above question is YES. Could the below > > mechanism be used so that it works on all kernels whether they support > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > (a) Extract BTF generation process outside of the kernel build. > > Use this to generate the equivalent BTF file for it. > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > info, but it's also possible to just embed BTF manually with a direct > invocation of pahole -J, see [1] on how it's done for > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > matter the version, and it will work with CO-RE relocations. > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > [VAMSI-2] Yes, this is exactly what I did. I extracted out the `gen_btf` from the `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF file for the 4.14.0 kernel. > > (b) Make changes to `libbpf` to look for BTF not only at the > > standard locations but also at a user specified location. The BTF file > > generated in (a) can be presented here. > > You can already do that, actually, though it's not very obvious. You > can specify (or override) kernel BTF location by using > bpf_object__load_xattr() and passing target_btf_path pointing to your > BTF location (see [2]). I've been meaning to add it instead to a > bpf_object_open_opts, btw, to make its use possible with a BPF > skeleton. Also keep in mind that currently libbpf expects that custom > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > we can improve that, of course. > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 [VAMSI-2] I took a look at this and what you suggested above does not work as is. Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like `bpf_object__load_vmlinux_btf` is not yet modified to use the `target_btf_path` attribute. Only, the `bpf_object__relocate` looks at the `target_btf_path`. As you suggested enabling use from the BPF skeleton seems useful and I can possibly help with that. For now, just for proof of concept I modified the search options in `libbpf_find_kernel_btf` to include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 libbpf: Kernel doesn't support BTF, skipping uploading it. libbpf: kernel doesn't support global data libbpf: failed to load object 'tcpconnect_bpf' libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 failed to load BPF object: -95 This is the reason I had posted on the mailer. If the CO-RE executable has relocations resolved by the time of the BPF load. Why do we need to check for kernel support?. Also, does this mean what I am attempting to do will not work?. Best Regards. And again thanks a lot for your precious time. - Vamsi. > > > > This should provide us a way to enable CO-RE functionality on older > > kernel versions as well. I tried to make the above changes and tried > > against a 4.14 kernel and it did not work. Either I am not doing > > something right or my assumptions are wrong. > > > > Thanks in advance for your time. And I hope someone here can guide me > > in the right direction. > > > > Regards > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-07 18:12 ` Vamsi Kodavanty @ 2021-01-07 18:52 ` Andrii Nakryiko 2021-01-07 22:45 ` Vamsi Kodavanty 0 siblings, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-01-07 18:52 UTC (permalink / raw) To: Vamsi Kodavanty; +Cc: bpf On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > First of all thank you very much for your quick response. And helpful pointers. > It seems like you also think what I am attempting to do should work. > > Please see inline [VAMSI-2]. > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > <andrii.nakryiko@gmail.com> wrote: > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > <vamsi@araalinetworks.com> wrote: > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > CO-RE needs a supported kernel version and be compiled with > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > I also understand there are three pieces to enable CO-RE > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > BTF is not really a symbol table, rather a type information. Like > > simpler and more compact DWARF. > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > these 3 steps are done the resulting byte code is no different from > > > non-CO-RE byte code. > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > and guide me if I am stating something incorrectly. > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > generating and exposing the BTF file information on > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > `libbpf` can find the BTF information at a standard location?. > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > try to search for vmlinux image (and BTF info within it) in a few > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > even expose /sys/kernel/btf/vmlinux. > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > mechanism be used so that it works on all kernels whether they support > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > (a) Extract BTF generation process outside of the kernel build. > > > Use this to generate the equivalent BTF file for it. > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > info, but it's also possible to just embed BTF manually with a direct > > invocation of pahole -J, see [1] on how it's done for > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > matter the version, and it will work with CO-RE relocations. > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > `gen_btf` from the > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > file for the > 4.14.0 kernel. > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > standard locations but also at a user specified location. The BTF file > > > generated in (a) can be presented here. > > > > You can already do that, actually, though it's not very obvious. You > > can specify (or override) kernel BTF location by using > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > BTF location (see [2]). I've been meaning to add it instead to a > > bpf_object_open_opts, btw, to make its use possible with a BPF > > skeleton. Also keep in mind that currently libbpf expects that custom > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > we can improve that, of course. > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > [VAMSI-2] I took a look at this and what you suggested above does not > work as is. > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > `bpf_object__load_vmlinux_btf` is not yet modified to use the > `target_btf_path` attribute. Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but since then added a bunch more use cases. So some libbpf changes are needed to make this work. But it should still work for CO-RE to have a custom BTF. I'm not sure about making bpf_object__load_vmlinux_btf() load custom BTF as the real kernel BTF, because that will never work for fentry/fexit, struct_ops, etc. I think it is better to teach bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF if we need it only for CO-RE relocations *and* we have it overloaded with target_btf_path > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > you suggested enabling > use from the BPF skeleton seems useful and I can possibly help with that. Yeah, adding something like core_btf_path option to bpf_object_open_opts would go nicely with this change. > > For now, just for proof of concept I modified the search options in > `libbpf_find_kernel_btf` to > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 so here you successfully loaded custom BTF, which is good. > libbpf: Kernel doesn't support BTF, skipping uploading it. this just means that your BPF object's BTF won't be loaded into the kernel. That's no big deal, ignore this. > libbpf: kernel doesn't support global data But this means that your BPF programs rely on global variables, which are not supported by the kernel. So you need to change the code to not use global variables to make this work on very old kernels. > libbpf: failed to load object 'tcpconnect_bpf' > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > failed to load BPF object: -95 This is probably OPNOTSUPP from the global data above > > This is the reason I had posted on the mailer. If the CO-RE executable > has relocations > resolved by the time of the BPF load. Why do we need to check for > kernel support?. Also, > does this mean what I am attempting to do will not work?. > it will work with minimal libbpf logic changes. Nothing in principle prevents this. > Best Regards. And again thanks a lot for your precious time. > - Vamsi. > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > kernel versions as well. I tried to make the above changes and tried > > > against a 4.14 kernel and it did not work. Either I am not doing > > > something right or my assumptions are wrong. > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > in the right direction. > > > > > > Regards > > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-07 18:52 ` Andrii Nakryiko @ 2021-01-07 22:45 ` Vamsi Kodavanty 2021-01-07 23:32 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Vamsi Kodavanty @ 2021-01-07 22:45 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf Andrii, Thank you so much for the quick response. You were right. I was trying to CO-RE the `tcpconnect.py` program. It had some `.rodata` which I removed. Now the program goes much further. It now finishes the relocations successfully. But, fails at the next step `bpf_object__load_progs`. === libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50 (ALU/ALU64) imm 56 -> 56 libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No such file or directory libbpf: failed to determine kprobe perf type: No such file or directory libbpf: prog 'tcp_v4_connect': failed to create kprobe 'tcp_v4_connect' perf event: No such file or directory libbpf: failed to auto-attach program 'tcp_v4_connect': -2 === This host has a 4.14 kernel AmazonLinux2 and does not have the above file. It instead has this === $ sudo cat /sys/kernel/debug/tracing/kprobe_events p:kprobes/p_sys_execve_bcc_2566 sys_execve === I am guessing this is a backward compatibility issue?. I will try to look at an earlier version of libbpf to see how this was handled. Meanwhile, if you have further comments they are appreciated. And again I can't thank you enough for how helpful you have been and your time. Cheers!. On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty > <vamsi@araalinetworks.com> wrote: > > > > First of all thank you very much for your quick response. And helpful pointers. > > It seems like you also think what I am attempting to do should work. > > > > Please see inline [VAMSI-2]. > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > > <andrii.nakryiko@gmail.com> wrote: > > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > > CO-RE needs a supported kernel version and be compiled with > > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > > > I also understand there are three pieces to enable CO-RE > > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > > > BTF is not really a symbol table, rather a type information. Like > > > simpler and more compact DWARF. > > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > > these 3 steps are done the resulting byte code is no different from > > > > non-CO-RE byte code. > > > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > > and guide me if I am stating something incorrectly. > > > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > > generating and exposing the BTF file information on > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > > `libbpf` can find the BTF information at a standard location?. > > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > > try to search for vmlinux image (and BTF info within it) in a few > > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > > even expose /sys/kernel/btf/vmlinux. > > > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > > mechanism be used so that it works on all kernels whether they support > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > > (a) Extract BTF generation process outside of the kernel build. > > > > Use this to generate the equivalent BTF file for it. > > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > > info, but it's also possible to just embed BTF manually with a direct > > > invocation of pahole -J, see [1] on how it's done for > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > > matter the version, and it will work with CO-RE relocations. > > > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > > `gen_btf` from the > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > > file for the > > 4.14.0 kernel. > > > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > > standard locations but also at a user specified location. The BTF file > > > > generated in (a) can be presented here. > > > > > > You can already do that, actually, though it's not very obvious. You > > > can specify (or override) kernel BTF location by using > > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > > BTF location (see [2]). I've been meaning to add it instead to a > > > bpf_object_open_opts, btw, to make its use possible with a BPF > > > skeleton. Also keep in mind that currently libbpf expects that custom > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > > we can improve that, of course. > > > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > > > [VAMSI-2] I took a look at this and what you suggested above does not > > work as is. > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > > `bpf_object__load_vmlinux_btf` is not yet modified to use the > > `target_btf_path` attribute. > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but > since then added a bunch more use cases. So some libbpf changes are > needed to make this work. But it should still work for CO-RE to have a > custom BTF. > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom > BTF as the real kernel BTF, because that will never work for > fentry/fexit, struct_ops, etc. I think it is better to teach > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF > if we need it only for CO-RE relocations *and* we have it overloaded > with target_btf_path > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > > you suggested enabling > > use from the BPF skeleton seems useful and I can possibly help with that. > > Yeah, adding something like core_btf_path option to > bpf_object_open_opts would go nicely with this change. > > > > > For now, just for proof of concept I modified the search options in > > `libbpf_find_kernel_btf` to > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 > > so here you successfully loaded custom BTF, which is good. > > > libbpf: Kernel doesn't support BTF, skipping uploading it. > > this just means that your BPF object's BTF won't be loaded into the > kernel. That's no big deal, ignore this. > > > libbpf: kernel doesn't support global data > > But this means that your BPF programs rely on global variables, which > are not supported by the kernel. So you need to change the code to not > use global variables to make this work on very old kernels. > > > > libbpf: failed to load object 'tcpconnect_bpf' > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > > failed to load BPF object: -95 > > This is probably OPNOTSUPP from the global data above > > > > > This is the reason I had posted on the mailer. If the CO-RE executable > > has relocations > > resolved by the time of the BPF load. Why do we need to check for > > kernel support?. Also, > > does this mean what I am attempting to do will not work?. > > > > it will work with minimal libbpf logic changes. Nothing in principle > prevents this. > > > > Best Regards. And again thanks a lot for your precious time. > > - Vamsi. > > > > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > > kernel versions as well. I tried to make the above changes and tried > > > > against a 4.14 kernel and it did not work. Either I am not doing > > > > something right or my assumptions are wrong. > > > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > > in the right direction. > > > > > > > > Regards > > > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-07 22:45 ` Vamsi Kodavanty @ 2021-01-07 23:32 ` Andrii Nakryiko 2021-01-08 0:16 ` Vamsi Kodavanty 0 siblings, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-01-07 23:32 UTC (permalink / raw) To: Vamsi Kodavanty; +Cc: bpf On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > Andrii, > Thank you so much for the quick response. You were right. I was > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata` > which I removed. Now the program goes much further. It now finishes > the relocations successfully. But, fails at the next step > `bpf_object__load_progs`. It's discouraged to use top posting on the mailing list. So for the future please reply inline. > > === > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50 > (ALU/ALU64) imm 56 -> 56 > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No > such file or directory > libbpf: failed to determine kprobe perf type: No such file or directory > libbpf: prog 'tcp_v4_connect': failed to create kprobe > 'tcp_v4_connect' perf event: No such file or directory > libbpf: failed to auto-attach program 'tcp_v4_connect': -2 > === > > This host has a 4.14 kernel AmazonLinux2 and does not have the above > file. It instead has this > > === > $ sudo cat /sys/kernel/debug/tracing/kprobe_events > p:kprobes/p_sys_execve_bcc_2566 sys_execve > === > Right. Libbpf only supports a newer and safer way to attach to kprobes. For your experiments, try to stick to tracepoints and you'll have a better time. But it's another thing I've been meaning to add to libbpf for supporting older kernels. I even have code written to do legacy kprobe attachment, just need to find time to send a patch to add it as a fallback for kernels that don't support new kprobe interface. > I am guessing this is a backward compatibility issue?. I will try to > look at an earlier version of libbpf to see how this was handled. it was never supported by libbpf. BCC support both old and new APIs, but old APIs is more dangerous, as it's easy to leave attached kprobe BPF program active in the kernel unintentionally. E.g., if the process crashes. But with bpf_link and its expected use of bpf_link__destroy(), we can support the legacy API (it won't be any safer, but still). > Meanwhile, if you have further comments they are appreciated. > > And again I can't thank you enough for how helpful you have been and > your time. Cheers!. > > > > > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko > <andrii.nakryiko@gmail.com> wrote: > > > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty > > <vamsi@araalinetworks.com> wrote: > > > > > > First of all thank you very much for your quick response. And helpful pointers. > > > It seems like you also think what I am attempting to do should work. > > > > > > Please see inline [VAMSI-2]. > > > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > > > <andrii.nakryiko@gmail.com> wrote: > > > > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > > > CO-RE needs a supported kernel version and be compiled with > > > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > > > > > I also understand there are three pieces to enable CO-RE > > > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > > > > > BTF is not really a symbol table, rather a type information. Like > > > > simpler and more compact DWARF. > > > > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > > > these 3 steps are done the resulting byte code is no different from > > > > > non-CO-RE byte code. > > > > > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > > > and guide me if I am stating something incorrectly. > > > > > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > > > generating and exposing the BTF file information on > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > > > `libbpf` can find the BTF information at a standard location?. > > > > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > > > try to search for vmlinux image (and BTF info within it) in a few > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > > > even expose /sys/kernel/btf/vmlinux. > > > > > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > > > mechanism be used so that it works on all kernels whether they support > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > > > (a) Extract BTF generation process outside of the kernel build. > > > > > Use this to generate the equivalent BTF file for it. > > > > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > > > info, but it's also possible to just embed BTF manually with a direct > > > > invocation of pahole -J, see [1] on how it's done for > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > > > matter the version, and it will work with CO-RE relocations. > > > > > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > > > > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > > > `gen_btf` from the > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > > > file for the > > > 4.14.0 kernel. > > > > > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > > > standard locations but also at a user specified location. The BTF file > > > > > generated in (a) can be presented here. > > > > > > > > You can already do that, actually, though it's not very obvious. You > > > > can specify (or override) kernel BTF location by using > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > > > BTF location (see [2]). I've been meaning to add it instead to a > > > > bpf_object_open_opts, btw, to make its use possible with a BPF > > > > skeleton. Also keep in mind that currently libbpf expects that custom > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > > > we can improve that, of course. > > > > > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > > > > > [VAMSI-2] I took a look at this and what you suggested above does not > > > work as is. > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the > > > `target_btf_path` attribute. > > > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but > > since then added a bunch more use cases. So some libbpf changes are > > needed to make this work. But it should still work for CO-RE to have a > > custom BTF. > > > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom > > BTF as the real kernel BTF, because that will never work for > > fentry/fexit, struct_ops, etc. I think it is better to teach > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF > > if we need it only for CO-RE relocations *and* we have it overloaded > > with target_btf_path > > > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > > > you suggested enabling > > > use from the BPF skeleton seems useful and I can possibly help with that. > > > > Yeah, adding something like core_btf_path option to > > bpf_object_open_opts would go nicely with this change. > > > > > > > > For now, just for proof of concept I modified the search options in > > > `libbpf_find_kernel_btf` to > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 > > > > so here you successfully loaded custom BTF, which is good. > > > > > libbpf: Kernel doesn't support BTF, skipping uploading it. > > > > this just means that your BPF object's BTF won't be loaded into the > > kernel. That's no big deal, ignore this. > > > > > libbpf: kernel doesn't support global data > > > > But this means that your BPF programs rely on global variables, which > > are not supported by the kernel. So you need to change the code to not > > use global variables to make this work on very old kernels. > > > > > > > libbpf: failed to load object 'tcpconnect_bpf' > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > > > failed to load BPF object: -95 > > > > This is probably OPNOTSUPP from the global data above > > > > > > > > This is the reason I had posted on the mailer. If the CO-RE executable > > > has relocations > > > resolved by the time of the BPF load. Why do we need to check for > > > kernel support?. Also, > > > does this mean what I am attempting to do will not work?. > > > > > > > it will work with minimal libbpf logic changes. Nothing in principle > > prevents this. > > > > > > > Best Regards. And again thanks a lot for your precious time. > > > - Vamsi. > > > > > > > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > > > kernel versions as well. I tried to make the above changes and tried > > > > > against a 4.14 kernel and it did not work. Either I am not doing > > > > > something right or my assumptions are wrong. > > > > > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > > > in the right direction. > > > > > > > > > > Regards > > > > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-07 23:32 ` Andrii Nakryiko @ 2021-01-08 0:16 ` Vamsi Kodavanty 2021-01-08 1:31 ` Vamsi Kodavanty 0 siblings, 1 reply; 20+ messages in thread From: Vamsi Kodavanty @ 2021-01-08 0:16 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf On Thu, Jan 7, 2021 at 3:32 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > > > Andrii, > > Thank you so much for the quick response. You were right. I was > > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata` > > which I removed. Now the program goes much further. It now finishes > > the relocations successfully. But, fails at the next step > > `bpf_object__load_progs`. > > It's discouraged to use top posting on the mailing list. So for the > future please reply inline. > [VAMSI-3] Apologies. Will be mindful next time on. > > > > > === > > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50 > > (ALU/ALU64) imm 56 -> 56 > > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No > > such file or directory > > libbpf: failed to determine kprobe perf type: No such file or directory > > libbpf: prog 'tcp_v4_connect': failed to create kprobe > > 'tcp_v4_connect' perf event: No such file or directory > > libbpf: failed to auto-attach program 'tcp_v4_connect': -2 > > === > > > > This host has a 4.14 kernel AmazonLinux2 and does not have the above > > file. It instead has this > > > > === > > $ sudo cat /sys/kernel/debug/tracing/kprobe_events > > p:kprobes/p_sys_execve_bcc_2566 sys_execve > > === > > > > Right. Libbpf only supports a newer and safer way to attach to > kprobes. For your experiments, try to stick to tracepoints and you'll > have a better time. > > But it's another thing I've been meaning to add to libbpf for > supporting older kernels. I even have code written to do legacy kprobe > attachment, just need to find time to send a patch to add it as a > fallback for kernels that don't support new kprobe interface. > [VAMSI-3] I think that will be very helpful, as there are bound to be hosts with older kernels for the foreseeable future. Good to know it's being considered. > > > I am guessing this is a backward compatibility issue?. I will try to > > look at an earlier version of libbpf to see how this was handled. > > it was never supported by libbpf. BCC support both old and new APIs, > but old APIs is more dangerous, as it's easy to leave attached kprobe > BPF program active in the kernel unintentionally. E.g., if the process > crashes. But with bpf_link and its expected use of > bpf_link__destroy(), we can support the legacy API (it won't be any > safer, but still). > [VAMSI-3] Will use tracepoints to validate the process as you suggested. I will then try to see what I can do to use this on kprobe/kretprobe's. Will also check how BCC does this. I will start a new thread if needed. Again appreciate the inputs and direction very much. Best Regards Vamsi. > > Meanwhile, if you have further comments they are appreciated. > > > > And again I can't thank you enough for how helpful you have been and > > your time. Cheers!. > > > > > > > > > > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko > > <andrii.nakryiko@gmail.com> wrote: > > > > > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > First of all thank you very much for your quick response. And helpful pointers. > > > > It seems like you also think what I am attempting to do should work. > > > > > > > > Please see inline [VAMSI-2]. > > > > > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > > > > <andrii.nakryiko@gmail.com> wrote: > > > > > > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > > > > CO-RE needs a supported kernel version and be compiled with > > > > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > > > > > > > I also understand there are three pieces to enable CO-RE > > > > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > > > > > > > BTF is not really a symbol table, rather a type information. Like > > > > > simpler and more compact DWARF. > > > > > > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > > > > these 3 steps are done the resulting byte code is no different from > > > > > > non-CO-RE byte code. > > > > > > > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > > > > and guide me if I am stating something incorrectly. > > > > > > > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > > > > generating and exposing the BTF file information on > > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > > > > `libbpf` can find the BTF information at a standard location?. > > > > > > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > > > > try to search for vmlinux image (and BTF info within it) in a few > > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > > > > even expose /sys/kernel/btf/vmlinux. > > > > > > > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > > > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > > > > mechanism be used so that it works on all kernels whether they support > > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > > > > (a) Extract BTF generation process outside of the kernel build. > > > > > > Use this to generate the equivalent BTF file for it. > > > > > > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > > > > info, but it's also possible to just embed BTF manually with a direct > > > > > invocation of pahole -J, see [1] on how it's done for > > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > > > > matter the version, and it will work with CO-RE relocations. > > > > > > > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > > > > > > > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > > > > `gen_btf` from the > > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > > > > file for the > > > > 4.14.0 kernel. > > > > > > > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > > > > standard locations but also at a user specified location. The BTF file > > > > > > generated in (a) can be presented here. > > > > > > > > > > You can already do that, actually, though it's not very obvious. You > > > > > can specify (or override) kernel BTF location by using > > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > > > > BTF location (see [2]). I've been meaning to add it instead to a > > > > > bpf_object_open_opts, btw, to make its use possible with a BPF > > > > > skeleton. Also keep in mind that currently libbpf expects that custom > > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > > > > we can improve that, of course. > > > > > > > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > > > > > > > [VAMSI-2] I took a look at this and what you suggested above does not > > > > work as is. > > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the > > > > `target_btf_path` attribute. > > > > > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but > > > since then added a bunch more use cases. So some libbpf changes are > > > needed to make this work. But it should still work for CO-RE to have a > > > custom BTF. > > > > > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom > > > BTF as the real kernel BTF, because that will never work for > > > fentry/fexit, struct_ops, etc. I think it is better to teach > > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF > > > if we need it only for CO-RE relocations *and* we have it overloaded > > > with target_btf_path > > > > > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > > > > you suggested enabling > > > > use from the BPF skeleton seems useful and I can possibly help with that. > > > > > > Yeah, adding something like core_btf_path option to > > > bpf_object_open_opts would go nicely with this change. > > > > > > > > > > > For now, just for proof of concept I modified the search options in > > > > `libbpf_find_kernel_btf` to > > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > > > > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 > > > > > > so here you successfully loaded custom BTF, which is good. > > > > > > > libbpf: Kernel doesn't support BTF, skipping uploading it. > > > > > > this just means that your BPF object's BTF won't be loaded into the > > > kernel. That's no big deal, ignore this. > > > > > > > libbpf: kernel doesn't support global data > > > > > > But this means that your BPF programs rely on global variables, which > > > are not supported by the kernel. So you need to change the code to not > > > use global variables to make this work on very old kernels. > > > > > > > > > > libbpf: failed to load object 'tcpconnect_bpf' > > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > > > > failed to load BPF object: -95 > > > > > > This is probably OPNOTSUPP from the global data above > > > > > > > > > > > This is the reason I had posted on the mailer. If the CO-RE executable > > > > has relocations > > > > resolved by the time of the BPF load. Why do we need to check for > > > > kernel support?. Also, > > > > does this mean what I am attempting to do will not work?. > > > > > > > > > > it will work with minimal libbpf logic changes. Nothing in principle > > > prevents this. > > > > > > > > > > Best Regards. And again thanks a lot for your precious time. > > > > - Vamsi. > > > > > > > > > > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > > > > kernel versions as well. I tried to make the above changes and tried > > > > > > against a 4.14 kernel and it did not work. Either I am not doing > > > > > > something right or my assumptions are wrong. > > > > > > > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > > > > in the right direction. > > > > > > > > > > > > Regards > > > > > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-08 0:16 ` Vamsi Kodavanty @ 2021-01-08 1:31 ` Vamsi Kodavanty 2021-03-03 18:14 ` Rafael David Tinoco 0 siblings, 1 reply; 20+ messages in thread From: Vamsi Kodavanty @ 2021-01-08 1:31 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf On Thu, Jan 7, 2021 at 4:16 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > On Thu, Jan 7, 2021 at 3:32 PM Andrii Nakryiko > <andrii.nakryiko@gmail.com> wrote: > > > > On Thu, Jan 7, 2021 at 2:45 PM Vamsi Kodavanty <vamsi@araalinetworks.com> wrote: > > > > > > Andrii, > > > Thank you so much for the quick response. You were right. I was > > > trying to CO-RE the `tcpconnect.py` program. It had some `.rodata` > > > which I removed. Now the program goes much further. It now finishes > > > the relocations successfully. But, fails at the next step > > > `bpf_object__load_progs`. > > > > It's discouraged to use top posting on the mailing list. So for the > > future please reply inline. > > > > [VAMSI-3] Apologies. Will be mindful next time on. > > > > > > > > > === > > > libbpf: prog 'tcp_v6_connect_ret': relo #3: patched insn #50 > > > (ALU/ALU64) imm 56 -> 56 > > > libbpf: failed to open '/sys/bus/event_source/devices/kprobe/type': No > > > such file or directory > > > libbpf: failed to determine kprobe perf type: No such file or directory > > > libbpf: prog 'tcp_v4_connect': failed to create kprobe > > > 'tcp_v4_connect' perf event: No such file or directory > > > libbpf: failed to auto-attach program 'tcp_v4_connect': -2 > > > === > > > > > > This host has a 4.14 kernel AmazonLinux2 and does not have the above > > > file. It instead has this > > > > > > === > > > $ sudo cat /sys/kernel/debug/tracing/kprobe_events > > > p:kprobes/p_sys_execve_bcc_2566 sys_execve > > > === > > > > > > > Right. Libbpf only supports a newer and safer way to attach to > > kprobes. For your experiments, try to stick to tracepoints and you'll > > have a better time. > > > > But it's another thing I've been meaning to add to libbpf for > > supporting older kernels. I even have code written to do legacy kprobe > > attachment, just need to find time to send a patch to add it as a > > fallback for kernels that don't support new kprobe interface. > > > > [VAMSI-3] I think that will be very helpful, as there are bound to be hosts > with older kernels for the foreseeable future. Good to know it's being > considered. > > > > > > I am guessing this is a backward compatibility issue?. I will try to > > > look at an earlier version of libbpf to see how this was handled. > > > > it was never supported by libbpf. BCC support both old and new APIs, > > but old APIs is more dangerous, as it's easy to leave attached kprobe > > BPF program active in the kernel unintentionally. E.g., if the process > > crashes. But with bpf_link and its expected use of > > bpf_link__destroy(), we can support the legacy API (it won't be any > > safer, but still). > > > > [VAMSI-3] Will use tracepoints to validate the process as you suggested. I > will then try to see what I can do to use this on kprobe/kretprobe's. Will also > check how BCC does this. I will start a new thread if needed. > [VAMSI-4] Just to round up this thread. The tracepoints did work out of the box. I tried the `execsnoop` tool and it worked fine. Thank you again. Vamsi. > Again appreciate the inputs and direction very much. > > Best Regards > Vamsi. > > > > Meanwhile, if you have further comments they are appreciated. > > > > > > And again I can't thank you enough for how helpful you have been and > > > your time. Cheers!. > > > > > > > > > > > > > > > On Thu, Jan 7, 2021 at 10:52 AM Andrii Nakryiko > > > <andrii.nakryiko@gmail.com> wrote: > > > > > > > > On Thu, Jan 7, 2021 at 10:12 AM Vamsi Kodavanty > > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > > > First of all thank you very much for your quick response. And helpful pointers. > > > > > It seems like you also think what I am attempting to do should work. > > > > > > > > > > Please see inline [VAMSI-2]. > > > > > > > > > > On Wed, Jan 6, 2021 at 3:55 PM Andrii Nakryiko > > > > > <andrii.nakryiko@gmail.com> wrote: > > > > > > > > > > > > On Wed, Jan 6, 2021 at 10:04 AM Vamsi Kodavanty > > > > > > <vamsi@araalinetworks.com> wrote: > > > > > > > > > > > > > > Had a few questions on CO-RE dependencies and usage. From what I read > > > > > > > CO-RE needs a supported kernel version and be compiled with > > > > > > > `CONFIG_DEBUG_INFO_BTF=y`. > > > > > > > > > > > > > > I also understand there are three pieces to enable CO-RE > > > > > > > functionality. (1) The BTF format. For efficient/compressed kernel > > > > > > > symbol table. (2) clang changes to emit the BTF relocations. (3) > > > > > > > > > > > > BTF is not really a symbol table, rather a type information. Like > > > > > > simpler and more compact DWARF. > > > > > > > > > > > > > `libbpf` changes to locate a BTF file and fix-up relocations. Once > > > > > > > these 3 steps are done the resulting byte code is no different from > > > > > > > non-CO-RE byte code. > > > > > > > > > > > > > > Given this I am hoping the knowledgeable folks on this mailer correct > > > > > > > and guide me if I am stating something incorrectly. > > > > > > > > > > > > > > (1) Is the kernel support requirement ONLY for the purposes of > > > > > > > generating and exposing the BTF file information on > > > > > > > `/sys/kernel/btf/vmlinux`? So that the eBPF CO-RE applications > > > > > > > `libbpf` can find the BTF information at a standard location?. > > > > > > > > > > > > /sys/kernel/btf/vmlinux is a standardized place, but libbpf will also > > > > > > try to search for vmlinux image (and BTF info within it) in a few > > > > > > standard locations, see [0]. Early versions of in-kernel BTF didn't > > > > > > even expose /sys/kernel/btf/vmlinux. > > > > > > > > > > > > [0] https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4580 > > > > > > > > > > > > > > > > > > > > (2) If the answer to the above question is YES. Could the below > > > > > > > mechanism be used so that it works on all kernels whether they support > > > > > > > the `CONFIG_DEBUG_INFO_BTF` flag or not?. > > > > > > > (a) Extract BTF generation process outside of the kernel build. > > > > > > > Use this to generate the equivalent BTF file for it. > > > > > > > > > > > > Yes, CONFIG_DEBUG_INFO_BTF=y is the most convenient way to add BTF > > > > > > info, but it's also possible to just embed BTF manually with a direct > > > > > > invocation of pahole -J, see [1] on how it's done for > > > > > > CONFIG_DEBUG_INFO_BTF. You can do that for *any* kernel image, no > > > > > > matter the version, and it will work with CO-RE relocations. > > > > > > > > > > > > [1] https://github.com/torvalds/linux/blob/master/scripts/link-vmlinux.sh#L137-L170 > > > > > > > > > > > > > > > > [VAMSI-2] Yes, this is exactly what I did. I extracted out the > > > > > `gen_btf` from the > > > > > `link-vmlinux.sh` (which uses pahole -J) and used it to generate a BTF > > > > > file for the > > > > > 4.14.0 kernel. > > > > > > > > > > > > (b) Make changes to `libbpf` to look for BTF not only at the > > > > > > > standard locations but also at a user specified location. The BTF file > > > > > > > generated in (a) can be presented here. > > > > > > > > > > > > You can already do that, actually, though it's not very obvious. You > > > > > > can specify (or override) kernel BTF location by using > > > > > > bpf_object__load_xattr() and passing target_btf_path pointing to your > > > > > > BTF location (see [2]). I've been meaning to add it instead to a > > > > > > bpf_object_open_opts, btw, to make its use possible with a BPF > > > > > > skeleton. Also keep in mind that currently libbpf expects that custom > > > > > > BTF to be an ELF file with .BTF section, not just a raw BTF data. But > > > > > > we can improve that, of course. > > > > > > > > > > > > [2] https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L136-L141 > > > > > > > > > > [VAMSI-2] I took a look at this and what you suggested above does not > > > > > work as is. > > > > > Even if we used `bpf_object__load_xattr` with `target_btf_path`. It seems like > > > > > `bpf_object__load_vmlinux_btf` is not yet modified to use the > > > > > `target_btf_path` attribute. > > > > > > > > Ah, right. We used to need vmlinux BTF only for CO-RE relocations, but > > > > since then added a bunch more use cases. So some libbpf changes are > > > > needed to make this work. But it should still work for CO-RE to have a > > > > custom BTF. > > > > > > > > I'm not sure about making bpf_object__load_vmlinux_btf() load custom > > > > BTF as the real kernel BTF, because that will never work for > > > > fentry/fexit, struct_ops, etc. I think it is better to teach > > > > bpf_object__load_vmlinux_btf() to not attempt to load real kernel BTF > > > > if we need it only for CO-RE relocations *and* we have it overloaded > > > > with target_btf_path > > > > > > > > > Only, the `bpf_object__relocate` looks at the `target_btf_path`. As > > > > > you suggested enabling > > > > > use from the BPF skeleton seems useful and I can possibly help with that. > > > > > > > > Yeah, adding something like core_btf_path option to > > > > bpf_object_open_opts would go nicely with this change. > > > > > > > > > > > > > > For now, just for proof of concept I modified the search options in > > > > > `libbpf_find_kernel_btf` to > > > > > include my custom path. And on a 4.14 AmazonLinux2 VM I observe these failures. > > > > > > > > > > libbpf: loading kernel BTF '/home/ec2-user/vmlinux.btf': 0 > > > > > > > > so here you successfully loaded custom BTF, which is good. > > > > > > > > > libbpf: Kernel doesn't support BTF, skipping uploading it. > > > > > > > > this just means that your BPF object's BTF won't be loaded into the > > > > kernel. That's no big deal, ignore this. > > > > > > > > > libbpf: kernel doesn't support global data > > > > > > > > But this means that your BPF programs rely on global variables, which > > > > are not supported by the kernel. So you need to change the code to not > > > > use global variables to make this work on very old kernels. > > > > > > > > > > > > > libbpf: failed to load object 'tcpconnect_bpf' > > > > > libbpf: failed to load BPF skeleton 'tcpconnect_bpf': -95 > > > > > failed to load BPF object: -95 > > > > > > > > This is probably OPNOTSUPP from the global data above > > > > > > > > > > > > > > This is the reason I had posted on the mailer. If the CO-RE executable > > > > > has relocations > > > > > resolved by the time of the BPF load. Why do we need to check for > > > > > kernel support?. Also, > > > > > does this mean what I am attempting to do will not work?. > > > > > > > > > > > > > it will work with minimal libbpf logic changes. Nothing in principle > > > > prevents this. > > > > > > > > > > > > > Best Regards. And again thanks a lot for your precious time. > > > > > - Vamsi. > > > > > > > > > > > > > > > > > > > This should provide us a way to enable CO-RE functionality on older > > > > > > > kernel versions as well. I tried to make the above changes and tried > > > > > > > against a 4.14 kernel and it did not work. Either I am not doing > > > > > > > something right or my assumptions are wrong. > > > > > > > > > > > > > > Thanks in advance for your time. And I hope someone here can guide me > > > > > > > in the right direction. > > > > > > > > > > > > > > Regards > > > > > > > Vamsi. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-01-08 1:31 ` Vamsi Kodavanty @ 2021-03-03 18:14 ` Rafael David Tinoco 2021-03-04 7:05 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-03 18:14 UTC (permalink / raw) To: vamsi; +Cc: andrii.nakryiko, rafaeldtinoco, bpf > > From: Vamsi Kodavanty <vamsi@araalinetworks.com> > > Date: Thu, 7 Jan 2021 17:31:11 -0800 > > To: Andrii Nakryiko <andrii.nakryiko@gmail.com> > > Cc: bpf <bpf@vger.kernel.org> > > > > > > Right. Libbpf only supports a newer and safer way to attach to > > kprobes. For your experiments, try to stick to tracepoints and you'll > > have a better time. > > > > But it's another thing I've been meaning to add to libbpf for > > supporting older kernels. I even have code written to do legacy kprobe > > attachment, just need to find time to send a patch to add it as a > > fallback for kernels that don't support new kprobe interface. Initially I'd like to thank you *a lot* for this thread, it helped me creating: https://github.com/rafaeldtinoco/portablebpf showing up exactly what was discussed here AND I could run the same binary in v4.15 and v.5.8 kernels as long as BTF was generated with: https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch Specially the attach_kprobe_legacy() function: https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 I wanted to reply here in case others also face this. Only bad thing was kernel v4.15 missed global data support as showed in: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md But using perf event was good enough for an example. - rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-03 18:14 ` Rafael David Tinoco @ 2021-03-04 7:05 ` Andrii Nakryiko 2021-03-04 13:10 ` Arnaldo Carvalho de Melo 2021-03-05 6:32 ` Rafael David Tinoco 0 siblings, 2 replies; 20+ messages in thread From: Andrii Nakryiko @ 2021-03-04 7:05 UTC (permalink / raw) To: Rafael David Tinoco, Arnaldo Carvalho de Melo Cc: Vamsi Kodavanty, rafaeldtinoco, bpf On Wed, Mar 3, 2021 at 10:15 AM Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > > > From: Vamsi Kodavanty <vamsi@araalinetworks.com> > > > Date: Thu, 7 Jan 2021 17:31:11 -0800 > > > To: Andrii Nakryiko <andrii.nakryiko@gmail.com> > > > Cc: bpf <bpf@vger.kernel.org> > > > > > > > > > Right. Libbpf only supports a newer and safer way to attach to > > > kprobes. For your experiments, try to stick to tracepoints and you'll > > > have a better time. > > > > > > But it's another thing I've been meaning to add to libbpf for > > > supporting older kernels. I even have code written to do legacy kprobe > > > attachment, just need to find time to send a patch to add it as a > > > fallback for kernels that don't support new kprobe interface. > > Initially I'd like to thank you *a lot* for this thread, it helped me > creating: > > https://github.com/rafaeldtinoco/portablebpf > > showing up exactly what was discussed here AND I could run the same > binary in v4.15 and v.5.8 kernels as long as BTF was generated with: > > https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch I was wondering if it might be useful to have a script that would use pahole to do DWARF to BTF conversion for existing vmlinux image (e.g., from /boot/vmlinux-$(uname -r)), assuming DWARF is in that vmlinux (or could be found somewhere nearby), and then would spit out only .BTF contents as a binary file, which can be passed to libbpf on bpf_object__open(). That seems useful and there have been at least a few cases where people tried to use CO-RE on old kernels pre-CONFIG_DEBUG_INFO_BTF, but were always confused by how to get that BTF data. [cc Arnaldo] It would also simplify things a bunch if pahole had an option to emit .BTF into a separate non-ELF file, instead of modifying vmlinux in-place. WDYT? > > Specially the attach_kprobe_legacy() function: > > https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 > > I wanted to reply here in case others also face this. Great, glad it worked out. It would be great if you could contribute legacy kprobe support for libbpf as a proper patch, since it probably would be useful for a bunch of other people stuck with old kernels. > > Only bad thing was kernel v4.15 missed global data support as showed in: > > https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md > > But using perf event was good enough for an example. > > - rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-04 7:05 ` Andrii Nakryiko @ 2021-03-04 13:10 ` Arnaldo Carvalho de Melo 2021-03-05 6:32 ` Rafael David Tinoco 1 sibling, 0 replies; 20+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-03-04 13:10 UTC (permalink / raw) To: Andrii Nakryiko Cc: Rafael David Tinoco, Arnaldo Carvalho de Melo, Vamsi Kodavanty, rafaeldtinoco, bpf Em Wed, Mar 03, 2021 at 11:05:44PM -0800, Andrii Nakryiko escreveu: > On Wed, Mar 3, 2021 at 10:15 AM Rafael David Tinoco > <rafaeldtinoco@ubuntu.com> wrote: > > > > > > From: Vamsi Kodavanty <vamsi@araalinetworks.com> > > > > Date: Thu, 7 Jan 2021 17:31:11 -0800 > > > > To: Andrii Nakryiko <andrii.nakryiko@gmail.com> > > > > Cc: bpf <bpf@vger.kernel.org> > > > > > > > > > > > > Right. Libbpf only supports a newer and safer way to attach to > > > > kprobes. For your experiments, try to stick to tracepoints and you'll > > > > have a better time. > > > > > > > > But it's another thing I've been meaning to add to libbpf for > > > > supporting older kernels. I even have code written to do legacy kprobe > > > > attachment, just need to find time to send a patch to add it as a > > > > fallback for kernels that don't support new kprobe interface. > > > > Initially I'd like to thank you *a lot* for this thread, it helped me > > creating: > > > > https://github.com/rafaeldtinoco/portablebpf > > > > showing up exactly what was discussed here AND I could run the same > > binary in v4.15 and v.5.8 kernels as long as BTF was generated with: > > > > https://github.com/rafaeldtinoco/portablebpf/blob/master/patches/link-vmlinux.sh.patch > > I was wondering if it might be useful to have a script that would use > pahole to do DWARF to BTF conversion for existing vmlinux image (e.g., > from /boot/vmlinux-$(uname -r)), assuming DWARF is in that vmlinux (or > could be found somewhere nearby), and then would spit out only .BTF > contents as a binary file, which can be passed to libbpf on > bpf_object__open(). That seems useful and there have been at least a > few cases where people tried to use CO-RE on old kernels > pre-CONFIG_DEBUG_INFO_BTF, but were always confused by how to get that > BTF data. > > [cc Arnaldo] > It would also simplify things a bunch if pahole had an option to emit > .BTF into a separate non-ELF file, instead of modifying vmlinux > in-place. WDYT? Sure, that is a nice addition, makes it more flexible to cover this usecase. - Arnaldo > > > > Specially the attach_kprobe_legacy() function: > > > > https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 > > > > I wanted to reply here in case others also face this. > > Great, glad it worked out. It would be great if you could contribute > legacy kprobe support for libbpf as a proper patch, since it probably > would be useful for a bunch of other people stuck with old kernels. > > > > > Only bad thing was kernel v4.15 missed global data support as showed in: > > > > https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md > > > > But using perf event was good enough for an example. > > > > - rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-04 7:05 ` Andrii Nakryiko 2021-03-04 13:10 ` Arnaldo Carvalho de Melo @ 2021-03-05 6:32 ` Rafael David Tinoco [not found] ` <67E3C788-2835-4793-8A9C-51C5D807C294@ubuntu.com> 1 sibling, 1 reply; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-05 6:32 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf >> Specially the attach_kprobe_legacy() function: >> >> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 >> >> I wanted to reply here in case others also face this. > > Great, glad it worked out. It would be great if you could contribute > legacy kprobe support for libbpf as a proper patch, since it probably > would be useful for a bunch of other people stuck with old kernels. Definitely! Will suggest a patch for this soon. Thanks for the feedback. -rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <67E3C788-2835-4793-8A9C-51C5D807C294@ubuntu.com>]
* Fwd: [BPF CO-RE clarification] Use CO-RE on older kernel versions. [not found] ` <67E3C788-2835-4793-8A9C-51C5D807C294@ubuntu.com> @ 2021-03-10 6:00 ` Rafael David Tinoco 2021-03-10 19:19 ` Andrii Nakryiko 1 sibling, 0 replies; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-10 6:00 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf [-- Attachment #1: Type: text/plain, Size: 2369 bytes --] > On 5 Mar 2021, at 03:32, Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > >>> Specially the attach_kprobe_legacy() function: >>> >>> https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 >>> >>> I wanted to reply here in case others also face this. >> >> Great, glad it worked out. It would be great if you could contribute >> legacy kprobe support for libbpf as a proper patch, since it probably >> would be useful for a bunch of other people stuck with old kernels. > (sending this again in proper format for the list) I’m sorry to come back to this but I’d like to clarify something, if you allow me. If I recompile old kernels (4.x.y) with the “scripts/link-vmlinux.sh" patch (setting $btf_vmlinux_bin_o and gen_btf()) I’m able to use "pahole -J" to generate the .BTF ELF section from a vmlinux file (out of the debugging package, for example) using its DWARF data. Using objcopy, I’m also able to extract only the .BTF ELF section from it and use the generated file (smaller) as a base BTF file for libbpf (since old kernels don’t have /sys/kernel/btf/vmlinux interface). So, in my case, with this, I can get an ~30MB ELF file (from a an almost 600MB vmlinux) with BTF data that can feed libbpf to do needed relocations for my BPF object. Execution works perfectly and I can have the same libbpf based code to run in a 4.15 and a 5.8 kernel, smooth. What is not entirely clear to me yet is … why can’t I use a “vmlinux” file from a previous compiled kernel (that has not been compiled with a changed link-vmlinux.sh file) and do the same: generate the BTF section from its DWARF data with pahole and use generated file (or the BTF section extract only) as input to libbpf. I mean, I can do, but it does not work… Assumption: it only works for the ones I build with patched link-vmlinux.sh (not the ones already built and provided as packages). The code execution output (debug=1 on libbpf) is at : https://pastebin.ubuntu.com/p/bx6tygY8p2/ The difference for a new 4.x.y kernel and the existing ones (older packaged kernels) is the vmlinux_link() function linking the BTF object file in each of the 3 tmp_kallsyms steps. Is there a way I can get the already existing kernels to work with only pahole DWARF to BTF conversion data ? Thank you! -rafaeldtinoco > [-- Attachment #2: Message signed with OpenPGP --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. [not found] ` <67E3C788-2835-4793-8A9C-51C5D807C294@ubuntu.com> 2021-03-10 6:00 ` Fwd: " Rafael David Tinoco @ 2021-03-10 19:19 ` Andrii Nakryiko 2021-03-10 22:45 ` Rafael David Tinoco 1 sibling, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-03-10 19:19 UTC (permalink / raw) To: Rafael David Tinoco; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf On Tue, Mar 9, 2021 at 9:58 PM Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > > > On 5 Mar 2021, at 03:32, Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > > Specially the attach_kprobe_legacy() function: > > https://github.com/rafaeldtinoco/portablebpf/blob/master/mine.c#L31 > > I wanted to reply here in case others also face this. > > > Great, glad it worked out. It would be great if you could contribute > legacy kprobe support for libbpf as a proper patch, since it probably > would be useful for a bunch of other people stuck with old kernels. > > > > I’m sorry to come back to this but I’d like to clarify something, if you allow me. > > If I recompile old kernels (4.x.y) with the “scripts/link-vmlinux.sh" patch (setting $btf_vmlinux_bin_o and gen_btf()) I’m able to use "pahole -J" to generate the .BTF ELF section from a vmlinux file (out of the debugging package, for example) using its DWARF data. > > Using objcopy, I’m also able to extract only the .BTF ELF section from it and use the generated file (smaller) as a base BTF file for libbpf (since old kernels don’t have /sys/kernel/btf/vmlinux interface). > > So, in my case, with this, I can get an ~30MB ELF file (from a an almost 600MB vmlinux) with BTF data that can feed libbpf to do needed relocations for my BPF object. Execution works perfectly and I can have the same libbpf based code to run in a 4.15 and a 5.8 kernel, smooth. Surprised that .BTF is so big at 30MB. It depends on kernel config you are using, but that's still few times bigger than what I normally see. Otherwise, yeah, that's how it should work (except see the patch that adds core_btf_path and discussion around it). > > What is not entirely clear to me yet is … why can’t I use a “vmlinux” file from a previous compiled kernel (that has not been compiled with a changed link-vmlinux.sh file) and do the same: generate the BTF section from its DWARF data with pahole and use generated file (or the BTF section extract only) as input to libbpf. > > I mean, I can do, but it does not work… Assumption: it only works for the ones I build with patched link-vmlinux.sh (not the ones already built and provided as packages). The code execution output (debug=1 on libbpf) is at : https://pastebin.ubuntu.com/p/bx6tygY8p2/ > From what I see all the CO-RE relocations applied successfully (even though all the offsets stayed the same, so presumably you compiled your BPF program with vmlinux.h from the exact same kernel you are running it on?). Are you sure that vmlinux image you are providing corresponds to the actual kernel you are running on? I'd start by comparing libbpf logs for vmlinux you get with modified link-vmlinux.sh script and with just explicit pahole -J. If all the CO-RE parts are identical, the problem is somewhere else most probably. I see "libbpf: load bpf program failed: Invalid argument" in that log, which means that CO-RE was done and successful and only when trying to load your BPF program into the kernel it failed. > The difference for a new 4.x.y kernel and the existing ones (older packaged kernels) is the vmlinux_link() function linking the BTF object file in each of the 3 tmp_kallsyms steps. > > Is there a way I can get the already existing kernels to work with only pahole DWARF to BTF conversion data ? Yes and you've found it, I think. There is no difference to libbpf and .BTF itself whether it's run in link-vmlinux.sh or with explicit pahole -J. Look for the problem somewhere else. > > Thank you! > > -rafaeldtinoco > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-10 19:19 ` Andrii Nakryiko @ 2021-03-10 22:45 ` Rafael David Tinoco 2021-03-12 18:36 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-10 22:45 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf > From what I see all the CO-RE relocations applied successfully (even > though all the offsets stayed the same, so presumably you compiled > your BPF program with vmlinux.h from the exact same kernel you are > running it on?). Are you sure that vmlinux image you are providing > corresponds to the actual kernel you are running on? Yep, I have created the following: https://pastebin.ubuntu.com/p/h58YyNr4HR/ to make this easier. It is a 4.15.0-1080 kernel and a 4.15.18+. They are pretty close despite the versioning (second was generated with make deb-dpkg). I’m using same .config file for both, only difference is that the 4.15.18+ was compiled with the changed link-vmlinux.sh file. The /usr/lib/debug/boot/vmlinux files are generated by the same build and I have tried 2 or 3 of the existing packaged kernels. The only thing I did was “pahole -J” to /usr/lib/debug/boot/vmlinux-XXX files (adding the BTF section to them). Running same binary in a 5.8.0-43 kernel works out-of-the-box: https://pastebin.ubuntu.com/p/VKTjMcp6Xs/ > > I'd start by comparing libbpf logs for vmlinux you get with modified > link-vmlinux.sh script and with just explicit pahole -J. If all the > CO-RE parts are identical, the problem is somewhere else most > probably. The difference between (1) and (2) from the paste (error and success): libbpf: CO-RE relocating [0] struct task_struct: found target candidate [17361] struct task_struct in [vmlinux] libbpf: CO-RE relocating [0] struct task_struct: found target candidate [17360] struct task_struct in [vmlinux] libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17361] struct task_struct.comm (0:90 @ offset 2640) libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17360] struct task_struct.comm (0:90 @ offset 2640) Code is at: https://github.com/rafaeldtinoco/portablebpf and it is not much different than any other libbpf example. thanks again for verifying this! -rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-10 22:45 ` Rafael David Tinoco @ 2021-03-12 18:36 ` Andrii Nakryiko 2021-03-17 4:39 ` Rafael David Tinoco 0 siblings, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-03-12 18:36 UTC (permalink / raw) To: Rafael David Tinoco; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > > > From what I see all the CO-RE relocations applied successfully (even > > though all the offsets stayed the same, so presumably you compiled > > your BPF program with vmlinux.h from the exact same kernel you are > > running it on?). Are you sure that vmlinux image you are providing > > corresponds to the actual kernel you are running on? > > Yep, I have created the following: > > https://pastebin.ubuntu.com/p/h58YyNr4HR/ > Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux is some way (but then why would kernel start and only fail to load/verify your BPF program?). It's also clear that CO-RE is doing exactly the same instruction patching, so shouldn't be some invalid CO-RE relocation applied, I think. So no idea and not sure how to investigate this. But I think I'd never do `pahole -J` on actual vmlinux image you are going to run. It's much safer and more convenient to make a copy, generate .BTF and then extract just .BTF section into a small binary, which can be provided separately. > to make this easier. > > It is a 4.15.0-1080 kernel and a 4.15.18+. They are pretty close Also this. Seems like it is two different kernels still, however small the difference between them is. Have you tried building *exactly* the same kernel with exactly the same config, but just with pahole -J during the build on vmlinux.o (which is later linked into a final vmlinux) vs running pahole -J on final vmlinux after the build. Notice that two approach differ in terms of which object file is being modified. With link-vmlinux.sh, kernel linker scripts dictate final layout, while if you run `pahole -J` on final vmlinux, it can modify the layout and potentially break something sensitive. pahole is using llvm-objcopy internally, and llvm-objcopy isn't a very trivial tool. > despite the versioning (second was generated with make deb-dpkg). > > I’m using same .config file for both, only difference is that the > 4.15.18+ was compiled with the changed link-vmlinux.sh file. > > The /usr/lib/debug/boot/vmlinux files are generated by the same > build and I have tried 2 or 3 of the existing packaged kernels. The > only thing I did was “pahole -J” to /usr/lib/debug/boot/vmlinux-XXX > files (adding the BTF section to them). > > Running same binary in a 5.8.0-43 kernel works out-of-the-box: > > https://pastebin.ubuntu.com/p/VKTjMcp6Xs/ > > > > > I'd start by comparing libbpf logs for vmlinux you get with modified > > link-vmlinux.sh script and with just explicit pahole -J. If all the > > CO-RE parts are identical, the problem is somewhere else most > > probably. > > The difference between (1) and (2) from the paste (error and success): > > libbpf: CO-RE relocating [0] struct task_struct: found target > candidate [17361] struct task_struct in [vmlinux] > libbpf: CO-RE relocating [0] struct task_struct: found target > candidate [17360] struct task_struct in [vmlinux] > > libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17361] > struct task_struct.comm (0:90 @ offset 2640) > libbpf: prog 'tcp_connect': relo #0: matching candidate #1 [17360] > struct task_struct.comm (0:90 @ offset 2640) > > Code is at: > > https://github.com/rafaeldtinoco/portablebpf > > and it is not much different than any other libbpf example. > > thanks again for verifying this! > > -rafaeldtinoco > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-12 18:36 ` Andrii Nakryiko @ 2021-03-17 4:39 ` Rafael David Tinoco 2021-03-17 14:31 ` Rafael David Tinoco 0 siblings, 1 reply; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-17 4:39 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > On Wed, Mar 10, 2021 at 2:45 PM Rafael David Tinoco > <rafaeldtinoco@ubuntu.com> wrote: >>> From what I see all the CO-RE relocations applied successfully (even >>> though all the offsets stayed the same, so presumably you compiled >>> your BPF program with vmlinux.h from the exact same kernel you are >>> running it on?). Are you sure that vmlinux image you are providing >>> corresponds to the actual kernel you are running on? >> >> Yep, I have created the following: >> >> https://pastebin.ubuntu.com/p/h58YyNr4HR/ > > Ok, I have no idea, tbh. Maybe `pahole -j` is subtly modifying vmlinux > is some way (but then why would kernel start and only fail to > load/verify your BPF program?). It's also clear that CO-RE is doing > exactly the same instruction patching, so shouldn't be some invalid > CO-RE relocation applied, I think. So no idea and not sure how to > investigate this. > > But I think I'd never do `pahole -J` on actual vmlinux image you are > going to run. It's much safer and more convenient to make a copy, > generate .BTF and then extract just .BTF section into a small binary, > which can be provided separately. > FOUND the cause of the issue... Compiling the EXACT same kernel with different building scripts (deb-pkg vs debian/rules in my case) resulted in a very similar kernel (same .config, same autoconf.h, no visible changes). Unfortunately one of the kernels worked fined (reading the BTF extracted section OR same section within a vmlinux entire file).. but the other did not. Instrumenting this bad 4.15 kernel (out of debian/rules build) I found that the following sanity checks took place in kernel: bpf_check(): if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level || !log->ubuf) and a simple change in libbpf (mitigation of course): attr.log_buf = 0; attr.log_level = 0; attr.log_size = 0; before fd = sys_bpf_prog_load(&attr, sizeof(attr)); made the code to also run in this second kernel (built with the debian/rules building scripts): https://pastebin.ubuntu.com/p/scJDM3D9Zr/ Now I still have to discover why log_buf is miss-behaving in this kernel being built with debian/rules* scripts and not with the vanilla building scripts (despite config file being the same). (FYIO, documenting it here for others also)... ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-17 4:39 ` Rafael David Tinoco @ 2021-03-17 14:31 ` Rafael David Tinoco 2021-03-19 4:36 ` Andrii Nakryiko 0 siblings, 1 reply; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-17 14:31 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf >> >>>> From what I see all the CO-RE relocations applied successfully (even >>>> though all the offsets stayed the same, so presumably you compiled >>>> your BPF program with vmlinux.h from the exact same kernel you are >>>> running it on?). Are you sure that vmlinux image you are providing >>>> corresponds to the actual kernel you are running on? > > FOUND the cause of the issue… > ... > > bpf_check(): > > if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level > || !log->ubuf) > > and a simple change in libbpf (mitigation of course): > > attr.log_buf = 0; > attr.log_level = 0; > attr.log_size = 0; > > before > > fd = sys_bpf_prog_load(&attr, sizeof(attr)); With instrumented kernel… no changes to libbpf or code: attr.log_buf = (nil) attr.log_level = 0 attr.log_size = 0 load_attr.log_buf = 0x7fd1c0a92010 load_attr.log_level = 0 load_attr.log_size = 16777215 libbpf: load bpf program failed: Invalid argument libbpf: failed to load program 'ip_set_create' libbpf: failed to load object 'mine_bpf' libbpf: failed to load BPF skeleton 'mine_bpf': -22 failed to load BPF object: -22 [ 27.857858] MINE: BPF_PROG_TYPE_KPROBE KERNEL VERSION ISSUE [ 27.857993] MINE: LINUX_VERSION_CODE: 266002 [ 27.858131] MINE: YOUR VERSION: 265984 2 problems here: 0) attr.kern_version - looks like for some reason attr.kern_version is different from the one running - bypassing kernel BPF_KPROBE version check, I get: 1) load_attr has log_buf and log_size but not log_level for some reason. - this triggers an issue in bpf_check() within kernel IF I bypass the BPF_KPROBE version check (kerne 4.15). NOW, If I hard-code attr.kern_version in bpf.c to: attr.kern_version = 266002; fd = sys_bpf_prog_load(&attr, sizeof(attr)); then attr.log_buf = (nil) attr.log_level = 0 attr.log_size = 0 load_attr.log_buf = (nil) load_attr.log_level = 0 load_attr.log_size = 0 Tracing... Hit Ctrl-C to end. I don’t have the 2 problems, as log_level is zeroed, together with log_buf and log_size. Any clue on why this happens ? Thank you! -rafaeldtinoco ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-17 14:31 ` Rafael David Tinoco @ 2021-03-19 4:36 ` Andrii Nakryiko 2021-03-19 4:42 ` Rafael David Tinoco 0 siblings, 1 reply; 20+ messages in thread From: Andrii Nakryiko @ 2021-03-19 4:36 UTC (permalink / raw) To: Rafael David Tinoco; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf On Wed, Mar 17, 2021 at 7:31 AM Rafael David Tinoco <rafaeldtinoco@ubuntu.com> wrote: > > > >> > >>>> From what I see all the CO-RE relocations applied successfully (even > >>>> though all the offsets stayed the same, so presumably you compiled > >>>> your BPF program with vmlinux.h from the exact same kernel you are > >>>> running it on?). Are you sure that vmlinux image you are providing > >>>> corresponds to the actual kernel you are running on? > > > > FOUND the cause of the issue… > > > > ... > > > > > bpf_check(): > > > > if (log->len_total < 128 || log->len_total > UINT_MAX >> 8 || !log->level > > || !log->ubuf) > > > > and a simple change in libbpf (mitigation of course): > > > > attr.log_buf = 0; > > attr.log_level = 0; > > attr.log_size = 0; > > > > before > > > > fd = sys_bpf_prog_load(&attr, sizeof(attr)); > > With instrumented kernel… no changes to libbpf or code: > > attr.log_buf = (nil) > attr.log_level = 0 > attr.log_size = 0 > load_attr.log_buf = 0x7fd1c0a92010 > load_attr.log_level = 0 > load_attr.log_size = 16777215 > libbpf: load bpf program failed: Invalid argument > libbpf: failed to load program 'ip_set_create' > libbpf: failed to load object 'mine_bpf' > libbpf: failed to load BPF skeleton 'mine_bpf': -22 > failed to load BPF object: -22 > > [ 27.857858] MINE: BPF_PROG_TYPE_KPROBE KERNEL VERSION ISSUE > [ 27.857993] MINE: LINUX_VERSION_CODE: 266002 > [ 27.858131] MINE: YOUR VERSION: 265984 > > 2 problems here: > > 0) attr.kern_version > > - looks like for some reason attr.kern_version is different from > the one running > > - bypassing kernel BPF_KPROBE version check, I get: > > 1) load_attr has log_buf and log_size but not log_level for some reason. > > - this triggers an issue in bpf_check() within kernel IF I bypass > the BPF_KPROBE version check (kerne 4.15). > > NOW, If I hard-code attr.kern_version in bpf.c to: > > attr.kern_version = 266002; > fd = sys_bpf_prog_load(&attr, sizeof(attr)); > > then > > attr.log_buf = (nil) > attr.log_level = 0 > attr.log_size = 0 > load_attr.log_buf = (nil) > load_attr.log_level = 0 > load_attr.log_size = 0 > Tracing... Hit Ctrl-C to end. > > I don’t have the 2 problems, as log_level is zeroed, together with > log_buf and log_size. > > Any clue on why this happens ? So seems like you figured out kernel_version check, right? And it seems like log_buf is not really a problem as well? Or it still causing issues? > > Thank you! > > -rafaeldtinoco > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [BPF CO-RE clarification] Use CO-RE on older kernel versions. 2021-03-19 4:36 ` Andrii Nakryiko @ 2021-03-19 4:42 ` Rafael David Tinoco 0 siblings, 0 replies; 20+ messages in thread From: Rafael David Tinoco @ 2021-03-19 4:42 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: Arnaldo Carvalho de Melo, Vamsi Kodavanty, bpf >> Any clue on why this happens ? > > So seems like you figured out kernel_version check, right? And it > seems like log_buf is not really a problem as well? Or it still > causing issues? > All good here Andrii. Working on the legacy kprobe and kern_version patches and will address anything else there. tks, cheers ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-03-19 4:44 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-06 18:02 [BPF CO-RE clarification] Use CO-RE on older kernel versions Vamsi Kodavanty 2021-01-06 23:55 ` Andrii Nakryiko 2021-01-07 18:12 ` Vamsi Kodavanty 2021-01-07 18:52 ` Andrii Nakryiko 2021-01-07 22:45 ` Vamsi Kodavanty 2021-01-07 23:32 ` Andrii Nakryiko 2021-01-08 0:16 ` Vamsi Kodavanty 2021-01-08 1:31 ` Vamsi Kodavanty 2021-03-03 18:14 ` Rafael David Tinoco 2021-03-04 7:05 ` Andrii Nakryiko 2021-03-04 13:10 ` Arnaldo Carvalho de Melo 2021-03-05 6:32 ` Rafael David Tinoco [not found] ` <67E3C788-2835-4793-8A9C-51C5D807C294@ubuntu.com> 2021-03-10 6:00 ` Fwd: " Rafael David Tinoco 2021-03-10 19:19 ` Andrii Nakryiko 2021-03-10 22:45 ` Rafael David Tinoco 2021-03-12 18:36 ` Andrii Nakryiko 2021-03-17 4:39 ` Rafael David Tinoco 2021-03-17 14:31 ` Rafael David Tinoco 2021-03-19 4:36 ` Andrii Nakryiko 2021-03-19 4:42 ` Rafael David Tinoco
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.