* Help using libbpf with kernel 4.14 @ 2020-09-25 23:56 Yaniv Agman 2020-09-28 5:50 ` Andrii Nakryiko 0 siblings, 1 reply; 13+ messages in thread From: Yaniv Agman @ 2020-09-25 23:56 UTC (permalink / raw) To: bpf Hello, I'm developing a tool which is now based on BCC, and would like to make the move to libbpf. I need the tool to support a minimal kernel version 4.14, which doesn't have CO-RE. I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, but both only deal with newer kernels, and I failed to change them to run with a 4.14 kernel. Although some of the bpf samples in the kernel source don't use CO-RE, they all use bpf_load.h, and have dependencies on the tools dir, which I would like to avoid. I would appreciate it if someone can help with a simple working example of using libbpf on 4.14 kernel, without having any dependencies. Specifically, I'm looking for an example makefile, and to know how to load my bpf code with libbpf. Thanks, Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-25 23:56 Help using libbpf with kernel 4.14 Yaniv Agman @ 2020-09-28 5:50 ` Andrii Nakryiko 2020-09-28 20:08 ` Yaniv Agman 0 siblings, 1 reply; 13+ messages in thread From: Andrii Nakryiko @ 2020-09-28 5:50 UTC (permalink / raw) To: Yaniv Agman; +Cc: bpf On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > Hello, > > I'm developing a tool which is now based on BCC, and would like to > make the move to libbpf. > I need the tool to support a minimal kernel version 4.14, which > doesn't have CO-RE. You don't need kernel itself to support CO-RE, you just need that kernel to have BTF in it. If the kernel is too old to have CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole -J <path-to-vmlinux-image>`, if that's at all an option for your setup. > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > but both only deal with newer kernels, and I failed to change them to > run with a 4.14 kernel. > > Although some of the bpf samples in the kernel source don't use CO-RE, > they all use bpf_load.h, > and have dependencies on the tools dir, which I would like to avoid. Depending on what exactly you are trying to achieve with your BPF application, you might not need BPF CO-RE, and using libbpf without CO-RE would be enough for your needs. This would be the case if you don't need to access any of the kernel data structures (e.g., all sort of networking BPF apps: TC programs, cgroup sock progs, XDP). But if you need to do anything tracing related (e.g., looking at kernel's task_struct or any other internal structure), then you have no choice and you either have to do on-the-target-host runtime compilation (BCC way) or relocations (libbpf + BPF CO-RE). This is because of changing memory layout of kernel structures. So, unless you can compile one specific version of your BPF code for a one specific version of the kernel, you need either BCC or BPF CO-RE. > > I would appreciate it if someone can help with a simple working > example of using libbpf on 4.14 kernel, without having any > dependencies. Specifically, I'm looking for an example makefile, and > to know how to load my bpf code with libbpf. libbpf-tools's Makefile would still work. Just drop dependency on vmlinux.h and include system headers directly, if necessary (and if you considered implications of kernel memory layout changes). > > Thanks, > Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-28 5:50 ` Andrii Nakryiko @ 2020-09-28 20:08 ` Yaniv Agman 2020-09-28 20:24 ` Andrii Nakryiko 0 siblings, 1 reply; 13+ messages in thread From: Yaniv Agman @ 2020-09-28 20:08 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko <andrii.nakryiko@gmail.com>: > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > Hello, > > > > I'm developing a tool which is now based on BCC, and would like to > > make the move to libbpf. > > I need the tool to support a minimal kernel version 4.14, which > > doesn't have CO-RE. > > You don't need kernel itself to support CO-RE, you just need that > kernel to have BTF in it. If the kernel is too old to have > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole > -J <path-to-vmlinux-image>`, if that's at all an option for your > setup. > Thanks, I didn't know that > > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > > but both only deal with newer kernels, and I failed to change them to > > run with a 4.14 kernel. > > > > Although some of the bpf samples in the kernel source don't use CO-RE, > > they all use bpf_load.h, > > and have dependencies on the tools dir, which I would like to avoid. > > Depending on what exactly you are trying to achieve with your BPF > application, you might not need BPF CO-RE, and using libbpf without > CO-RE would be enough for your needs. This would be the case if you > don't need to access any of the kernel data structures (e.g., all sort > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if > you need to do anything tracing related (e.g., looking at kernel's > task_struct or any other internal structure), then you have no choice > and you either have to do on-the-target-host runtime compilation (BCC > way) or relocations (libbpf + BPF CO-RE). This is because of changing > memory layout of kernel structures. > > So, unless you can compile one specific version of your BPF code for a > one specific version of the kernel, you need either BCC or BPF CO-RE. > I'm working on a tracing application (https://github.com/aquasecurity/tracee) which now uses bcc. We now require a minimal kernel version 4.14, and bcc, but eventually we would like to support CO-RE. I thought that we could do the move in two steps. First moving to libbpf and keeping the 4.14 minimal requirement, then adding CO-RE support in the future. In order to do that, I thought of changing bcc requirement to clang requirement, and compile the program once during installation on the target host. This way we get the added value of fast start time without the need to compile every time the program starts (like bcc does), plus having an easier move to CO-RE in the future. A problem that I encountered with kernel 4.14 and libbpf was that when using bpf_prog_load (If I remember correctly), it returned an error of invalid argument (-22). Doing a small investigation I saw that it happened when trying to create bpf maps with names. Indeed I saw that libbpf API changed between kernel 4.14 and 4.15 and the function bpf_create_map_node now takes map name as an argument. Is there a way to workaround this with kernel 4.14 and still use map names in userspace to refer to bpf maps with libbpf? > > > > I would appreciate it if someone can help with a simple working > > example of using libbpf on 4.14 kernel, without having any > > dependencies. Specifically, I'm looking for an example makefile, and > > to know how to load my bpf code with libbpf. > > libbpf-tools's Makefile would still work. Just drop dependency on > vmlinux.h and include system headers directly, if necessary (and if > you considered implications of kernel memory layout changes). > Thanks, I'll try that > > > > Thanks, > > Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-28 20:08 ` Yaniv Agman @ 2020-09-28 20:24 ` Andrii Nakryiko 2020-09-28 20:39 ` Yaniv Agman 2020-09-29 0:00 ` Yaniv Agman 0 siblings, 2 replies; 13+ messages in thread From: Andrii Nakryiko @ 2020-09-28 20:24 UTC (permalink / raw) To: Yaniv Agman; +Cc: bpf On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko > <andrii.nakryiko@gmail.com>: > > > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > Hello, > > > > > > I'm developing a tool which is now based on BCC, and would like to > > > make the move to libbpf. > > > I need the tool to support a minimal kernel version 4.14, which > > > doesn't have CO-RE. > > > > You don't need kernel itself to support CO-RE, you just need that > > kernel to have BTF in it. If the kernel is too old to have > > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole > > -J <path-to-vmlinux-image>`, if that's at all an option for your > > setup. > > > > Thanks, I didn't know that > > > > > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > > > but both only deal with newer kernels, and I failed to change them to > > > run with a 4.14 kernel. > > > > > > Although some of the bpf samples in the kernel source don't use CO-RE, > > > they all use bpf_load.h, > > > and have dependencies on the tools dir, which I would like to avoid. > > > > Depending on what exactly you are trying to achieve with your BPF > > application, you might not need BPF CO-RE, and using libbpf without > > CO-RE would be enough for your needs. This would be the case if you > > don't need to access any of the kernel data structures (e.g., all sort > > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if > > you need to do anything tracing related (e.g., looking at kernel's > > task_struct or any other internal structure), then you have no choice > > and you either have to do on-the-target-host runtime compilation (BCC > > way) or relocations (libbpf + BPF CO-RE). This is because of changing > > memory layout of kernel structures. > > > > So, unless you can compile one specific version of your BPF code for a > > one specific version of the kernel, you need either BCC or BPF CO-RE. > > > > I'm working on a tracing application > (https://github.com/aquasecurity/tracee) which now uses bcc. We now > require a minimal kernel version 4.14, and bcc, but eventually we > would like to support CO-RE. I thought that we could do the move in > two steps. First moving to libbpf and keeping the 4.14 minimal > requirement, then adding CO-RE support in the future. > In order to do that, I thought of changing bcc requirement to clang > requirement, and compile the program once during installation on the > target host. This way we get the added value of fast start time > without the need to compile every time the program starts (like bcc > does), plus having an easier move to CO-RE in the future. Right, pre-compiling on the target machine with host kernel headers should work. So just don't use any of CO-RE features (no CO-RE relocations, no vmlinux.h), and it should just work. > > A problem that I encountered with kernel 4.14 and libbpf was that when > using bpf_prog_load (If I remember correctly), it returned an error of > invalid argument (-22). Doing a small investigation I saw that it > happened when trying to create bpf maps with names. Indeed I saw that > libbpf API changed between kernel 4.14 and 4.15 and the function > bpf_create_map_node now takes map name as an argument. Is there a way > to workaround this with kernel 4.14 and still use map names in > userspace to refer to bpf maps with libbpf? So we do run a few simple tests loading BPF programs (using libbpf) on 4.9 kernel, so map name should definitely not be a problem at all (libbpf is smart about detecting what's not supported in kernel and omitting non-essential things). It might be because of bpf_prog_load itself, which was long deprecated and you shouldn't use it for real-world applications. Please either use BPF skeleton or bpf_object APIs. It should just work, but if it doesn't please report back. > > > > > > > I would appreciate it if someone can help with a simple working > > > example of using libbpf on 4.14 kernel, without having any > > > dependencies. Specifically, I'm looking for an example makefile, and > > > to know how to load my bpf code with libbpf. > > > > libbpf-tools's Makefile would still work. Just drop dependency on > > vmlinux.h and include system headers directly, if necessary (and if > > you considered implications of kernel memory layout changes). > > > > Thanks, I'll try that > > > > > > > Thanks, > > > Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-28 20:24 ` Andrii Nakryiko @ 2020-09-28 20:39 ` Yaniv Agman 2020-09-29 0:00 ` Yaniv Agman 1 sibling, 0 replies; 13+ messages in thread From: Yaniv Agman @ 2020-09-28 20:39 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת Andrii Nakryiko <andrii.nakryiko@gmail.com>: > > On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko > > <andrii.nakryiko@gmail.com>: > > > > > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > > > Hello, > > > > > > > > I'm developing a tool which is now based on BCC, and would like to > > > > make the move to libbpf. > > > > I need the tool to support a minimal kernel version 4.14, which > > > > doesn't have CO-RE. > > > > > > You don't need kernel itself to support CO-RE, you just need that > > > kernel to have BTF in it. If the kernel is too old to have > > > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole > > > -J <path-to-vmlinux-image>`, if that's at all an option for your > > > setup. > > > > > > > Thanks, I didn't know that > > > > > > > > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > > > > but both only deal with newer kernels, and I failed to change them to > > > > run with a 4.14 kernel. > > > > > > > > Although some of the bpf samples in the kernel source don't use CO-RE, > > > > they all use bpf_load.h, > > > > and have dependencies on the tools dir, which I would like to avoid. > > > > > > Depending on what exactly you are trying to achieve with your BPF > > > application, you might not need BPF CO-RE, and using libbpf without > > > CO-RE would be enough for your needs. This would be the case if you > > > don't need to access any of the kernel data structures (e.g., all sort > > > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if > > > you need to do anything tracing related (e.g., looking at kernel's > > > task_struct or any other internal structure), then you have no choice > > > and you either have to do on-the-target-host runtime compilation (BCC > > > way) or relocations (libbpf + BPF CO-RE). This is because of changing > > > memory layout of kernel structures. > > > > > > So, unless you can compile one specific version of your BPF code for a > > > one specific version of the kernel, you need either BCC or BPF CO-RE. > > > > > > > I'm working on a tracing application > > (https://github.com/aquasecurity/tracee) which now uses bcc. We now > > require a minimal kernel version 4.14, and bcc, but eventually we > > would like to support CO-RE. I thought that we could do the move in > > two steps. First moving to libbpf and keeping the 4.14 minimal > > requirement, then adding CO-RE support in the future. > > In order to do that, I thought of changing bcc requirement to clang > > requirement, and compile the program once during installation on the > > target host. This way we get the added value of fast start time > > without the need to compile every time the program starts (like bcc > > does), plus having an easier move to CO-RE in the future. > > Right, pre-compiling on the target machine with host kernel headers > should work. So just don't use any of CO-RE features (no CO-RE > relocations, no vmlinux.h), and it should just work. > > > > > A problem that I encountered with kernel 4.14 and libbpf was that when > > using bpf_prog_load (If I remember correctly), it returned an error of > > invalid argument (-22). Doing a small investigation I saw that it > > happened when trying to create bpf maps with names. Indeed I saw that > > libbpf API changed between kernel 4.14 and 4.15 and the function > > bpf_create_map_node now takes map name as an argument. Is there a way > > to workaround this with kernel 4.14 and still use map names in > > userspace to refer to bpf maps with libbpf? > > So we do run a few simple tests loading BPF programs (using libbpf) on > 4.9 kernel, so map name should definitely not be a problem at all > (libbpf is smart about detecting what's not supported in kernel and > omitting non-essential things). It might be because of bpf_prog_load > itself, which was long deprecated and you shouldn't use it for > real-world applications. Please either use BPF skeleton or bpf_object > APIs. It should just work, but if it doesn't please report back. > Thanks, I'll try that and report on any issues found > > > > > > > > > > I would appreciate it if someone can help with a simple working > > > > example of using libbpf on 4.14 kernel, without having any > > > > dependencies. Specifically, I'm looking for an example makefile, and > > > > to know how to load my bpf code with libbpf. > > > > > > libbpf-tools's Makefile would still work. Just drop dependency on > > > vmlinux.h and include system headers directly, if necessary (and if > > > you considered implications of kernel memory layout changes). > > > > > > > Thanks, I'll try that > > > > > > > > > > Thanks, > > > > Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-28 20:24 ` Andrii Nakryiko 2020-09-28 20:39 ` Yaniv Agman @ 2020-09-29 0:00 ` Yaniv Agman 2020-09-29 0:07 ` Yonghong Song 2020-09-29 1:28 ` Andrii Nakryiko 1 sibling, 2 replies; 13+ messages in thread From: Yaniv Agman @ 2020-09-29 0:00 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf Hi Andrii, I used BPF skeleton as you suggested, which did work with kernel 4.19 but not with 4.14. I used the exact same program, same environment, only changed the kernel version. The error message I get on 4.14: libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 libbpf: failed to determine kprobe perf type: No such file or directory libbpf: prog 'kprobe__do_sys_open': failed to create kprobe 'do_sys_open' perf event: No such file or directory libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 failed to attach BPF programs: No such file or directory As the program I made is small, I'm copying it here: =========================================== #include <linux/version.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #include <uapi/linux/bpf.h> #include <uapi/linux/ptrace.h> struct bpf_map_def SEC("maps") open_fds = { .type = BPF_MAP_TYPE_LRU_HASH, .key_size = sizeof(int), .value_size = sizeof(int), .max_entries = 1024, }; SEC("kprobe/do_sys_open") int BPF_KPROBE(kprobe__do_sys_open) { int err; u32 id = bpf_get_current_pid_tgid(); int dfd = PT_REGS_PARM1(ctx); if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) { char log[] = "bpf_map_update_elem %d\n"; bpf_trace_printk(log, sizeof(log), err); return 1; } return 0; } char LICENSE[] SEC("license") = "GPL"; ================================================== Can you think of a reason why this only happens on 4.14? Thanks, Yaniv בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת Andrii Nakryiko <andrii.nakryiko@gmail.com>: > > On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko > > <andrii.nakryiko@gmail.com>: > > > > > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > > > Hello, > > > > > > > > I'm developing a tool which is now based on BCC, and would like to > > > > make the move to libbpf. > > > > I need the tool to support a minimal kernel version 4.14, which > > > > doesn't have CO-RE. > > > > > > You don't need kernel itself to support CO-RE, you just need that > > > kernel to have BTF in it. If the kernel is too old to have > > > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole > > > -J <path-to-vmlinux-image>`, if that's at all an option for your > > > setup. > > > > > > > Thanks, I didn't know that > > > > > > > > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > > > > but both only deal with newer kernels, and I failed to change them to > > > > run with a 4.14 kernel. > > > > > > > > Although some of the bpf samples in the kernel source don't use CO-RE, > > > > they all use bpf_load.h, > > > > and have dependencies on the tools dir, which I would like to avoid. > > > > > > Depending on what exactly you are trying to achieve with your BPF > > > application, you might not need BPF CO-RE, and using libbpf without > > > CO-RE would be enough for your needs. This would be the case if you > > > don't need to access any of the kernel data structures (e.g., all sort > > > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if > > > you need to do anything tracing related (e.g., looking at kernel's > > > task_struct or any other internal structure), then you have no choice > > > and you either have to do on-the-target-host runtime compilation (BCC > > > way) or relocations (libbpf + BPF CO-RE). This is because of changing > > > memory layout of kernel structures. > > > > > > So, unless you can compile one specific version of your BPF code for a > > > one specific version of the kernel, you need either BCC or BPF CO-RE. > > > > > > > I'm working on a tracing application > > (https://github.com/aquasecurity/tracee) which now uses bcc. We now > > require a minimal kernel version 4.14, and bcc, but eventually we > > would like to support CO-RE. I thought that we could do the move in > > two steps. First moving to libbpf and keeping the 4.14 minimal > > requirement, then adding CO-RE support in the future. > > In order to do that, I thought of changing bcc requirement to clang > > requirement, and compile the program once during installation on the > > target host. This way we get the added value of fast start time > > without the need to compile every time the program starts (like bcc > > does), plus having an easier move to CO-RE in the future. > > Right, pre-compiling on the target machine with host kernel headers > should work. So just don't use any of CO-RE features (no CO-RE > relocations, no vmlinux.h), and it should just work. > > > > > A problem that I encountered with kernel 4.14 and libbpf was that when > > using bpf_prog_load (If I remember correctly), it returned an error of > > invalid argument (-22). Doing a small investigation I saw that it > > happened when trying to create bpf maps with names. Indeed I saw that > > libbpf API changed between kernel 4.14 and 4.15 and the function > > bpf_create_map_node now takes map name as an argument. Is there a way > > to workaround this with kernel 4.14 and still use map names in > > userspace to refer to bpf maps with libbpf? > > So we do run a few simple tests loading BPF programs (using libbpf) on > 4.9 kernel, so map name should definitely not be a problem at all > (libbpf is smart about detecting what's not supported in kernel and > omitting non-essential things). It might be because of bpf_prog_load > itself, which was long deprecated and you shouldn't use it for > real-world applications. Please either use BPF skeleton or bpf_object > APIs. It should just work, but if it doesn't please report back. > > > > > > > > > > > I would appreciate it if someone can help with a simple working > > > > example of using libbpf on 4.14 kernel, without having any > > > > dependencies. Specifically, I'm looking for an example makefile, and > > > > to know how to load my bpf code with libbpf. > > > > > > libbpf-tools's Makefile would still work. Just drop dependency on > > > vmlinux.h and include system headers directly, if necessary (and if > > > you considered implications of kernel memory layout changes). > > > > > > > Thanks, I'll try that > > > > > > > > > > Thanks, > > > > Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-29 0:00 ` Yaniv Agman @ 2020-09-29 0:07 ` Yonghong Song 2020-09-29 0:16 ` Yaniv Agman 2020-09-29 1:28 ` Andrii Nakryiko 1 sibling, 1 reply; 13+ messages in thread From: Yonghong Song @ 2020-09-29 0:07 UTC (permalink / raw) To: Yaniv Agman, Andrii Nakryiko; +Cc: bpf On 9/28/20 5:00 PM, Yaniv Agman wrote: > Hi Andrii, > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > but not with 4.14. > I used the exact same program, same environment, only changed the > kernel version. > The error message I get on 4.14: > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > libbpf: failed to determine kprobe perf type: No such file or directory > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > 'do_sys_open' perf event: No such file or directory > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > failed to attach BPF programs: No such file or directory > > As the program I made is small, I'm copying it here: > > =========================================== > #include <linux/version.h> > #include <bpf/bpf_helpers.h> > #include <bpf/bpf_tracing.h> > #include <uapi/linux/bpf.h> > #include <uapi/linux/ptrace.h> > > struct bpf_map_def SEC("maps") open_fds = { > .type = BPF_MAP_TYPE_LRU_HASH, > .key_size = sizeof(int), > .value_size = sizeof(int), > .max_entries = 1024, > }; > > SEC("kprobe/do_sys_open") > int BPF_KPROBE(kprobe__do_sys_open) > { > int err; > > u32 id = bpf_get_current_pid_tgid(); > int dfd = PT_REGS_PARM1(ctx); > > if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) { > char log[] = "bpf_map_update_elem %d\n"; put the above definition as global like const char log[] = "bpf_map_update_elem %d\n"; might help. > bpf_trace_printk(log, sizeof(log), err); > return 1; > } > > return 0; > } > > char LICENSE[] SEC("license") = "GPL"; > ================================================== > > Can you think of a reason why this only happens on 4.14? > > Thanks, > Yaniv > > בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת Andrii Nakryiko > <andrii.nakryiko@gmail.com>: >> >> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote: >>> >>> בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko >>> <andrii.nakryiko@gmail.com>: >>>> >>>> On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I'm developing a tool which is now based on BCC, and would like to >>>>> make the move to libbpf. >>>>> I need the tool to support a minimal kernel version 4.14, which >>>>> doesn't have CO-RE. >>>> >>>> You don't need kernel itself to support CO-RE, you just need that >>>> kernel to have BTF in it. If the kernel is too old to have >>>> CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole >>>> -J <path-to-vmlinux-image>`, if that's at all an option for your >>>> setup. >>>> >>> >>> Thanks, I didn't know that >>> >>>>> >>>>> I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, >>>>> but both only deal with newer kernels, and I failed to change them to >>>>> run with a 4.14 kernel. >>>>> >>>>> Although some of the bpf samples in the kernel source don't use CO-RE, >>>>> they all use bpf_load.h, >>>>> and have dependencies on the tools dir, which I would like to avoid. >>>> >>>> Depending on what exactly you are trying to achieve with your BPF >>>> application, you might not need BPF CO-RE, and using libbpf without >>>> CO-RE would be enough for your needs. This would be the case if you >>>> don't need to access any of the kernel data structures (e.g., all sort >>>> of networking BPF apps: TC programs, cgroup sock progs, XDP). But if >>>> you need to do anything tracing related (e.g., looking at kernel's >>>> task_struct or any other internal structure), then you have no choice >>>> and you either have to do on-the-target-host runtime compilation (BCC >>>> way) or relocations (libbpf + BPF CO-RE). This is because of changing >>>> memory layout of kernel structures. >>>> >>>> So, unless you can compile one specific version of your BPF code for a >>>> one specific version of the kernel, you need either BCC or BPF CO-RE. >>>> >>> >>> I'm working on a tracing application >>> (https://github.com/aquasecurity/tracee) which now uses bcc. We now >>> require a minimal kernel version 4.14, and bcc, but eventually we >>> would like to support CO-RE. I thought that we could do the move in >>> two steps. First moving to libbpf and keeping the 4.14 minimal >>> requirement, then adding CO-RE support in the future. >>> In order to do that, I thought of changing bcc requirement to clang >>> requirement, and compile the program once during installation on the >>> target host. This way we get the added value of fast start time >>> without the need to compile every time the program starts (like bcc >>> does), plus having an easier move to CO-RE in the future. >> >> Right, pre-compiling on the target machine with host kernel headers >> should work. So just don't use any of CO-RE features (no CO-RE >> relocations, no vmlinux.h), and it should just work. >> >>> >>> A problem that I encountered with kernel 4.14 and libbpf was that when >>> using bpf_prog_load (If I remember correctly), it returned an error of >>> invalid argument (-22). Doing a small investigation I saw that it >>> happened when trying to create bpf maps with names. Indeed I saw that >>> libbpf API changed between kernel 4.14 and 4.15 and the function >>> bpf_create_map_node now takes map name as an argument. Is there a way >>> to workaround this with kernel 4.14 and still use map names in >>> userspace to refer to bpf maps with libbpf? >> >> So we do run a few simple tests loading BPF programs (using libbpf) on >> 4.9 kernel, so map name should definitely not be a problem at all >> (libbpf is smart about detecting what's not supported in kernel and >> omitting non-essential things). It might be because of bpf_prog_load >> itself, which was long deprecated and you shouldn't use it for >> real-world applications. Please either use BPF skeleton or bpf_object >> APIs. It should just work, but if it doesn't please report back. >> >>> >>>>> >>>>> I would appreciate it if someone can help with a simple working >>>>> example of using libbpf on 4.14 kernel, without having any >>>>> dependencies. Specifically, I'm looking for an example makefile, and >>>>> to know how to load my bpf code with libbpf. >>>> >>>> libbpf-tools's Makefile would still work. Just drop dependency on >>>> vmlinux.h and include system headers directly, if necessary (and if >>>> you considered implications of kernel memory layout changes). >>>> >>> >>> Thanks, I'll try that >>> >>>>> >>>>> Thanks, >>>>> Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-29 0:07 ` Yonghong Song @ 2020-09-29 0:16 ` Yaniv Agman 0 siblings, 0 replies; 13+ messages in thread From: Yaniv Agman @ 2020-09-29 0:16 UTC (permalink / raw) To: Yonghong Song; +Cc: Andrii Nakryiko, bpf Doing this, I get: "libbpf: kernel doesn't support global data" And now it failed earlier (in the loading phase and not the attach phase as before) Anyways, I think that what you suggested may resolve the "skipping unrecognized data section" and not the probe attachment errors בתאריך יום ג׳, 29 בספט׳ 2020 ב-3:07 מאת Yonghong Song <yhs@fb.com>: > > > > On 9/28/20 5:00 PM, Yaniv Agman wrote: > > Hi Andrii, > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > > but not with 4.14. > > I used the exact same program, same environment, only changed the > > kernel version. > > The error message I get on 4.14: > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > > libbpf: failed to determine kprobe perf type: No such file or directory > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > > 'do_sys_open' perf event: No such file or directory > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > > failed to attach BPF programs: No such file or directory > > > > As the program I made is small, I'm copying it here: > > > > =========================================== > > #include <linux/version.h> > > #include <bpf/bpf_helpers.h> > > #include <bpf/bpf_tracing.h> > > #include <uapi/linux/bpf.h> > > #include <uapi/linux/ptrace.h> > > > > struct bpf_map_def SEC("maps") open_fds = { > > .type = BPF_MAP_TYPE_LRU_HASH, > > .key_size = sizeof(int), > > .value_size = sizeof(int), > > .max_entries = 1024, > > }; > > > > SEC("kprobe/do_sys_open") > > int BPF_KPROBE(kprobe__do_sys_open) > > { > > int err; > > > > u32 id = bpf_get_current_pid_tgid(); > > int dfd = PT_REGS_PARM1(ctx); > > > > if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) { > > char log[] = "bpf_map_update_elem %d\n"; > > put the above definition as global like > const char log[] = "bpf_map_update_elem %d\n"; > might help. > > > bpf_trace_printk(log, sizeof(log), err); > > return 1; > > } > > > > return 0; > > } > > > > char LICENSE[] SEC("license") = "GPL"; > > ================================================== > > > > Can you think of a reason why this only happens on 4.14? > > > > Thanks, > > Yaniv > > > > בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת Andrii Nakryiko > > <andrii.nakryiko@gmail.com>: > >> > >> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote: > >>> > >>> בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת Andrii Nakryiko > >>> <andrii.nakryiko@gmail.com>: > >>>> > >>>> On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote: > >>>>> > >>>>> Hello, > >>>>> > >>>>> I'm developing a tool which is now based on BCC, and would like to > >>>>> make the move to libbpf. > >>>>> I need the tool to support a minimal kernel version 4.14, which > >>>>> doesn't have CO-RE. > >>>> > >>>> You don't need kernel itself to support CO-RE, you just need that > >>>> kernel to have BTF in it. If the kernel is too old to have > >>>> CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole > >>>> -J <path-to-vmlinux-image>`, if that's at all an option for your > >>>> setup. > >>>> > >>> > >>> Thanks, I didn't know that > >>> > >>>>> > >>>>> I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc, > >>>>> but both only deal with newer kernels, and I failed to change them to > >>>>> run with a 4.14 kernel. > >>>>> > >>>>> Although some of the bpf samples in the kernel source don't use CO-RE, > >>>>> they all use bpf_load.h, > >>>>> and have dependencies on the tools dir, which I would like to avoid. > >>>> > >>>> Depending on what exactly you are trying to achieve with your BPF > >>>> application, you might not need BPF CO-RE, and using libbpf without > >>>> CO-RE would be enough for your needs. This would be the case if you > >>>> don't need to access any of the kernel data structures (e.g., all sort > >>>> of networking BPF apps: TC programs, cgroup sock progs, XDP). But if > >>>> you need to do anything tracing related (e.g., looking at kernel's > >>>> task_struct or any other internal structure), then you have no choice > >>>> and you either have to do on-the-target-host runtime compilation (BCC > >>>> way) or relocations (libbpf + BPF CO-RE). This is because of changing > >>>> memory layout of kernel structures. > >>>> > >>>> So, unless you can compile one specific version of your BPF code for a > >>>> one specific version of the kernel, you need either BCC or BPF CO-RE. > >>>> > >>> > >>> I'm working on a tracing application > >>> (https://github.com/aquasecurity/tracee) which now uses bcc. We now > >>> require a minimal kernel version 4.14, and bcc, but eventually we > >>> would like to support CO-RE. I thought that we could do the move in > >>> two steps. First moving to libbpf and keeping the 4.14 minimal > >>> requirement, then adding CO-RE support in the future. > >>> In order to do that, I thought of changing bcc requirement to clang > >>> requirement, and compile the program once during installation on the > >>> target host. This way we get the added value of fast start time > >>> without the need to compile every time the program starts (like bcc > >>> does), plus having an easier move to CO-RE in the future. > >> > >> Right, pre-compiling on the target machine with host kernel headers > >> should work. So just don't use any of CO-RE features (no CO-RE > >> relocations, no vmlinux.h), and it should just work. > >> > >>> > >>> A problem that I encountered with kernel 4.14 and libbpf was that when > >>> using bpf_prog_load (If I remember correctly), it returned an error of > >>> invalid argument (-22). Doing a small investigation I saw that it > >>> happened when trying to create bpf maps with names. Indeed I saw that > >>> libbpf API changed between kernel 4.14 and 4.15 and the function > >>> bpf_create_map_node now takes map name as an argument. Is there a way > >>> to workaround this with kernel 4.14 and still use map names in > >>> userspace to refer to bpf maps with libbpf? > >> > >> So we do run a few simple tests loading BPF programs (using libbpf) on > >> 4.9 kernel, so map name should definitely not be a problem at all > >> (libbpf is smart about detecting what's not supported in kernel and > >> omitting non-essential things). It might be because of bpf_prog_load > >> itself, which was long deprecated and you shouldn't use it for > >> real-world applications. Please either use BPF skeleton or bpf_object > >> APIs. It should just work, but if it doesn't please report back. > >> > >>> > >>>>> > >>>>> I would appreciate it if someone can help with a simple working > >>>>> example of using libbpf on 4.14 kernel, without having any > >>>>> dependencies. Specifically, I'm looking for an example makefile, and > >>>>> to know how to load my bpf code with libbpf. > >>>> > >>>> libbpf-tools's Makefile would still work. Just drop dependency on > >>>> vmlinux.h and include system headers directly, if necessary (and if > >>>> you considered implications of kernel memory layout changes). > >>>> > >>> > >>> Thanks, I'll try that > >>> > >>>>> > >>>>> Thanks, > >>>>> Yaniv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-29 0:00 ` Yaniv Agman 2020-09-29 0:07 ` Yonghong Song @ 2020-09-29 1:28 ` Andrii Nakryiko 2020-09-29 8:25 ` Yaniv Agman 1 sibling, 1 reply; 13+ messages in thread From: Andrii Nakryiko @ 2020-09-29 1:28 UTC (permalink / raw) To: Yaniv Agman; +Cc: bpf On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > Hi Andrii, > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > but not with 4.14. > I used the exact same program, same environment, only changed the > kernel version. > The error message I get on 4.14: > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > libbpf: failed to determine kprobe perf type: No such file or directory This means that your kernel doesn't support attaching to kprobe/tracepoint through perf_event subsystem. That's currently the only way that libbpf supports for kprobe/tracapoint programs. It was added in 4.17 kernel, which explains what is happening in your case. It is still possible to attach to kprobe using legacy ways, but libbpf doesn't provide that out of the box. We had a discussion a while ago (about 1 year ago) about adding that to libbpf, but at that time we didn't have a good testing infrastructure to validate such legacy interfaces, plus it's a bit on the unsafe side as far as APIs go (there is no auto-detachment and cleanup with how old kernels allow to do kprobe/tracepoint). But we might reconsider, given it's not a first time I see people get confused and blocked by this. Anyways, here's how you can do it without waiting for libbpf to do this out of the box: int poke_kprobe_events(bool add, const char* name, bool ret) { char buf[256]; int fd, err; fd = open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY | O_APPEND, 0); if (fd < 0) { err = -errno; fprintf(stderr, "failed to open kprobe_events file: %d\n", err); return err; } if (add) snprintf(buf, sizeof(buf), "%c:kprobes/%s %s", ret ? 'r' : 'p', name, name); else snprintf(buf, sizeof(buf), "-:kprobes/%s", name); err = write(fd, buf, strlen(buf)); if (err < 0) { err = -errno; fprintf( stderr, "failed to %s kprobe '%s': %d\n", add ? "add" : "remove", buf, err); } close(fd); return err >= 0 ? 0 : err; } int add_kprobe_event(const char* func_name, bool is_kretprobe) { return poke_kprobe_events(true /*add*/, func_name, is_kretprobe); } int remove_kprobe_event(const char* func_name, bool is_kretprobe) { return poke_kprobe_events(false /*remove*/, func_name, is_kretprobe); } struct bpf_link* attach_kprobe_legacy( struct bpf_program* prog, const char* func_name, bool is_kretprobe) { char fname[256], buf[256]; struct perf_event_attr attr; struct bpf_link* link; int fd = -1, err, id; FILE* f = NULL; err = add_kprobe_event(func_name, is_kretprobe); if (err) { fprintf(stderr, "failed to create kprobe event: %d\n", err); return NULL; } snprintf( fname, sizeof(fname), "/sys/kernel/debug/tracing/events/kprobes/%s/id", func_name); f = fopen(fname, "r"); if (!f) { fprintf(stderr, "failed to open kprobe id file '%s': %d\n", fname, -errno); goto err_out; } if (fscanf(f, "%d\n", &id) != 1) { fprintf(stderr, "failed to read kprobe id from '%s': %d\n", fname, -errno); goto err_out; } fclose(f); f = NULL; memset(&attr, 0, sizeof(attr)); attr.size = sizeof(attr); attr.config = id; attr.type = PERF_TYPE_TRACEPOINT; attr.sample_period = 1; attr.wakeup_events = 1; fd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, PERF_FLAG_FD_CLOEXEC); if (fd < 0) { fprintf( stderr, "failed to create perf event for kprobe ID %d: %d\n", id, -errno); goto err_out; } link = bpf_program__attach_perf_event(prog, fd); err = libbpf_get_error(link); if (err) { fprintf(stderr, "failed to attach to perf event FD %d: %d\n", fd, err); goto err_out; } return link; err_out: if (f) fclose(f); if (fd >= 0) close(fd); remove_kprobe_event(func_name, is_kretprobe); return NULL; } Then you'd use it in your application as: ... skel->links.handler = attach_kprobe_legacy( skel->progs.handler, "do_sys_open", false /* is_kretprobe */); if (!skel->links.handler) { fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n"); err = 1; goto out; } ... kprobe is attached here ... out: /* first clean up step */ bpf_link__destroy(skel->links.handler); /* this is second necessary clean up step */ remove_kprobe_event("do_sys_open", false /* is_kretprobe */); Let me know if that worked. > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > 'do_sys_open' perf event: No such file or directory > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > failed to attach BPF programs: No such file or directory > [...] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-29 1:28 ` Andrii Nakryiko @ 2020-09-29 8:25 ` Yaniv Agman 2020-09-30 18:34 ` Andrii Nakryiko 0 siblings, 1 reply; 13+ messages in thread From: Yaniv Agman @ 2020-09-29 8:25 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת Andrii Nakryiko <andrii.nakryiko@gmail.com>: > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > Hi Andrii, > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > > but not with 4.14. > > I used the exact same program, same environment, only changed the > > kernel version. > > The error message I get on 4.14: > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > > libbpf: failed to determine kprobe perf type: No such file or directory > > This means that your kernel doesn't support attaching to > kprobe/tracepoint through perf_event subsystem. That's currently the > only way that libbpf supports for kprobe/tracapoint programs. It was > added in 4.17 kernel, which explains what is happening in your case. > It is still possible to attach to kprobe using legacy ways, but libbpf > doesn't provide that out of the box. We had a discussion a while ago > (about 1 year ago) about adding that to libbpf, but at that time we > didn't have a good testing infrastructure to validate such legacy > interfaces, plus it's a bit on the unsafe side as far as APIs go > (there is no auto-detachment and cleanup with how old kernels allow to > do kprobe/tracepoint). But we might reconsider, given it's not a first > time I see people get confused and blocked by this. > > Anyways, here's how you can do it without waiting for libbpf to do > this out of the box: > > > int poke_kprobe_events(bool add, const char* name, bool ret) { > char buf[256]; > int fd, err; > > fd = open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY | O_APPEND, 0); > if (fd < 0) { > err = -errno; > fprintf(stderr, "failed to open kprobe_events file: %d\n", err); > return err; > } > > if (add) > snprintf(buf, sizeof(buf), "%c:kprobes/%s %s", ret ? 'r' : 'p', name, name); > else > snprintf(buf, sizeof(buf), "-:kprobes/%s", name); > > err = write(fd, buf, strlen(buf)); > if (err < 0) { > err = -errno; > fprintf( > stderr, > "failed to %s kprobe '%s': %d\n", > add ? "add" : "remove", > buf, > err); > } > close(fd); > return err >= 0 ? 0 : err; > } > > int add_kprobe_event(const char* func_name, bool is_kretprobe) { > return poke_kprobe_events(true /*add*/, func_name, is_kretprobe); > } > > int remove_kprobe_event(const char* func_name, bool is_kretprobe) { > return poke_kprobe_events(false /*remove*/, func_name, is_kretprobe); > } > > struct bpf_link* attach_kprobe_legacy( > struct bpf_program* prog, > const char* func_name, > bool is_kretprobe) { > char fname[256], buf[256]; > struct perf_event_attr attr; > struct bpf_link* link; > int fd = -1, err, id; > FILE* f = NULL; > > err = add_kprobe_event(func_name, is_kretprobe); > if (err) { > fprintf(stderr, "failed to create kprobe event: %d\n", err); > return NULL; > } > > snprintf( > fname, > sizeof(fname), > "/sys/kernel/debug/tracing/events/kprobes/%s/id", > func_name); > f = fopen(fname, "r"); > if (!f) { > fprintf(stderr, "failed to open kprobe id file '%s': %d\n", fname, -errno); > goto err_out; > } > > if (fscanf(f, "%d\n", &id) != 1) { > fprintf(stderr, "failed to read kprobe id from '%s': %d\n", fname, -errno); > goto err_out; > } > > fclose(f); > f = NULL; > > memset(&attr, 0, sizeof(attr)); > attr.size = sizeof(attr); > attr.config = id; > attr.type = PERF_TYPE_TRACEPOINT; > attr.sample_period = 1; > attr.wakeup_events = 1; > > fd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, PERF_FLAG_FD_CLOEXEC); > if (fd < 0) { > fprintf( > stderr, > "failed to create perf event for kprobe ID %d: %d\n", > id, > -errno); > goto err_out; > } > > link = bpf_program__attach_perf_event(prog, fd); > err = libbpf_get_error(link); > if (err) { > fprintf(stderr, "failed to attach to perf event FD %d: %d\n", fd, err); > goto err_out; > } > > return link; > > err_out: > if (f) > fclose(f); > if (fd >= 0) > close(fd); > remove_kprobe_event(func_name, is_kretprobe); > return NULL; > } > > > Then you'd use it in your application as: > > ... > > skel->links.handler = attach_kprobe_legacy( > skel->progs.handler, "do_sys_open", false /* is_kretprobe */); > if (!skel->links.handler) { > fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n"); > err = 1; > goto out; > } > > ... kprobe is attached here ... > > out: > /* first clean up step */ > bpf_link__destroy(skel->links.handler); > /* this is second necessary clean up step */ > remove_kprobe_event("do_sys_open", false /* is_kretprobe */); > > > Let me know if that worked. > Thanks Andrii, I made a small change for the code to compile: skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs After compiling the code, I'm now getting the following error: failed to create perf event for kprobe ID 1930: -2 Failed to attach kprobe using legacy debugfs API! failed to remove kprobe '-:kprobes/do_sys_open': -2 As our application is written in go, I hoped libbpf would support kernel 3.14 out of the box, so we can just call libbpf functions using cgo wrappers. I can do further checks if you'd like, but I think we will also consider updating the minimal kernel version requirement to 4.18 > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > > 'do_sys_open' perf event: No such file or directory > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > > failed to attach BPF programs: No such file or directory > > > > [...] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-29 8:25 ` Yaniv Agman @ 2020-09-30 18:34 ` Andrii Nakryiko 2020-10-04 22:52 ` Yaniv Agman 0 siblings, 1 reply; 13+ messages in thread From: Andrii Nakryiko @ 2020-09-30 18:34 UTC (permalink / raw) To: Yaniv Agman; +Cc: bpf On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote: > > בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת Andrii Nakryiko > <andrii.nakryiko@gmail.com>: > > > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > Hi Andrii, > > > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > > > but not with 4.14. > > > I used the exact same program, same environment, only changed the > > > kernel version. > > > The error message I get on 4.14: > > > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > > > libbpf: failed to determine kprobe perf type: No such file or directory > > > > This means that your kernel doesn't support attaching to > > kprobe/tracepoint through perf_event subsystem. That's currently the > > only way that libbpf supports for kprobe/tracapoint programs. It was > > added in 4.17 kernel, which explains what is happening in your case. > > It is still possible to attach to kprobe using legacy ways, but libbpf > > doesn't provide that out of the box. We had a discussion a while ago > > (about 1 year ago) about adding that to libbpf, but at that time we > > didn't have a good testing infrastructure to validate such legacy > > interfaces, plus it's a bit on the unsafe side as far as APIs go > > (there is no auto-detachment and cleanup with how old kernels allow to > > do kprobe/tracepoint). But we might reconsider, given it's not a first > > time I see people get confused and blocked by this. > > > > Anyways, here's how you can do it without waiting for libbpf to do > > this out of the box: > > > > [...] > > > > > > Then you'd use it in your application as: > > > > ... > > > > skel->links.handler = attach_kprobe_legacy( > > skel->progs.handler, "do_sys_open", false /* is_kretprobe */); > > if (!skel->links.handler) { > > fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n"); > > err = 1; > > goto out; > > } > > > > ... kprobe is attached here ... > > > > out: > > /* first clean up step */ > > bpf_link__destroy(skel->links.handler); > > /* this is second necessary clean up step */ > > remove_kprobe_event("do_sys_open", false /* is_kretprobe */); > > > > > > Let me know if that worked. > > > > Thanks Andrii, > > I made a small change for the code to compile: > skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs > > After compiling the code, I'm now getting the following error: > failed to create perf event for kprobe ID 1930: -2 > Failed to attach kprobe using legacy debugfs API! > failed to remove kprobe '-:kprobes/do_sys_open': -2 I've successfully used that code on the kernel as old as 4.9, so this must be something about your kernel configuration. E.g., check that CONFIG_KPROBE_EVENTS is enabled. > > As our application is written in go, > I hoped libbpf would support kernel 3.14 out of the box, so we can > just call libbpf functions using cgo wrappers. > I can do further checks if you'd like, but I think we will also > consider updating the minimal kernel version requirement to 4.18 It's up to you. Of course using a more recent kernel would be much better, if you can get away with it. > > > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > > > 'do_sys_open' perf event: No such file or directory > > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > > > failed to attach BPF programs: No such file or directory > > > > > > > [...] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-09-30 18:34 ` Andrii Nakryiko @ 2020-10-04 22:52 ` Yaniv Agman 2020-10-05 4:29 ` Andrii Nakryiko 0 siblings, 1 reply; 13+ messages in thread From: Yaniv Agman @ 2020-10-04 22:52 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf בתאריך יום ד׳, 30 בספט׳ 2020 ב-21:34 מאת Andrii Nakryiko <andrii.nakryiko@gmail.com>: > > On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת Andrii Nakryiko > > <andrii.nakryiko@gmail.com>: > > > > > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > > > Hi Andrii, > > > > > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > > > > but not with 4.14. > > > > I used the exact same program, same environment, only changed the > > > > kernel version. > > > > The error message I get on 4.14: > > > > > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > > > > libbpf: failed to determine kprobe perf type: No such file or directory > > > > > > This means that your kernel doesn't support attaching to > > > kprobe/tracepoint through perf_event subsystem. That's currently the > > > only way that libbpf supports for kprobe/tracapoint programs. It was > > > added in 4.17 kernel, which explains what is happening in your case. > > > It is still possible to attach to kprobe using legacy ways, but libbpf > > > doesn't provide that out of the box. We had a discussion a while ago > > > (about 1 year ago) about adding that to libbpf, but at that time we > > > didn't have a good testing infrastructure to validate such legacy > > > interfaces, plus it's a bit on the unsafe side as far as APIs go > > > (there is no auto-detachment and cleanup with how old kernels allow to > > > do kprobe/tracepoint). But we might reconsider, given it's not a first > > > time I see people get confused and blocked by this. > > > > > > Anyways, here's how you can do it without waiting for libbpf to do > > > this out of the box: > > > > > > > > [...] > > > > > > > > > > Then you'd use it in your application as: > > > > > > ... > > > > > > skel->links.handler = attach_kprobe_legacy( > > > skel->progs.handler, "do_sys_open", false /* is_kretprobe */); > > > if (!skel->links.handler) { > > > fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n"); > > > err = 1; > > > goto out; > > > } > > > > > > ... kprobe is attached here ... > > > > > > out: > > > /* first clean up step */ > > > bpf_link__destroy(skel->links.handler); > > > /* this is second necessary clean up step */ > > > remove_kprobe_event("do_sys_open", false /* is_kretprobe */); > > > > > > > > > Let me know if that worked. > > > > > > > Thanks Andrii, > > > > I made a small change for the code to compile: > > skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs > > > > After compiling the code, I'm now getting the following error: > > failed to create perf event for kprobe ID 1930: -2 > > Failed to attach kprobe using legacy debugfs API! > > failed to remove kprobe '-:kprobes/do_sys_open': -2 > > I've successfully used that code on the kernel as old as 4.9, so this > must be something about your kernel configuration. E.g., check that > CONFIG_KPROBE_EVENTS is enabled. Just wanted to update that this code works! I didn't include <linux/unistd.h> and for some reason the compiler didn't complain... Thank you very much Andrii! > > > > > As our application is written in go, > > I hoped libbpf would support kernel 3.14 out of the box, so we can > > just call libbpf functions using cgo wrappers. > > I can do further checks if you'd like, but I think we will also > > consider updating the minimal kernel version requirement to 4.18 > > It's up to you. Of course using a more recent kernel would be much > better, if you can get away with it. > > > > > > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > > > > 'do_sys_open' perf event: No such file or directory > > > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > > > > failed to attach BPF programs: No such file or directory > > > > > > > > > > [...] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Help using libbpf with kernel 4.14 2020-10-04 22:52 ` Yaniv Agman @ 2020-10-05 4:29 ` Andrii Nakryiko 0 siblings, 0 replies; 13+ messages in thread From: Andrii Nakryiko @ 2020-10-05 4:29 UTC (permalink / raw) To: Yaniv Agman; +Cc: bpf On Sun, Oct 4, 2020 at 3:52 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > בתאריך יום ד׳, 30 בספט׳ 2020 ב-21:34 מאת Andrii Nakryiko > <andrii.nakryiko@gmail.com>: > > > > On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת Andrii Nakryiko > > > <andrii.nakryiko@gmail.com>: > > > > > > > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote: > > > > > > > > > > Hi Andrii, > > > > > > > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19 > > > > > but not with 4.14. > > > > > I used the exact same program, same environment, only changed the > > > > > kernel version. > > > > > The error message I get on 4.14: > > > > > > > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1 > > > > > libbpf: failed to determine kprobe perf type: No such file or directory > > > > > > > > This means that your kernel doesn't support attaching to > > > > kprobe/tracepoint through perf_event subsystem. That's currently the > > > > only way that libbpf supports for kprobe/tracapoint programs. It was > > > > added in 4.17 kernel, which explains what is happening in your case. > > > > It is still possible to attach to kprobe using legacy ways, but libbpf > > > > doesn't provide that out of the box. We had a discussion a while ago > > > > (about 1 year ago) about adding that to libbpf, but at that time we > > > > didn't have a good testing infrastructure to validate such legacy > > > > interfaces, plus it's a bit on the unsafe side as far as APIs go > > > > (there is no auto-detachment and cleanup with how old kernels allow to > > > > do kprobe/tracepoint). But we might reconsider, given it's not a first > > > > time I see people get confused and blocked by this. > > > > > > > > Anyways, here's how you can do it without waiting for libbpf to do > > > > this out of the box: > > > > > > > > > > > > [...] > > > > > > > > > > > > > > Then you'd use it in your application as: > > > > > > > > ... > > > > > > > > skel->links.handler = attach_kprobe_legacy( > > > > skel->progs.handler, "do_sys_open", false /* is_kretprobe */); > > > > if (!skel->links.handler) { > > > > fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n"); > > > > err = 1; > > > > goto out; > > > > } > > > > > > > > ... kprobe is attached here ... > > > > > > > > out: > > > > /* first clean up step */ > > > > bpf_link__destroy(skel->links.handler); > > > > /* this is second necessary clean up step */ > > > > remove_kprobe_event("do_sys_open", false /* is_kretprobe */); > > > > > > > > > > > > Let me know if that worked. > > > > > > > > > > Thanks Andrii, > > > > > > I made a small change for the code to compile: > > > skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs > > > > > > After compiling the code, I'm now getting the following error: > > > failed to create perf event for kprobe ID 1930: -2 > > > Failed to attach kprobe using legacy debugfs API! > > > failed to remove kprobe '-:kprobes/do_sys_open': -2 > > > > I've successfully used that code on the kernel as old as 4.9, so this > > must be something about your kernel configuration. E.g., check that > > CONFIG_KPROBE_EVENTS is enabled. > > Just wanted to update that this code works! > I didn't include <linux/unistd.h> and for some reason the compiler > didn't complain... > Thank you very much Andrii! You are welcome, I'm glad you figured it out. > > > > > > > > > As our application is written in go, > > > I hoped libbpf would support kernel 3.14 out of the box, so we can > > > just call libbpf functions using cgo wrappers. > > > I can do further checks if you'd like, but I think we will also > > > consider updating the minimal kernel version requirement to 4.18 > > > > It's up to you. Of course using a more recent kernel would be much > > better, if you can get away with it. > > > > > > > > > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe > > > > > 'do_sys_open' perf event: No such file or directory > > > > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2 > > > > > failed to attach BPF programs: No such file or directory > > > > > > > > > > > > > [...] ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2020-10-05 4:30 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-09-25 23:56 Help using libbpf with kernel 4.14 Yaniv Agman 2020-09-28 5:50 ` Andrii Nakryiko 2020-09-28 20:08 ` Yaniv Agman 2020-09-28 20:24 ` Andrii Nakryiko 2020-09-28 20:39 ` Yaniv Agman 2020-09-29 0:00 ` Yaniv Agman 2020-09-29 0:07 ` Yonghong Song 2020-09-29 0:16 ` Yaniv Agman 2020-09-29 1:28 ` Andrii Nakryiko 2020-09-29 8:25 ` Yaniv Agman 2020-09-30 18:34 ` Andrii Nakryiko 2020-10-04 22:52 ` Yaniv Agman 2020-10-05 4:29 ` Andrii Nakryiko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).