Help using libbpf with kernel 4.14

Help using libbpf with kernel 4.14

[Yaniv Agman]

Hello,

I'm developing a tool which is now based on BCC, and would like to
make the move to libbpf.
I need the tool to support a minimal kernel version 4.14, which
doesn't have CO-RE.

I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
but both only deal with newer kernels, and I failed to change them to
run with a 4.14 kernel.

Although some of the bpf samples in the kernel source don't use CO-RE,
they all use bpf_load.h,
and have dependencies on the tools dir, which I would like to avoid.

I would appreciate it if someone can help with a simple working
example of using libbpf on 4.14 kernel, without having any
dependencies. Specifically, I'm looking for an example makefile, and
to know how to load my bpf code with libbpf.

Thanks,
Yaniv

Re: Help using libbpf with kernel 4.14

[Andrii Nakryiko]

On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> Hello,
>
> I'm developing a tool which is now based on BCC, and would like to
> make the move to libbpf.
> I need the tool to support a minimal kernel version 4.14, which
> doesn't have CO-RE.

You don't need kernel itself to support CO-RE, you just need that
kernel to have BTF in it. If the kernel is too old to have
CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
-J <path-to-vmlinux-image>`, if that's at all an option for your
setup.

>
> I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> but both only deal with newer kernels, and I failed to change them to
> run with a 4.14 kernel.
>
> Although some of the bpf samples in the kernel source don't use CO-RE,
> they all use bpf_load.h,
> and have dependencies on the tools dir, which I would like to avoid.

Depending on what exactly you are trying to achieve with your BPF
application, you might not need BPF CO-RE, and using libbpf without
CO-RE would be enough for your needs. This would be the case if you
don't need to access any of the kernel data structures (e.g., all sort
of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
you need to do anything tracing related (e.g., looking at kernel's
task_struct or any other internal structure), then you have no choice
and you either have to do on-the-target-host runtime compilation (BCC
way) or relocations (libbpf + BPF CO-RE). This is because of changing
memory layout of kernel structures.

So, unless you can compile one specific version of your BPF code for a
one specific version of the kernel, you need either BCC or BPF CO-RE.

>
> I would appreciate it if someone can help with a simple working
> example of using libbpf on 4.14 kernel, without having any
> dependencies. Specifically, I'm looking for an example makefile, and
> to know how to load my bpf code with libbpf.

libbpf-tools's Makefile would still work. Just drop dependency on
vmlinux.h and include system headers directly, if necessary (and if
you considered implications of kernel memory layout changes).

>
> Thanks,
> Yaniv

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > Hello,
> >
> > I'm developing a tool which is now based on BCC, and would like to
> > make the move to libbpf.
> > I need the tool to support a minimal kernel version 4.14, which
> > doesn't have CO-RE.
>
> You don't need kernel itself to support CO-RE, you just need that
> kernel to have BTF in it. If the kernel is too old to have
> CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
> -J <path-to-vmlinux-image>`, if that's at all an option for your
> setup.
>

Thanks, I didn't know that

> >
> > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> > but both only deal with newer kernels, and I failed to change them to
> > run with a 4.14 kernel.
> >
> > Although some of the bpf samples in the kernel source don't use CO-RE,
> > they all use bpf_load.h,
> > and have dependencies on the tools dir, which I would like to avoid.
>
> Depending on what exactly you are trying to achieve with your BPF
> application, you might not need BPF CO-RE, and using libbpf without
> CO-RE would be enough for your needs. This would be the case if you
> don't need to access any of the kernel data structures (e.g., all sort
> of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
> you need to do anything tracing related (e.g., looking at kernel's
> task_struct or any other internal structure), then you have no choice
> and you either have to do on-the-target-host runtime compilation (BCC
> way) or relocations (libbpf + BPF CO-RE). This is because of changing
> memory layout of kernel structures.
>
> So, unless you can compile one specific version of your BPF code for a
> one specific version of the kernel, you need either BCC or BPF CO-RE.
>

I'm working on a tracing application
(https://github.com/aquasecurity/tracee) which now uses bcc. We now
require a minimal kernel version 4.14, and bcc, but eventually we
would like to support CO-RE. I thought that we could do the move in
two steps. First moving to libbpf and keeping the 4.14 minimal
requirement, then adding CO-RE support in the future.
In order to do that, I thought of changing bcc requirement to clang
requirement, and compile the program once during installation on the
target host. This way we get the added value of fast start time
without the need to compile every time the program starts (like bcc
does), plus having an easier move to CO-RE in the future.

A problem that I encountered with kernel 4.14 and libbpf was that when
using bpf_prog_load (If I remember correctly), it returned an error of
invalid argument (-22). Doing a small investigation I saw that it
happened when trying to create bpf maps with names. Indeed I saw that
libbpf API changed between kernel 4.14 and 4.15 and the function
bpf_create_map_node now takes map name as an argument. Is there a way
to workaround this with kernel 4.14 and still use map names in
userspace to refer to bpf maps with libbpf?

> >
> > I would appreciate it if someone can help with a simple working
> > example of using libbpf on 4.14 kernel, without having any
> > dependencies. Specifically, I'm looking for an example makefile, and
> > to know how to load my bpf code with libbpf.
>
> libbpf-tools's Makefile would still work. Just drop dependency on
> vmlinux.h and include system headers directly, if necessary (and if
> you considered implications of kernel memory layout changes).
>

Thanks, I'll try that

> >
> > Thanks,
> > Yaniv

Re: Help using libbpf with kernel 4.14

[Andrii Nakryiko]

On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >
> > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I'm developing a tool which is now based on BCC, and would like to
> > > make the move to libbpf.
> > > I need the tool to support a minimal kernel version 4.14, which
> > > doesn't have CO-RE.
> >
> > You don't need kernel itself to support CO-RE, you just need that
> > kernel to have BTF in it. If the kernel is too old to have
> > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
> > -J <path-to-vmlinux-image>`, if that's at all an option for your
> > setup.
> >
>
> Thanks, I didn't know that
>
> > >
> > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> > > but both only deal with newer kernels, and I failed to change them to
> > > run with a 4.14 kernel.
> > >
> > > Although some of the bpf samples in the kernel source don't use CO-RE,
> > > they all use bpf_load.h,
> > > and have dependencies on the tools dir, which I would like to avoid.
> >
> > Depending on what exactly you are trying to achieve with your BPF
> > application, you might not need BPF CO-RE, and using libbpf without
> > CO-RE would be enough for your needs. This would be the case if you
> > don't need to access any of the kernel data structures (e.g., all sort
> > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
> > you need to do anything tracing related (e.g., looking at kernel's
> > task_struct or any other internal structure), then you have no choice
> > and you either have to do on-the-target-host runtime compilation (BCC
> > way) or relocations (libbpf + BPF CO-RE). This is because of changing
> > memory layout of kernel structures.
> >
> > So, unless you can compile one specific version of your BPF code for a
> > one specific version of the kernel, you need either BCC or BPF CO-RE.
> >
>
> I'm working on a tracing application
> (https://github.com/aquasecurity/tracee) which now uses bcc. We now
> require a minimal kernel version 4.14, and bcc, but eventually we
> would like to support CO-RE. I thought that we could do the move in
> two steps. First moving to libbpf and keeping the 4.14 minimal
> requirement, then adding CO-RE support in the future.
> In order to do that, I thought of changing bcc requirement to clang
> requirement, and compile the program once during installation on the
> target host. This way we get the added value of fast start time
> without the need to compile every time the program starts (like bcc
> does), plus having an easier move to CO-RE in the future.

Right, pre-compiling on the target machine with host kernel headers
should work. So just don't use any of CO-RE features (no CO-RE
relocations, no vmlinux.h), and it should just work.

>
> A problem that I encountered with kernel 4.14 and libbpf was that when
> using bpf_prog_load (If I remember correctly), it returned an error of
> invalid argument (-22). Doing a small investigation I saw that it
> happened when trying to create bpf maps with names. Indeed I saw that
> libbpf API changed between kernel 4.14 and 4.15 and the function
> bpf_create_map_node now takes map name as an argument. Is there a way
> to workaround this with kernel 4.14 and still use map names in
> userspace to refer to bpf maps with libbpf?

So we do run a few simple tests loading BPF programs (using libbpf) on
4.9 kernel, so map name should definitely not be a problem at all
(libbpf is smart about detecting what's not supported in kernel and
omitting non-essential things). It might be because of bpf_prog_load
itself, which was long deprecated and you shouldn't use it for
real-world applications. Please either use BPF skeleton or bpf_object
APIs. It should just work, but if it doesn't please report back.

>
> > >
> > > I would appreciate it if someone can help with a simple working
> > > example of using libbpf on 4.14 kernel, without having any
> > > dependencies. Specifically, I'm looking for an example makefile, and
> > > to know how to load my bpf code with libbpf.
> >
> > libbpf-tools's Makefile would still work. Just drop dependency on
> > vmlinux.h and include system headers directly, if necessary (and if
> > you considered implications of kernel memory layout changes).
> >
>
> Thanks, I'll try that
>
> > >
> > > Thanks,
> > > Yaniv

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
> > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > >
> > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I'm developing a tool which is now based on BCC, and would like to
> > > > make the move to libbpf.
> > > > I need the tool to support a minimal kernel version 4.14, which
> > > > doesn't have CO-RE.
> > >
> > > You don't need kernel itself to support CO-RE, you just need that
> > > kernel to have BTF in it. If the kernel is too old to have
> > > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
> > > -J <path-to-vmlinux-image>`, if that's at all an option for your
> > > setup.
> > >
> >
> > Thanks, I didn't know that
> >
> > > >
> > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> > > > but both only deal with newer kernels, and I failed to change them to
> > > > run with a 4.14 kernel.
> > > >
> > > > Although some of the bpf samples in the kernel source don't use CO-RE,
> > > > they all use bpf_load.h,
> > > > and have dependencies on the tools dir, which I would like to avoid.
> > >
> > > Depending on what exactly you are trying to achieve with your BPF
> > > application, you might not need BPF CO-RE, and using libbpf without
> > > CO-RE would be enough for your needs. This would be the case if you
> > > don't need to access any of the kernel data structures (e.g., all sort
> > > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
> > > you need to do anything tracing related (e.g., looking at kernel's
> > > task_struct or any other internal structure), then you have no choice
> > > and you either have to do on-the-target-host runtime compilation (BCC
> > > way) or relocations (libbpf + BPF CO-RE). This is because of changing
> > > memory layout of kernel structures.
> > >
> > > So, unless you can compile one specific version of your BPF code for a
> > > one specific version of the kernel, you need either BCC or BPF CO-RE.
> > >
> >
> > I'm working on a tracing application
> > (https://github.com/aquasecurity/tracee) which now uses bcc. We now
> > require a minimal kernel version 4.14, and bcc, but eventually we
> > would like to support CO-RE. I thought that we could do the move in
> > two steps. First moving to libbpf and keeping the 4.14 minimal
> > requirement, then adding CO-RE support in the future.
> > In order to do that, I thought of changing bcc requirement to clang
> > requirement, and compile the program once during installation on the
> > target host. This way we get the added value of fast start time
> > without the need to compile every time the program starts (like bcc
> > does), plus having an easier move to CO-RE in the future.
>
> Right, pre-compiling on the target machine with host kernel headers
> should work. So just don't use any of CO-RE features (no CO-RE
> relocations, no vmlinux.h), and it should just work.
>
> >
> > A problem that I encountered with kernel 4.14 and libbpf was that when
> > using bpf_prog_load (If I remember correctly), it returned an error of
> > invalid argument (-22). Doing a small investigation I saw that it
> > happened when trying to create bpf maps with names. Indeed I saw that
> > libbpf API changed between kernel 4.14 and 4.15 and the function
> > bpf_create_map_node now takes map name as an argument. Is there a way
> > to workaround this with kernel 4.14 and still use map names in
> > userspace to refer to bpf maps with libbpf?
>
> So we do run a few simple tests loading BPF programs (using libbpf) on
> 4.9 kernel, so map name should definitely not be a problem at all
> (libbpf is smart about detecting what's not supported in kernel and
> omitting non-essential things). It might be because of bpf_prog_load
> itself, which was long deprecated and you shouldn't use it for
> real-world applications. Please either use BPF skeleton or bpf_object
> APIs. It should just work, but if it doesn't please report back.
>

Thanks, I'll try that and report on any issues found

> >
> > > >
> > > > I would appreciate it if someone can help with a simple working
> > > > example of using libbpf on 4.14 kernel, without having any
> > > > dependencies. Specifically, I'm looking for an example makefile, and
> > > > to know how to load my bpf code with libbpf.
> > >
> > > libbpf-tools's Makefile would still work. Just drop dependency on
> > > vmlinux.h and include system headers directly, if necessary (and if
> > > you considered implications of kernel memory layout changes).
> > >
> >
> > Thanks, I'll try that
> >
> > > >
> > > > Thanks,
> > > > Yaniv

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

Hi Andrii,

I used BPF skeleton as you suggested, which did work with kernel 4.19
but not with 4.14.
I used the exact same program,  same environment, only changed the
kernel version.
The error message I get on 4.14:

libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
libbpf: failed to determine kprobe perf type: No such file or directory
libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
'do_sys_open' perf event: No such file or directory
libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
failed to attach BPF programs: No such file or directory

As the program I made is small, I'm copying it here:

===========================================
#include <linux/version.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <uapi/linux/bpf.h>
#include <uapi/linux/ptrace.h>

struct bpf_map_def SEC("maps") open_fds = {
  .type = BPF_MAP_TYPE_LRU_HASH,
  .key_size = sizeof(int),
  .value_size = sizeof(int),
  .max_entries = 1024,
};

SEC("kprobe/do_sys_open")
int BPF_KPROBE(kprobe__do_sys_open)
{
  int err;

  u32 id = bpf_get_current_pid_tgid();
  int dfd = PT_REGS_PARM1(ctx);

  if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) {
    char log[] = "bpf_map_update_elem %d\n";
    bpf_trace_printk(log, sizeof(log), err);
    return 1;
  }

  return 0;
}

char LICENSE[] SEC("license") = "GPL";
==================================================

Can you think of a reason why this only happens on 4.14?

Thanks,
Yaniv

‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
> > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > >
> > > On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I'm developing a tool which is now based on BCC, and would like to
> > > > make the move to libbpf.
> > > > I need the tool to support a minimal kernel version 4.14, which
> > > > doesn't have CO-RE.
> > >
> > > You don't need kernel itself to support CO-RE, you just need that
> > > kernel to have BTF in it. If the kernel is too old to have
> > > CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
> > > -J <path-to-vmlinux-image>`, if that's at all an option for your
> > > setup.
> > >
> >
> > Thanks, I didn't know that
> >
> > > >
> > > > I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> > > > but both only deal with newer kernels, and I failed to change them to
> > > > run with a 4.14 kernel.
> > > >
> > > > Although some of the bpf samples in the kernel source don't use CO-RE,
> > > > they all use bpf_load.h,
> > > > and have dependencies on the tools dir, which I would like to avoid.
> > >
> > > Depending on what exactly you are trying to achieve with your BPF
> > > application, you might not need BPF CO-RE, and using libbpf without
> > > CO-RE would be enough for your needs. This would be the case if you
> > > don't need to access any of the kernel data structures (e.g., all sort
> > > of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
> > > you need to do anything tracing related (e.g., looking at kernel's
> > > task_struct or any other internal structure), then you have no choice
> > > and you either have to do on-the-target-host runtime compilation (BCC
> > > way) or relocations (libbpf + BPF CO-RE). This is because of changing
> > > memory layout of kernel structures.
> > >
> > > So, unless you can compile one specific version of your BPF code for a
> > > one specific version of the kernel, you need either BCC or BPF CO-RE.
> > >
> >
> > I'm working on a tracing application
> > (https://github.com/aquasecurity/tracee) which now uses bcc. We now
> > require a minimal kernel version 4.14, and bcc, but eventually we
> > would like to support CO-RE. I thought that we could do the move in
> > two steps. First moving to libbpf and keeping the 4.14 minimal
> > requirement, then adding CO-RE support in the future.
> > In order to do that, I thought of changing bcc requirement to clang
> > requirement, and compile the program once during installation on the
> > target host. This way we get the added value of fast start time
> > without the need to compile every time the program starts (like bcc
> > does), plus having an easier move to CO-RE in the future.
>
> Right, pre-compiling on the target machine with host kernel headers
> should work. So just don't use any of CO-RE features (no CO-RE
> relocations, no vmlinux.h), and it should just work.
>
> >
> > A problem that I encountered with kernel 4.14 and libbpf was that when
> > using bpf_prog_load (If I remember correctly), it returned an error of
> > invalid argument (-22). Doing a small investigation I saw that it
> > happened when trying to create bpf maps with names. Indeed I saw that
> > libbpf API changed between kernel 4.14 and 4.15 and the function
> > bpf_create_map_node now takes map name as an argument. Is there a way
> > to workaround this with kernel 4.14 and still use map names in
> > userspace to refer to bpf maps with libbpf?
>
> So we do run a few simple tests loading BPF programs (using libbpf) on
> 4.9 kernel, so map name should definitely not be a problem at all
> (libbpf is smart about detecting what's not supported in kernel and
> omitting non-essential things). It might be because of bpf_prog_load
> itself, which was long deprecated and you shouldn't use it for
> real-world applications. Please either use BPF skeleton or bpf_object
> APIs. It should just work, but if it doesn't please report back.
>
> >
> > > >
> > > > I would appreciate it if someone can help with a simple working
> > > > example of using libbpf on 4.14 kernel, without having any
> > > > dependencies. Specifically, I'm looking for an example makefile, and
> > > > to know how to load my bpf code with libbpf.
> > >
> > > libbpf-tools's Makefile would still work. Just drop dependency on
> > > vmlinux.h and include system headers directly, if necessary (and if
> > > you considered implications of kernel memory layout changes).
> > >
> >
> > Thanks, I'll try that
> >
> > > >
> > > > Thanks,
> > > > Yaniv

Re: Help using libbpf with kernel 4.14

[Yonghong Song]

On 9/28/20 5:00 PM, Yaniv Agman wrote:
> Hi Andrii,
> 
> I used BPF skeleton as you suggested, which did work with kernel 4.19
> but not with 4.14.
> I used the exact same program,  same environment, only changed the
> kernel version.
> The error message I get on 4.14:
> 
> libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> libbpf: failed to determine kprobe perf type: No such file or directory
> libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> 'do_sys_open' perf event: No such file or directory
> libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> failed to attach BPF programs: No such file or directory
> 
> As the program I made is small, I'm copying it here:
> 
> ===========================================
> #include <linux/version.h>
> #include <bpf/bpf_helpers.h>
> #include <bpf/bpf_tracing.h>
> #include <uapi/linux/bpf.h>
> #include <uapi/linux/ptrace.h>
> 
> struct bpf_map_def SEC("maps") open_fds = {
>    .type = BPF_MAP_TYPE_LRU_HASH,
>    .key_size = sizeof(int),
>    .value_size = sizeof(int),
>    .max_entries = 1024,
> };
> 
> SEC("kprobe/do_sys_open")
> int BPF_KPROBE(kprobe__do_sys_open)
> {
>    int err;
> 
>    u32 id = bpf_get_current_pid_tgid();
>    int dfd = PT_REGS_PARM1(ctx);
> 
>    if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) {
>      char log[] = "bpf_map_update_elem %d\n";

put the above definition as global like
const char log[] = "bpf_map_update_elem %d\n";
might help.

>      bpf_trace_printk(log, sizeof(log), err);
>      return 1;
>    }
> 
>    return 0;
> }
> 
> char LICENSE[] SEC("license") = "GPL";
> ==================================================
> 
> Can you think of a reason why this only happens on 4.14?
> 
> Thanks,
> Yaniv
> 
> ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
>>
>> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>>>
>>> ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
>>> <‪andrii.nakryiko@gmail.com‬‏>:‬
>>>>
>>>> On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm developing a tool which is now based on BCC, and would like to
>>>>> make the move to libbpf.
>>>>> I need the tool to support a minimal kernel version 4.14, which
>>>>> doesn't have CO-RE.
>>>>
>>>> You don't need kernel itself to support CO-RE, you just need that
>>>> kernel to have BTF in it. If the kernel is too old to have
>>>> CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
>>>> -J <path-to-vmlinux-image>`, if that's at all an option for your
>>>> setup.
>>>>
>>>
>>> Thanks, I didn't know that
>>>
>>>>>
>>>>> I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
>>>>> but both only deal with newer kernels, and I failed to change them to
>>>>> run with a 4.14 kernel.
>>>>>
>>>>> Although some of the bpf samples in the kernel source don't use CO-RE,
>>>>> they all use bpf_load.h,
>>>>> and have dependencies on the tools dir, which I would like to avoid.
>>>>
>>>> Depending on what exactly you are trying to achieve with your BPF
>>>> application, you might not need BPF CO-RE, and using libbpf without
>>>> CO-RE would be enough for your needs. This would be the case if you
>>>> don't need to access any of the kernel data structures (e.g., all sort
>>>> of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
>>>> you need to do anything tracing related (e.g., looking at kernel's
>>>> task_struct or any other internal structure), then you have no choice
>>>> and you either have to do on-the-target-host runtime compilation (BCC
>>>> way) or relocations (libbpf + BPF CO-RE). This is because of changing
>>>> memory layout of kernel structures.
>>>>
>>>> So, unless you can compile one specific version of your BPF code for a
>>>> one specific version of the kernel, you need either BCC or BPF CO-RE.
>>>>
>>>
>>> I'm working on a tracing application
>>> (https://github.com/aquasecurity/tracee) which now uses bcc. We now
>>> require a minimal kernel version 4.14, and bcc, but eventually we
>>> would like to support CO-RE. I thought that we could do the move in
>>> two steps. First moving to libbpf and keeping the 4.14 minimal
>>> requirement, then adding CO-RE support in the future.
>>> In order to do that, I thought of changing bcc requirement to clang
>>> requirement, and compile the program once during installation on the
>>> target host. This way we get the added value of fast start time
>>> without the need to compile every time the program starts (like bcc
>>> does), plus having an easier move to CO-RE in the future.
>>
>> Right, pre-compiling on the target machine with host kernel headers
>> should work. So just don't use any of CO-RE features (no CO-RE
>> relocations, no vmlinux.h), and it should just work.
>>
>>>
>>> A problem that I encountered with kernel 4.14 and libbpf was that when
>>> using bpf_prog_load (If I remember correctly), it returned an error of
>>> invalid argument (-22). Doing a small investigation I saw that it
>>> happened when trying to create bpf maps with names. Indeed I saw that
>>> libbpf API changed between kernel 4.14 and 4.15 and the function
>>> bpf_create_map_node now takes map name as an argument. Is there a way
>>> to workaround this with kernel 4.14 and still use map names in
>>> userspace to refer to bpf maps with libbpf?
>>
>> So we do run a few simple tests loading BPF programs (using libbpf) on
>> 4.9 kernel, so map name should definitely not be a problem at all
>> (libbpf is smart about detecting what's not supported in kernel and
>> omitting non-essential things). It might be because of bpf_prog_load
>> itself, which was long deprecated and you shouldn't use it for
>> real-world applications. Please either use BPF skeleton or bpf_object
>> APIs. It should just work, but if it doesn't please report back.
>>
>>>
>>>>>
>>>>> I would appreciate it if someone can help with a simple working
>>>>> example of using libbpf on 4.14 kernel, without having any
>>>>> dependencies. Specifically, I'm looking for an example makefile, and
>>>>> to know how to load my bpf code with libbpf.
>>>>
>>>> libbpf-tools's Makefile would still work. Just drop dependency on
>>>> vmlinux.h and include system headers directly, if necessary (and if
>>>> you considered implications of kernel memory layout changes).
>>>>
>>>
>>> Thanks, I'll try that
>>>
>>>>>
>>>>> Thanks,
>>>>> Yaniv

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

Doing this, I get:
"libbpf: kernel doesn't support global data"

And now it failed earlier (in the loading phase and not the attach
phase as before)
Anyways, I think that what you suggested may resolve the "skipping
unrecognized data section" and not the probe attachment errors

‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-3:07 מאת ‪Yonghong Song‬‏ <‪yhs@fb.com‬‏>:‬
>
>
>
> On 9/28/20 5:00 PM, Yaniv Agman wrote:
> > Hi Andrii,
> >
> > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > but not with 4.14.
> > I used the exact same program,  same environment, only changed the
> > kernel version.
> > The error message I get on 4.14:
> >
> > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > libbpf: failed to determine kprobe perf type: No such file or directory
> > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > 'do_sys_open' perf event: No such file or directory
> > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > failed to attach BPF programs: No such file or directory
> >
> > As the program I made is small, I'm copying it here:
> >
> > ===========================================
> > #include <linux/version.h>
> > #include <bpf/bpf_helpers.h>
> > #include <bpf/bpf_tracing.h>
> > #include <uapi/linux/bpf.h>
> > #include <uapi/linux/ptrace.h>
> >
> > struct bpf_map_def SEC("maps") open_fds = {
> >    .type = BPF_MAP_TYPE_LRU_HASH,
> >    .key_size = sizeof(int),
> >    .value_size = sizeof(int),
> >    .max_entries = 1024,
> > };
> >
> > SEC("kprobe/do_sys_open")
> > int BPF_KPROBE(kprobe__do_sys_open)
> > {
> >    int err;
> >
> >    u32 id = bpf_get_current_pid_tgid();
> >    int dfd = PT_REGS_PARM1(ctx);
> >
> >    if ((err = bpf_map_update_elem(&open_fds, &id, &dfd, BPF_ANY))) {
> >      char log[] = "bpf_map_update_elem %d\n";
>
> put the above definition as global like
> const char log[] = "bpf_map_update_elem %d\n";
> might help.
>
> >      bpf_trace_printk(log, sizeof(log), err);
> >      return 1;
> >    }
> >
> >    return 0;
> > }
> >
> > char LICENSE[] SEC("license") = "GPL";
> > ==================================================
> >
> > Can you think of a reason why this only happens on 4.14?
> >
> > Thanks,
> > Yaniv
> >
> > ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-23:24 מאת ‪Andrii Nakryiko‬‏
> > <‪andrii.nakryiko@gmail.com‬‏>:‬
> >>
> >> On Mon, Sep 28, 2020 at 1:08 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >>>
> >>> ‫בתאריך יום ב׳, 28 בספט׳ 2020 ב-8:50 מאת ‪Andrii Nakryiko‬‏
> >>> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >>>>
> >>>> On Fri, Sep 25, 2020 at 4:58 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm developing a tool which is now based on BCC, and would like to
> >>>>> make the move to libbpf.
> >>>>> I need the tool to support a minimal kernel version 4.14, which
> >>>>> doesn't have CO-RE.
> >>>>
> >>>> You don't need kernel itself to support CO-RE, you just need that
> >>>> kernel to have BTF in it. If the kernel is too old to have
> >>>> CONFIG_DEBUG_INFO_BTF config, you can still add BTF by running `pahole
> >>>> -J <path-to-vmlinux-image>`, if that's at all an option for your
> >>>> setup.
> >>>>
> >>>
> >>> Thanks, I didn't know that
> >>>
> >>>>>
> >>>>> I have read bcc-to-libbpf-howto-guide, and looked at the libbpf-tools of bcc,
> >>>>> but both only deal with newer kernels, and I failed to change them to
> >>>>> run with a 4.14 kernel.
> >>>>>
> >>>>> Although some of the bpf samples in the kernel source don't use CO-RE,
> >>>>> they all use bpf_load.h,
> >>>>> and have dependencies on the tools dir, which I would like to avoid.
> >>>>
> >>>> Depending on what exactly you are trying to achieve with your BPF
> >>>> application, you might not need BPF CO-RE, and using libbpf without
> >>>> CO-RE would be enough for your needs. This would be the case if you
> >>>> don't need to access any of the kernel data structures (e.g., all sort
> >>>> of networking BPF apps: TC programs, cgroup sock progs, XDP). But if
> >>>> you need to do anything tracing related (e.g., looking at kernel's
> >>>> task_struct or any other internal structure), then you have no choice
> >>>> and you either have to do on-the-target-host runtime compilation (BCC
> >>>> way) or relocations (libbpf + BPF CO-RE). This is because of changing
> >>>> memory layout of kernel structures.
> >>>>
> >>>> So, unless you can compile one specific version of your BPF code for a
> >>>> one specific version of the kernel, you need either BCC or BPF CO-RE.
> >>>>
> >>>
> >>> I'm working on a tracing application
> >>> (https://github.com/aquasecurity/tracee) which now uses bcc. We now
> >>> require a minimal kernel version 4.14, and bcc, but eventually we
> >>> would like to support CO-RE. I thought that we could do the move in
> >>> two steps. First moving to libbpf and keeping the 4.14 minimal
> >>> requirement, then adding CO-RE support in the future.
> >>> In order to do that, I thought of changing bcc requirement to clang
> >>> requirement, and compile the program once during installation on the
> >>> target host. This way we get the added value of fast start time
> >>> without the need to compile every time the program starts (like bcc
> >>> does), plus having an easier move to CO-RE in the future.
> >>
> >> Right, pre-compiling on the target machine with host kernel headers
> >> should work. So just don't use any of CO-RE features (no CO-RE
> >> relocations, no vmlinux.h), and it should just work.
> >>
> >>>
> >>> A problem that I encountered with kernel 4.14 and libbpf was that when
> >>> using bpf_prog_load (If I remember correctly), it returned an error of
> >>> invalid argument (-22). Doing a small investigation I saw that it
> >>> happened when trying to create bpf maps with names. Indeed I saw that
> >>> libbpf API changed between kernel 4.14 and 4.15 and the function
> >>> bpf_create_map_node now takes map name as an argument. Is there a way
> >>> to workaround this with kernel 4.14 and still use map names in
> >>> userspace to refer to bpf maps with libbpf?
> >>
> >> So we do run a few simple tests loading BPF programs (using libbpf) on
> >> 4.9 kernel, so map name should definitely not be a problem at all
> >> (libbpf is smart about detecting what's not supported in kernel and
> >> omitting non-essential things). It might be because of bpf_prog_load
> >> itself, which was long deprecated and you shouldn't use it for
> >> real-world applications. Please either use BPF skeleton or bpf_object
> >> APIs. It should just work, but if it doesn't please report back.
> >>
> >>>
> >>>>>
> >>>>> I would appreciate it if someone can help with a simple working
> >>>>> example of using libbpf on 4.14 kernel, without having any
> >>>>> dependencies. Specifically, I'm looking for an example makefile, and
> >>>>> to know how to load my bpf code with libbpf.
> >>>>
> >>>> libbpf-tools's Makefile would still work. Just drop dependency on
> >>>> vmlinux.h and include system headers directly, if necessary (and if
> >>>> you considered implications of kernel memory layout changes).
> >>>>
> >>>
> >>> Thanks, I'll try that
> >>>
> >>>>>
> >>>>> Thanks,
> >>>>> Yaniv

Re: Help using libbpf with kernel 4.14

[Andrii Nakryiko]

On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> Hi Andrii,
>
> I used BPF skeleton as you suggested, which did work with kernel 4.19
> but not with 4.14.
> I used the exact same program,  same environment, only changed the
> kernel version.
> The error message I get on 4.14:
>
> libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> libbpf: failed to determine kprobe perf type: No such file or directory

This means that your kernel doesn't support attaching to
kprobe/tracepoint through perf_event subsystem. That's currently the
only way that libbpf supports for kprobe/tracapoint programs. It was
added in 4.17 kernel, which explains what is happening in your case.
It is still possible to attach to kprobe using legacy ways, but libbpf
doesn't provide that out of the box. We had a discussion a while ago
(about 1 year ago) about adding that to libbpf, but at that time we
didn't have a good testing infrastructure to validate such legacy
interfaces, plus it's a bit on the unsafe side as far as APIs go
(there is no auto-detachment and cleanup with how old kernels allow to
do kprobe/tracepoint). But we might reconsider, given it's not a first
time I see people get confused and blocked by this.

Anyways, here's how you can do it without waiting for libbpf to do
this out of the box:


int poke_kprobe_events(bool add, const char* name, bool ret) {
  char buf[256];
  int fd, err;

  fd = open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY | O_APPEND, 0);
  if (fd < 0) {
    err = -errno;
    fprintf(stderr, "failed to open kprobe_events file: %d\n", err);
    return err;
  }

  if (add)
    snprintf(buf, sizeof(buf), "%c:kprobes/%s %s", ret ? 'r' : 'p', name, name);
  else
    snprintf(buf, sizeof(buf), "-:kprobes/%s", name);

  err = write(fd, buf, strlen(buf));
  if (err < 0) {
    err = -errno;
    fprintf(
        stderr,
        "failed to %s kprobe '%s': %d\n",
        add ? "add" : "remove",
        buf,
        err);
  }
  close(fd);
  return err >= 0 ? 0 : err;
}

int add_kprobe_event(const char* func_name, bool is_kretprobe) {
  return poke_kprobe_events(true /*add*/, func_name, is_kretprobe);
}

int remove_kprobe_event(const char* func_name, bool is_kretprobe) {
  return poke_kprobe_events(false /*remove*/, func_name, is_kretprobe);
}

struct bpf_link* attach_kprobe_legacy(
    struct bpf_program* prog,
    const char* func_name,
    bool is_kretprobe) {
  char fname[256], buf[256];
  struct perf_event_attr attr;
  struct bpf_link* link;
  int fd = -1, err, id;
  FILE* f = NULL;

  err = add_kprobe_event(func_name, is_kretprobe);
  if (err) {
    fprintf(stderr, "failed to create kprobe event: %d\n", err);
    return NULL;
  }

  snprintf(
      fname,
      sizeof(fname),
      "/sys/kernel/debug/tracing/events/kprobes/%s/id",
      func_name);
  f = fopen(fname, "r");
  if (!f) {
    fprintf(stderr, "failed to open kprobe id file '%s': %d\n", fname, -errno);
    goto err_out;
  }

  if (fscanf(f, "%d\n", &id) != 1) {
    fprintf(stderr, "failed to read kprobe id from '%s': %d\n", fname, -errno);
    goto err_out;
  }

  fclose(f);
  f = NULL;

  memset(&attr, 0, sizeof(attr));
  attr.size = sizeof(attr);
  attr.config = id;
  attr.type = PERF_TYPE_TRACEPOINT;
  attr.sample_period = 1;
  attr.wakeup_events = 1;

  fd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, PERF_FLAG_FD_CLOEXEC);
  if (fd < 0) {
    fprintf(
        stderr,
        "failed to create perf event for kprobe ID %d: %d\n",
        id,
        -errno);
    goto err_out;
  }

  link = bpf_program__attach_perf_event(prog, fd);
  err = libbpf_get_error(link);
  if (err) {
    fprintf(stderr, "failed to attach to perf event FD %d: %d\n", fd, err);
    goto err_out;
  }

  return link;

err_out:
  if (f)
    fclose(f);
  if (fd >= 0)
    close(fd);
  remove_kprobe_event(func_name, is_kretprobe);
  return NULL;
}


Then you'd use it in your application as:

...

  skel->links.handler = attach_kprobe_legacy(
      skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
  if (!skel->links.handler) {
    fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
    err = 1;
    goto out;
  }

  ... kprobe is attached here ...

out:
  /* first clean up step */
  bpf_link__destroy(skel->links.handler);
  /* this is second necessary clean up step */
  remove_kprobe_event("do_sys_open", false /* is_kretprobe */);


Let me know if that worked.

> libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> 'do_sys_open' perf event: No such file or directory
> libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> failed to attach BPF programs: No such file or directory
>

[...]

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > Hi Andrii,
> >
> > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > but not with 4.14.
> > I used the exact same program,  same environment, only changed the
> > kernel version.
> > The error message I get on 4.14:
> >
> > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > libbpf: failed to determine kprobe perf type: No such file or directory
>
> This means that your kernel doesn't support attaching to
> kprobe/tracepoint through perf_event subsystem. That's currently the
> only way that libbpf supports for kprobe/tracapoint programs. It was
> added in 4.17 kernel, which explains what is happening in your case.
> It is still possible to attach to kprobe using legacy ways, but libbpf
> doesn't provide that out of the box. We had a discussion a while ago
> (about 1 year ago) about adding that to libbpf, but at that time we
> didn't have a good testing infrastructure to validate such legacy
> interfaces, plus it's a bit on the unsafe side as far as APIs go
> (there is no auto-detachment and cleanup with how old kernels allow to
> do kprobe/tracepoint). But we might reconsider, given it's not a first
> time I see people get confused and blocked by this.
>
> Anyways, here's how you can do it without waiting for libbpf to do
> this out of the box:
>
>
> int poke_kprobe_events(bool add, const char* name, bool ret) {
>   char buf[256];
>   int fd, err;
>
>   fd = open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY | O_APPEND, 0);
>   if (fd < 0) {
>     err = -errno;
>     fprintf(stderr, "failed to open kprobe_events file: %d\n", err);
>     return err;
>   }
>
>   if (add)
>     snprintf(buf, sizeof(buf), "%c:kprobes/%s %s", ret ? 'r' : 'p', name, name);
>   else
>     snprintf(buf, sizeof(buf), "-:kprobes/%s", name);
>
>   err = write(fd, buf, strlen(buf));
>   if (err < 0) {
>     err = -errno;
>     fprintf(
>         stderr,
>         "failed to %s kprobe '%s': %d\n",
>         add ? "add" : "remove",
>         buf,
>         err);
>   }
>   close(fd);
>   return err >= 0 ? 0 : err;
> }
>
> int add_kprobe_event(const char* func_name, bool is_kretprobe) {
>   return poke_kprobe_events(true /*add*/, func_name, is_kretprobe);
> }
>
> int remove_kprobe_event(const char* func_name, bool is_kretprobe) {
>   return poke_kprobe_events(false /*remove*/, func_name, is_kretprobe);
> }
>
> struct bpf_link* attach_kprobe_legacy(
>     struct bpf_program* prog,
>     const char* func_name,
>     bool is_kretprobe) {
>   char fname[256], buf[256];
>   struct perf_event_attr attr;
>   struct bpf_link* link;
>   int fd = -1, err, id;
>   FILE* f = NULL;
>
>   err = add_kprobe_event(func_name, is_kretprobe);
>   if (err) {
>     fprintf(stderr, "failed to create kprobe event: %d\n", err);
>     return NULL;
>   }
>
>   snprintf(
>       fname,
>       sizeof(fname),
>       "/sys/kernel/debug/tracing/events/kprobes/%s/id",
>       func_name);
>   f = fopen(fname, "r");
>   if (!f) {
>     fprintf(stderr, "failed to open kprobe id file '%s': %d\n", fname, -errno);
>     goto err_out;
>   }
>
>   if (fscanf(f, "%d\n", &id) != 1) {
>     fprintf(stderr, "failed to read kprobe id from '%s': %d\n", fname, -errno);
>     goto err_out;
>   }
>
>   fclose(f);
>   f = NULL;
>
>   memset(&attr, 0, sizeof(attr));
>   attr.size = sizeof(attr);
>   attr.config = id;
>   attr.type = PERF_TYPE_TRACEPOINT;
>   attr.sample_period = 1;
>   attr.wakeup_events = 1;
>
>   fd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, PERF_FLAG_FD_CLOEXEC);
>   if (fd < 0) {
>     fprintf(
>         stderr,
>         "failed to create perf event for kprobe ID %d: %d\n",
>         id,
>         -errno);
>     goto err_out;
>   }
>
>   link = bpf_program__attach_perf_event(prog, fd);
>   err = libbpf_get_error(link);
>   if (err) {
>     fprintf(stderr, "failed to attach to perf event FD %d: %d\n", fd, err);
>     goto err_out;
>   }
>
>   return link;
>
> err_out:
>   if (f)
>     fclose(f);
>   if (fd >= 0)
>     close(fd);
>   remove_kprobe_event(func_name, is_kretprobe);
>   return NULL;
> }
>
>
> Then you'd use it in your application as:
>
> ...
>
>   skel->links.handler = attach_kprobe_legacy(
>       skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
>   if (!skel->links.handler) {
>     fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
>     err = 1;
>     goto out;
>   }
>
>   ... kprobe is attached here ...
>
> out:
>   /* first clean up step */
>   bpf_link__destroy(skel->links.handler);
>   /* this is second necessary clean up step */
>   remove_kprobe_event("do_sys_open", false /* is_kretprobe */);
>
>
> Let me know if that worked.
>

Thanks Andrii,

I made a small change for the code to compile:
skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs

After compiling the code, I'm now getting the following error:
failed to create perf event for kprobe ID 1930: -2
Failed to attach kprobe using legacy debugfs API!
failed to remove kprobe '-:kprobes/do_sys_open': -2

As our application is written in go,
I hoped libbpf would support kernel 3.14 out of the box, so we can
just call libbpf functions using cgo wrappers.
I can do further checks if you'd like, but I think we will also
consider updating the minimal kernel version requirement to 4.18

> > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > 'do_sys_open' perf event: No such file or directory
> > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > failed to attach BPF programs: No such file or directory
> >
>
> [...]

Re: Help using libbpf with kernel 4.14

[Andrii Nakryiko]

On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> ‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >
> > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > >
> > > Hi Andrii,
> > >
> > > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > > but not with 4.14.
> > > I used the exact same program,  same environment, only changed the
> > > kernel version.
> > > The error message I get on 4.14:
> > >
> > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > > libbpf: failed to determine kprobe perf type: No such file or directory
> >
> > This means that your kernel doesn't support attaching to
> > kprobe/tracepoint through perf_event subsystem. That's currently the
> > only way that libbpf supports for kprobe/tracapoint programs. It was
> > added in 4.17 kernel, which explains what is happening in your case.
> > It is still possible to attach to kprobe using legacy ways, but libbpf
> > doesn't provide that out of the box. We had a discussion a while ago
> > (about 1 year ago) about adding that to libbpf, but at that time we
> > didn't have a good testing infrastructure to validate such legacy
> > interfaces, plus it's a bit on the unsafe side as far as APIs go
> > (there is no auto-detachment and cleanup with how old kernels allow to
> > do kprobe/tracepoint). But we might reconsider, given it's not a first
> > time I see people get confused and blocked by this.
> >
> > Anyways, here's how you can do it without waiting for libbpf to do
> > this out of the box:
> >
> >

[...]

> >
> >
> > Then you'd use it in your application as:
> >
> > ...
> >
> >   skel->links.handler = attach_kprobe_legacy(
> >       skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
> >   if (!skel->links.handler) {
> >     fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
> >     err = 1;
> >     goto out;
> >   }
> >
> >   ... kprobe is attached here ...
> >
> > out:
> >   /* first clean up step */
> >   bpf_link__destroy(skel->links.handler);
> >   /* this is second necessary clean up step */
> >   remove_kprobe_event("do_sys_open", false /* is_kretprobe */);
> >
> >
> > Let me know if that worked.
> >
>
> Thanks Andrii,
>
> I made a small change for the code to compile:
> skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs
>
> After compiling the code, I'm now getting the following error:
> failed to create perf event for kprobe ID 1930: -2
> Failed to attach kprobe using legacy debugfs API!
> failed to remove kprobe '-:kprobes/do_sys_open': -2

I've successfully used that code on the kernel as old as 4.9, so this
must be something about your kernel configuration. E.g., check that
CONFIG_KPROBE_EVENTS is enabled.

>
> As our application is written in go,
> I hoped libbpf would support kernel 3.14 out of the box, so we can
> just call libbpf functions using cgo wrappers.
> I can do further checks if you'd like, but I think we will also
> consider updating the minimal kernel version requirement to 4.18

It's up to you. Of course using a more recent kernel would be much
better, if you can get away with it.

>
> > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > > 'do_sys_open' perf event: No such file or directory
> > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > > failed to attach BPF programs: No such file or directory
> > >
> >
> > [...]

Re: Help using libbpf with kernel 4.14

[Yaniv Agman]

‫בתאריך יום ד׳, 30 בספט׳ 2020 ב-21:34 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > ‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת ‪Andrii Nakryiko‬‏
> > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > >
> > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > >
> > > > Hi Andrii,
> > > >
> > > > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > > > but not with 4.14.
> > > > I used the exact same program,  same environment, only changed the
> > > > kernel version.
> > > > The error message I get on 4.14:
> > > >
> > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > > > libbpf: failed to determine kprobe perf type: No such file or directory
> > >
> > > This means that your kernel doesn't support attaching to
> > > kprobe/tracepoint through perf_event subsystem. That's currently the
> > > only way that libbpf supports for kprobe/tracapoint programs. It was
> > > added in 4.17 kernel, which explains what is happening in your case.
> > > It is still possible to attach to kprobe using legacy ways, but libbpf
> > > doesn't provide that out of the box. We had a discussion a while ago
> > > (about 1 year ago) about adding that to libbpf, but at that time we
> > > didn't have a good testing infrastructure to validate such legacy
> > > interfaces, plus it's a bit on the unsafe side as far as APIs go
> > > (there is no auto-detachment and cleanup with how old kernels allow to
> > > do kprobe/tracepoint). But we might reconsider, given it's not a first
> > > time I see people get confused and blocked by this.
> > >
> > > Anyways, here's how you can do it without waiting for libbpf to do
> > > this out of the box:
> > >
> > >
>
> [...]
>
> > >
> > >
> > > Then you'd use it in your application as:
> > >
> > > ...
> > >
> > >   skel->links.handler = attach_kprobe_legacy(
> > >       skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
> > >   if (!skel->links.handler) {
> > >     fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
> > >     err = 1;
> > >     goto out;
> > >   }
> > >
> > >   ... kprobe is attached here ...
> > >
> > > out:
> > >   /* first clean up step */
> > >   bpf_link__destroy(skel->links.handler);
> > >   /* this is second necessary clean up step */
> > >   remove_kprobe_event("do_sys_open", false /* is_kretprobe */);
> > >
> > >
> > > Let me know if that worked.
> > >
> >
> > Thanks Andrii,
> >
> > I made a small change for the code to compile:
> > skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs
> >
> > After compiling the code, I'm now getting the following error:
> > failed to create perf event for kprobe ID 1930: -2
> > Failed to attach kprobe using legacy debugfs API!
> > failed to remove kprobe '-:kprobes/do_sys_open': -2
>
> I've successfully used that code on the kernel as old as 4.9, so this
> must be something about your kernel configuration. E.g., check that
> CONFIG_KPROBE_EVENTS is enabled.

Just wanted to update that this code works!
I didn't include <linux/unistd.h> and for some reason the compiler
didn't complain...
Thank you very much Andrii!

>
> >
> > As our application is written in go,
> > I hoped libbpf would support kernel 3.14 out of the box, so we can
> > just call libbpf functions using cgo wrappers.
> > I can do further checks if you'd like, but I think we will also
> > consider updating the minimal kernel version requirement to 4.18
>
> It's up to you. Of course using a more recent kernel would be much
> better, if you can get away with it.
>
> >
> > > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > > > 'do_sys_open' perf event: No such file or directory
> > > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > > > failed to attach BPF programs: No such file or directory
> > > >
> > >
> > > [...]

Re: Help using libbpf with kernel 4.14

[Andrii Nakryiko]

On Sun, Oct 4, 2020 at 3:52 PM Yaniv Agman <yanivagman@gmail.com> wrote:
>
> ‫בתאריך יום ד׳, 30 בספט׳ 2020 ב-21:34 מאת ‪Andrii Nakryiko‬‏
> <‪andrii.nakryiko@gmail.com‬‏>:‬
> >
> > On Tue, Sep 29, 2020 at 1:25 AM Yaniv Agman <yanivagman@gmail.com> wrote:
> > >
> > > ‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת ‪Andrii Nakryiko‬‏
> > > <‪andrii.nakryiko@gmail.com‬‏>:‬
> > > >
> > > > On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> > > > >
> > > > > Hi Andrii,
> > > > >
> > > > > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > > > > but not with 4.14.
> > > > > I used the exact same program,  same environment, only changed the
> > > > > kernel version.
> > > > > The error message I get on 4.14:
> > > > >
> > > > > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > > > > libbpf: failed to determine kprobe perf type: No such file or directory
> > > >
> > > > This means that your kernel doesn't support attaching to
> > > > kprobe/tracepoint through perf_event subsystem. That's currently the
> > > > only way that libbpf supports for kprobe/tracapoint programs. It was
> > > > added in 4.17 kernel, which explains what is happening in your case.
> > > > It is still possible to attach to kprobe using legacy ways, but libbpf
> > > > doesn't provide that out of the box. We had a discussion a while ago
> > > > (about 1 year ago) about adding that to libbpf, but at that time we
> > > > didn't have a good testing infrastructure to validate such legacy
> > > > interfaces, plus it's a bit on the unsafe side as far as APIs go
> > > > (there is no auto-detachment and cleanup with how old kernels allow to
> > > > do kprobe/tracepoint). But we might reconsider, given it's not a first
> > > > time I see people get confused and blocked by this.
> > > >
> > > > Anyways, here's how you can do it without waiting for libbpf to do
> > > > this out of the box:
> > > >
> > > >
> >
> > [...]
> >
> > > >
> > > >
> > > > Then you'd use it in your application as:
> > > >
> > > > ...
> > > >
> > > >   skel->links.handler = attach_kprobe_legacy(
> > > >       skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
> > > >   if (!skel->links.handler) {
> > > >     fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
> > > >     err = 1;
> > > >     goto out;
> > > >   }
> > > >
> > > >   ... kprobe is attached here ...
> > > >
> > > > out:
> > > >   /* first clean up step */
> > > >   bpf_link__destroy(skel->links.handler);
> > > >   /* this is second necessary clean up step */
> > > >   remove_kprobe_event("do_sys_open", false /* is_kretprobe */);
> > > >
> > > >
> > > > Let me know if that worked.
> > > >
> > >
> > > Thanks Andrii,
> > >
> > > I made a small change for the code to compile:
> > > skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs
> > >
> > > After compiling the code, I'm now getting the following error:
> > > failed to create perf event for kprobe ID 1930: -2
> > > Failed to attach kprobe using legacy debugfs API!
> > > failed to remove kprobe '-:kprobes/do_sys_open': -2
> >
> > I've successfully used that code on the kernel as old as 4.9, so this
> > must be something about your kernel configuration. E.g., check that
> > CONFIG_KPROBE_EVENTS is enabled.
>
> Just wanted to update that this code works!
> I didn't include <linux/unistd.h> and for some reason the compiler
> didn't complain...
> Thank you very much Andrii!

You are welcome, I'm glad you figured it out.

>
> >
> > >
> > > As our application is written in go,
> > > I hoped libbpf would support kernel 3.14 out of the box, so we can
> > > just call libbpf functions using cgo wrappers.
> > > I can do further checks if you'd like, but I think we will also
> > > consider updating the minimal kernel version requirement to 4.18
> >
> > It's up to you. Of course using a more recent kernel would be much
> > better, if you can get away with it.
> >
> > >
> > > > > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > > > > 'do_sys_open' perf event: No such file or directory
> > > > > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > > > > failed to attach BPF programs: No such file or directory
> > > > >
> > > >
> > > > [...]