Re: Help using libbpf with kernel 4.14

[Yaniv Agman <yanivagman@gmail.com>]

‫בתאריך יום ג׳, 29 בספט׳ 2020 ב-4:29 מאת ‪Andrii Nakryiko‬‏
<‪andrii.nakryiko@gmail.com‬‏>:‬
>
> On Mon, Sep 28, 2020 at 5:01 PM Yaniv Agman <yanivagman@gmail.com> wrote:
> >
> > Hi Andrii,
> >
> > I used BPF skeleton as you suggested, which did work with kernel 4.19
> > but not with 4.14.
> > I used the exact same program,  same environment, only changed the
> > kernel version.
> > The error message I get on 4.14:
> >
> > libbpf: elf: skipping unrecognized data section(5) .rodata.str1.1
> > libbpf: failed to determine kprobe perf type: No such file or directory
>
> This means that your kernel doesn't support attaching to
> kprobe/tracepoint through perf_event subsystem. That's currently the
> only way that libbpf supports for kprobe/tracapoint programs. It was
> added in 4.17 kernel, which explains what is happening in your case.
> It is still possible to attach to kprobe using legacy ways, but libbpf
> doesn't provide that out of the box. We had a discussion a while ago
> (about 1 year ago) about adding that to libbpf, but at that time we
> didn't have a good testing infrastructure to validate such legacy
> interfaces, plus it's a bit on the unsafe side as far as APIs go
> (there is no auto-detachment and cleanup with how old kernels allow to
> do kprobe/tracepoint). But we might reconsider, given it's not a first
> time I see people get confused and blocked by this.
>
> Anyways, here's how you can do it without waiting for libbpf to do
> this out of the box:
>
>
> int poke_kprobe_events(bool add, const char* name, bool ret) {
>   char buf[256];
>   int fd, err;
>
>   fd = open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY | O_APPEND, 0);
>   if (fd < 0) {
>     err = -errno;
>     fprintf(stderr, "failed to open kprobe_events file: %d\n", err);
>     return err;
>   }
>
>   if (add)
>     snprintf(buf, sizeof(buf), "%c:kprobes/%s %s", ret ? 'r' : 'p', name, name);
>   else
>     snprintf(buf, sizeof(buf), "-:kprobes/%s", name);
>
>   err = write(fd, buf, strlen(buf));
>   if (err < 0) {
>     err = -errno;
>     fprintf(
>         stderr,
>         "failed to %s kprobe '%s': %d\n",
>         add ? "add" : "remove",
>         buf,
>         err);
>   }
>   close(fd);
>   return err >= 0 ? 0 : err;
> }
>
> int add_kprobe_event(const char* func_name, bool is_kretprobe) {
>   return poke_kprobe_events(true /*add*/, func_name, is_kretprobe);
> }
>
> int remove_kprobe_event(const char* func_name, bool is_kretprobe) {
>   return poke_kprobe_events(false /*remove*/, func_name, is_kretprobe);
> }
>
> struct bpf_link* attach_kprobe_legacy(
>     struct bpf_program* prog,
>     const char* func_name,
>     bool is_kretprobe) {
>   char fname[256], buf[256];
>   struct perf_event_attr attr;
>   struct bpf_link* link;
>   int fd = -1, err, id;
>   FILE* f = NULL;
>
>   err = add_kprobe_event(func_name, is_kretprobe);
>   if (err) {
>     fprintf(stderr, "failed to create kprobe event: %d\n", err);
>     return NULL;
>   }
>
>   snprintf(
>       fname,
>       sizeof(fname),
>       "/sys/kernel/debug/tracing/events/kprobes/%s/id",
>       func_name);
>   f = fopen(fname, "r");
>   if (!f) {
>     fprintf(stderr, "failed to open kprobe id file '%s': %d\n", fname, -errno);
>     goto err_out;
>   }
>
>   if (fscanf(f, "%d\n", &id) != 1) {
>     fprintf(stderr, "failed to read kprobe id from '%s': %d\n", fname, -errno);
>     goto err_out;
>   }
>
>   fclose(f);
>   f = NULL;
>
>   memset(&attr, 0, sizeof(attr));
>   attr.size = sizeof(attr);
>   attr.config = id;
>   attr.type = PERF_TYPE_TRACEPOINT;
>   attr.sample_period = 1;
>   attr.wakeup_events = 1;
>
>   fd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, PERF_FLAG_FD_CLOEXEC);
>   if (fd < 0) {
>     fprintf(
>         stderr,
>         "failed to create perf event for kprobe ID %d: %d\n",
>         id,
>         -errno);
>     goto err_out;
>   }
>
>   link = bpf_program__attach_perf_event(prog, fd);
>   err = libbpf_get_error(link);
>   if (err) {
>     fprintf(stderr, "failed to attach to perf event FD %d: %d\n", fd, err);
>     goto err_out;
>   }
>
>   return link;
>
> err_out:
>   if (f)
>     fclose(f);
>   if (fd >= 0)
>     close(fd);
>   remove_kprobe_event(func_name, is_kretprobe);
>   return NULL;
> }
>
>
> Then you'd use it in your application as:
>
> ...
>
>   skel->links.handler = attach_kprobe_legacy(
>       skel->progs.handler, "do_sys_open", false /* is_kretprobe */);
>   if (!skel->links.handler) {
>     fprintf(stderr, "Failed to attach kprobe using legacy debugfs API!\n");
>     err = 1;
>     goto out;
>   }
>
>   ... kprobe is attached here ...
>
> out:
>   /* first clean up step */
>   bpf_link__destroy(skel->links.handler);
>   /* this is second necessary clean up step */
>   remove_kprobe_event("do_sys_open", false /* is_kretprobe */);
>
>
> Let me know if that worked.
>

Thanks Andrii,

I made a small change for the code to compile:
skel->links.handler to skel->links.kprobe__do_sys_open and same for skel->progs

After compiling the code, I'm now getting the following error:
failed to create perf event for kprobe ID 1930: -2
Failed to attach kprobe using legacy debugfs API!
failed to remove kprobe '-:kprobes/do_sys_open': -2

As our application is written in go,
I hoped libbpf would support kernel 3.14 out of the box, so we can
just call libbpf functions using cgo wrappers.
I can do further checks if you'd like, but I think we will also
consider updating the minimal kernel version requirement to 4.18

> > libbpf: prog 'kprobe__do_sys_open': failed to create kprobe
> > 'do_sys_open' perf event: No such file or directory
> > libbpf: failed to auto-attach program 'kprobe__do_sys_open': -2
> > failed to attach BPF programs: No such file or directory
> >
>
> [...]