Fwd: [iovisor-dev] Building BPF programs and kernel persistence

Fwd: [iovisor-dev] Building BPF programs and kernel persistence

[Tristan Mayfield]

---------- Forwarded message ---------
From: Tristan Mayfield <mayfieldtristan@gmail.com>
Date: Mon, May 18, 2020 at 12:23 PM
Subject: Re: [iovisor-dev] Building BPF programs and kernel persistence
To: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: <iovisor-dev@lists.iovisor.org>, <bpf@vger.kernel.org>


Thanks for the reply Andrii.

Managed to get a build working outside of the kernel tree for BPF programs.
The two major things that I learned were that first, the order in
which files are
included in the build command is more important than I previously thought.
The second thing was learning how clang deals with asm differently than gcc.
I had to use samples/bpf/asm_goto_workaround.h to fix those errors.
The meat of the makefile is as follows:

CLANGINC := /usr/lib/llvm-10/lib/clang/10.0.0/include
INC_FLAGS := -nostdinc -isystem $(CLANGINC)
EXTRA_FLAGS := -O3 -emit-llvm

linuxhdrs := /usr/src/linux-headers-$(shell uname -r)
LINUXINCLUDE := -include $(linuxhdrs)/include/linux/kconfig.h \
                                -include asm_workaround.h \
                                -I$(linuxhdrs)/arch/x86/include/ \
                                -I$(linuxhdrs)/arch/x86/include/uapi \
                                -I$(linuxhdrs)/arch/x86/include/generated \
                                -I$(linuxhdrs)/arch/x86/include/generated/uapi \
                                -I$(linuxhdrs)/include \
                                -I$(linuxhdrs)/include/uapi \
                                -I$(linuxhdrs)/include/generated/uapi \

COMPILERFLAGS := -D__KERNEL__ -D__ASM_SYSREG_H \
                                -D__BPF_TRACING__ -D__TARGET_ARCH_$(ARCH) \

# Builds all the targets from corresponding .c files
$(BPFOBJDIR)/%.o:$(BPFSRCDIR)/%.c
        $(CC) $(INC_FLAGS) $(COMPILERFLAGS) \
                $(LINUXINCLUDE) $(LIBBPF_HDRS) \
                $(EXTRA_FLAGS) -c $< -o - | $(LLC) -march=bpf
-filetype obj -o $@

I wanted to include that sample for whatever soul in the future wants
to tread the
same path with similar systems experience levels.
I still get about 100+ warnings when building that are the same as or
similar to:

/usr/src/linux-headers-5.4.0-26-generic/arch/x86/include/asm/atomic.h:194:9:
warning: unused variable '__ptr' [-Wunused-variable]
        return arch_cmpxchg(&v->counter, old, new);
                  ^

/usr/src/linux-headers-5.4.0-26-generic/arch/x86/include/asm/msr.h:100:26:
warning: variable 'low' is uninitialized when used here
[-Wuninitialized]
        return EAX_EDX_VAL(val, low, high);
                                                    ^~~

I suspect that these warnings come from my aggressive warning flags during
compilation rather than from actual issues in the kernel.

> Right, pinning map or program doesn't ensure that program is still
> attached to whatever BPF hook you attached to it. As you mentioned,
> XDP, tc, cgroup-bpf programs are persistent. We are actually moving
> towards the model of auto-detachment for those as well. See recent
> activity around bpf_link. The solution with bpf_link to make such
> attachments persistent is through pinning **link** itself, not program
> or map. bpf_link is relatively recent addition, so on older kernels
> you'd have to make sure you still have some process around that would
> keep BPF attachment FD around.


I have been looking at the commits surrounding the pinning of
bpf_link. It looks like it's only
working in kernel 5.7? I did actually go through and attempt to attach
links for kprobes,
tracepoints, and raw_tracepoints in kernel 5.4 but, as you suggested,
it seems unsupported.
I have yet to try on kernel 5.5-5.7 so I'll take a look this week or next.

As I mentioned before, with basic functionality in place here, I'm
interested in working on
some sort of BPF tutorial similar to the XDP tutorial
(https://github.com/xdp-project/xdp-tutorial)
with perhaps a more in-depth look at the technology included as well.

I'm still fuzzy on the relationship between bpf(2) and perf(1). Would
it be correct to say that for
tracepoints, kprobes, and uprobes BPF leverages perf "under the hood"
while for XDP and tc,
this is more like classic BPF in that it's implementation doesn't involve perf?
If that's the case then is the bpf_link object the tool to bridge BPF
and perf? I noticed that when
checking for pinned BPF programs with bpftool in kernel 5.4 that
unless a kprobe, tracepoint,
or uprobe is listed in "bpftool perf list", the program doesn't seem
to be running. Is the use of
perf to load BPF programs potentially a way to make them "headless"
instead of pinning the bpf_link objects?

Regardless, I'm excited to have a more reliable build system than I
have in the past. I think I'll start looking more into CO-RE and
libbpf on kernels 5.5-5.7.

Hope everyone is staying healthy out there,
Tristan

On Thu, May 14, 2020 at 5:51 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 11, 2020 at 10:06 AM <mayfieldtristan@gmail.com> wrote:
> >
> >
> > Hi all, hope everyone is staying healthy out there.
>
> Hi! For the future, I think cc'ing bpf@vger.kernel.org would be a good
> idea, there are a lot of folks who are probably not watching iovisor
> mailing list, but could help with issues like this.
>
> >
> > I've been working on building BPF programs, and have run into a few issues that I think might be clang (vs gcc) based.
> > It seems that either clang isn't the most friendly of compilers when it comes to building Linux-native programs, or my lack of experience makes it seem so.
> > I've been trying to build the simple BPF program below:
> >
> >
> > #include "bpf_helpers.h"
> > #include <linux/bpf.h>
> > #include <linux/version.h>
> > #include <linux/types.h>
> > #include <linux/tcp.h>
> > #include <net/sock.h>
> >
> > struct inet_sock_set_state_args {
> >         long long pad;
> > const void * skaddr;
> > int oldstate;
> > int newstate;
> > u16 sport;
> >   u16 dport;
> > u16 family;
> > u8 protocol;
> > u8 saddr[4];
> > u8 daddr[4];
> > u8 saddr_v6[16];
> > u8 daddr_v6[16];
> > };
> >
> >
> > SEC("tracepoint/sock/inet_sock_set_state")
> > int bpf_prog(struct inet_sock_set_state_args *args) {
> >
> >   struct sock *sk = (struct sock *)args->skaddr;
> >   short lport = args->sport;
> >
> >   char msg[] = "lport: %d\n";
> >   bpf_trace_printk(msg, sizeof(msg), lport);
> >
> >   return 0;
> > }
> >
> > char _license[] SEC("license") = "GPL";
> >
> >
> >
> > I've been looking through selftests/bpf/, samples/bpf/, and examples on various blogs and articles.
> > From this, I've come up with the following makefile:
> >
> >
> > ## Build tools
> > LLC := llc
> > CC := clang
> > HOSTCC := clang
> > CLANGINC := /usr/lib/llvm-10/lib/clang/10.0.0/include
> >
> > ## Some useful flags
> > INC_FLAGS := -nostdinc -isystem $(CLANGINC)
> > EXTRA_FLAGS := -O3 -emit-llvm
> >
> > ## Includes
> > linuxhdrs := /usr/src/linux-headers-$(shell uname -r)
> > LINUXINCLUDE := -include $(linuxhdrs)/include/linux/kconfig.h \
> > -include /usr/include/linux/bpf.h \
> > -I$(linuxhdrs)/arch/x86/include/ \
> > -I$(linuxhdrs)/arch/x86/include/uapi \
> > -I$(linuxhdrs)/arch/x86/include/generated \
> > -I$(linuxhdrs)/arch/x86/include/generated/uapi \
> > -I$(linuxhdrs)/include \
> > -I$(linuxhdrs)/include/uapi \
> > -I$(linuxhdrs)/include/generated/uapi \
> > LIBBPF :=  -I/home/vagrant/libbpf/src/
> > OBJS := tcptest.bpf.o
> >
> > $(OBJS): %.o:%.c
> > $(CC) $(INC_FLAGS) \
> > -target bpf -D__KERNEL__ -D __ASM_SYSREG_H \
> > -D__BPF_TRACING__ -D__TARGET_ARCH_$(ARCH) \
> > -Wno-unused-value -Wno-pointer-sign \
> > -Wno-compare-distinct-pointer-types \
> > -Wno-gnu-variable-sized-type-not-at-end \
> > -Wno-address-of-packed-member \
> > -Wno-tautological-compare \
> > -Wno-unknown-warning-option \
> > -Wall -v \
> > $(LINUXINCLUDE) $(LIBBPF) \
> > $(EXTRA_FLAGS) -c $< -o - | $(LLC) -march=bpf -filetype obj -o $
> >
> >
> > Unfortunately, I keep running into what seems to be asm errors. I've tried reorganizing the list of include statements, taking out "-target bpf", not including some files, including other files, etc etc.
> > This stackoverflow post suggests that it's a kconfig.h error, but I seem to be including the file just fine (https://stackoverflow.com/questions/56975861/error-compiling-ebpf-c-code-out-of-kernel-tree/56990939#56990939).
> > I'm not really sure where to go from here with building BPF programs and including files that have the kernel datatypes. Maybe I'm missing something that's obvious that I'm just ignorant of?
>
> I'd start with actually specifying what compilation errors you run
> into. Also check out
> https://github.com/iovisor/bcc/blob/master/libbpf-tools/Makefile to
> see how BPF programs can be compiled properly outside of kernel tree.
> Though that one pretty much assumes vmlinux.h, which simplifies a
> bunch of compilation issues, probably.
>
> >
> >
> >
> > As additional information, and regarding kernel persistence, I am working on a monitoring project that uses BPF programs to continuously monitor the system without the bulky dependencies that BCC includes. I'm concurrently working on a BTF/CO-RE solution but I'm emphasizing a non-CO-RE approach at the moment. I can load and run BPF programs but upon termination of my userspace loader the BPF programs themselves also terminate.
> >
> >
> >
> > I would like to have the BPF program persist in the kernel even after the user space loader has completed its execution. I read in various documentation and in a 2015 LWN article that persistent BPF programs can be created by pinning programs and maps to the BPF vfs so as to keep the fds open. I have attempted pinning the entire BPF object, various programs and various maps, and no matter what I've tried the kernel BPF program terminates when the userspace process terminates. Using bpftool I have verified that the BPF files are pinned to the location and that BPF programs themselves all work. I know that persistent BPF programs are a part of projects like XDP and tc. Is there a way to do this for a generic BPF loader without having to implement customized kernel functions?  Below I have included a simplified version of my code. In which I outline the basic steps I take to load the compiled bpf programs and attempt to make persistent instances of them.
>
> Right, pinning map or program doesn't ensure that program is still
> attached to whatever BPF hook you attached to it. As you mentioned,
> XDP, tc, cgroup-bpf programs are persistent. We are actually moving
> towards the model of auto-detachment for those as well. See recent
> activity around bpf_link. The solution with bpf_link to make such
> attachments persistent is through pinning **link** itself, not program
> or map. bpf_link is relatively recent addition, so on older kernels
> you'd have to make sure you still have some process around that would
> keep BPF attachment FD around.
>
>
> >
> >
> >
> > #include <stdio.h>
> >
> > #include <stdlib.h>
> >
> > #include <string.h>
> >
> > #include <errno.h>
> >
> > #include <getopt.h>
> >
> > #include <dirent.h>
> >
> > #include <sys/stat.h>
> >
> > #include <unistd.h>
> >
> > #include <assert.h>
> >
> > #include <linux/version.h>
> >
> >
> >
> > #include "libbpf.h"
> >
> > #include "bpf.h"
> >
> > #include "loader_helpers.h"
> >
> >
> >
> > #include <stdbool.h>
> >
> > #include <fcntl.h>
> >
> > #include <poll.h>
> >
> > #include <linux/perf_event.h>
> >
> > #include <assert.h>
> >
> > #include <sys/syscall.h>
> >
> > #include <sys/ioctl.h>
> >
> > #include <sys/mman.h>
> >
> > #include <time.h>
> >
> > #include <signal.h>
> >
> > #include <linux/ptrace.h>
> >
> >
> >
> > int main(int argc, char **argv) {
> >
> >
> >
> >     struct bpf_object *bpf_obj;
> >
> >     struct bpf_program *bpf_prog;
> >
> >     struct bpf_map *map;
> >
> >     char * license = "GPL";
> >
> >     __u32 kernelvers = LINUX_VERSION_CODE;
> >
> >     struct bpf_link * link;
> >
> >     int err;
> >
> >     int prog_fd;
> >
> >
> >
> >     bpf_obj = bpf_object__open("test_file.bpf.o");
> >
> >
> >
> >     bpf_prog = bpf_program__next(NULL, bpf_obj);
> >
> >
> >
> >     err = bpf_program__set_tracepoint(bpf_prog);
> >
> >     if(err) {
> >
> >         fprintf(stderr, "ERR couldn't setup program type\n");
> >
> >         return -1;
> >
> >     }
> >
> >     err = bpf_program__load(bpf_prog, license, kernelvers);
> >
> >     if(err) {
> >
> >         fprintf(stderr, "ERR couldn't setup program phase\n");
> >
> >         return -1;
> >
> >     }
> >
> >     prog_fd = bpf_program__fd(bpf_prog);
> >
> >
> >
> >     link = bpf_program__attach_tracepoint(bpf_prog, "syscalls", "sys_enter_openat");
> >
> >     if(!link) {
> >
> >         fprintf(stderr, "ERROR ATTACHING TRACEPOINT\n");
> >
> >         return -1;
> >
> >     }
> >
> >
> >
> >     assert(bpf_program__is_tracepoint(bpf_prog));
> >
> >
> >
> > pin:
> >
> >     err = bpf_program__pin(bpf_prog, "/sys/fs/bpf/tpprogram");
> >
> >     if(err) {
> >
> >         if(err == -17) {
> >
> >             printf("Program exists...trying to unpin and retry!\n");
> >
> >             err = bpf_program__unpin(bpf_prog, "/sys/fs/bpf/tpprogram");
> >
> >             if(!err) {
> >
> >                 goto pin;
> >
> >            }
> >
> >             printf("The pining already exists but it couldn't be removed...\n");
> >
> >             return -1;
> >
> >         }
> >
> >         printf("We couldn't pin...%d\n", err);
> >
> >         return -1;
> >
> >     }
> >
> >
> >
> >     printf("Program pinned and working...\n");
> >
> >
> >
> >     return 0;
> >
> > }
> >
> >
> >
> >
> > Thanks for having a look and I hope these issues can be cleared up. Seems like building is the last major hurdle I have to get rolling with better engineering solutions than manually including structs in my files.
> > Hope everyone stays well!
>
>
> Hope above helped. Please cc bpf@vger.kernel.org (and ideally send
> plain-text emails, kernel mailing lists don't accept HTML emails).
>
> >
> > _._,_._,_
> > ________________________________
> > Links:
> >
> > You receive all messages sent to this group.
> >
> > View/Reply Online (#1847) | Reply To Sender | Reply To Group | Mute This Topic | New Topic
> >
> > Your Subscription | Contact Group Owner | Unsubscribe [andrii.nakryiko@gmail.com]
> >
> > _._,_._,_

Re: [iovisor-dev] Building BPF programs and kernel persistence

[Andrii Nakryiko]

On Mon, May 18, 2020 at 9:23 AM Tristan Mayfield
<mayfieldtristan@gmail.com> wrote:
>
> Thanks for the reply Andrii.
>
> Managed to get a build working outside of the kernel tree for BPF programs.
> The two major things that I learned were that first, the order in which files are
> included in the build command is more important than I previously thought.
> The second thing was learning how clang deals with asm differently than gcc.
> I had to use samples/bpf/asm_goto_workaround.h to fix those errors.
> The meat of the makefile is as follows:
>
> CLANGINC := /usr/lib/llvm-10/lib/clang/10.0.0/include
> INC_FLAGS := -nostdinc -isystem $(CLANGINC)
> EXTRA_FLAGS := -O3 -emit-llvm

BTW, everyone seems to be using -O2 for compiling BPF programs. Not
sure how well-supported -O3 will be.

[...]

>
> I suspect that these warnings come from my aggressive warning flags during
> compilation rather than from actual issues in the kernel.
>
>> Right, pinning map or program doesn't ensure that program is still
>> attached to whatever BPF hook you attached to it. As you mentioned,
>> XDP, tc, cgroup-bpf programs are persistent. We are actually moving
>> towards the model of auto-detachment for those as well. See recent
>> activity around bpf_link. The solution with bpf_link to make such
>> attachments persistent is through pinning **link** itself, not program
>> or map. bpf_link is relatively recent addition, so on older kernels
>> you'd have to make sure you still have some process around that would
>> keep BPF attachment FD around.
>
>
> I have been looking at the commits surrounding the pinning of bpf_link. It looks like it's only
> working in kernel 5.7? I did actually go through and attempt to attach links for kprobes,
> tracepoints, and raw_tracepoints in kernel 5.4 but, as you suggested, it seems unsupported.
> I have yet to try on kernel 5.5-5.7 so I'll take a look this week or next.
>
> As I mentioned before, with basic functionality in place here, I'm interested in working on
> some sort of BPF tutorial similar to the XDP tutorial (https://github.com/xdp-project/xdp-tutorial)
> with perhaps a more in-depth look at the technology included as well.
>
> I'm still fuzzy on the relationship between bpf(2) and perf(1). Would it be correct to say that for
> tracepoints, kprobes, and uprobes BPF leverages perf "under the hood" while for XDP and tc,
> this is more like classic BPF in that it's implementation doesn't involve perf?

"classic BPF" is entirely different thing, don't use that term in this
context, it will just confuse people.

perf is used as a means to trigger BPF program execution for
tracepoint and kprobes. It is, essentially, a BPF hook provider, if
you will. For XDP, BPF hook is provided by networking layer and
drivers. For cgroup BPF programs, hooks are "provided", in a sense, by
cgroup subsystem. So perf is just one of many ways to specify where
and when BPF program is going to be executed, and with what context.

> If that's the case then is the bpf_link object the tool to bridge BPF and perf? I noticed that when
> checking for pinned BPF programs with bpftool in kernel 5.4 that unless a kprobe, tracepoint,
> or uprobe is listed in "bpftool perf list", the program doesn't seem to be running. Is the use of
> perf to load BPF programs potentially a way to make them "headless" instead of pinning the bpf_link objects?

no, bpf_link is a way to marry BPF hook with BPF program. It's not
specific to perf or XDP, or whatever. Actually, right now perf-based
BPF hooks (kprobe, tracepoint) actually do not create a bpf_link under
cover, so you won't be able to pin them.


>
> Regardless, I'm excited to have a more reliable build system than I have in the past. I think I'll start looking more into CO-RE and libbpf on kernels 5.5-5.7.
>

Awesome, have fun!

> Hope everyone is staying healthy out there,
> Tristan
>