Re: bpf libraries and static variables. Was: [PATCH v2 bpf-next 2/6] libbpf: rename static variables during linking

[Alexei Starovoitov <alexei.starovoitov@gmail.com>]

On Thu, May 13, 2021 at 09:37:13AM +0100, Lorenz Bauer wrote:
> On Wed, 12 May 2021 at 19:50, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> 
> ...
> 
> > So at least for BPF skeleton, the flow I was imagining would be
> > like this.
> 
> Thank you for the worked out example, it's really helpful.
> 
> >
> > 1. BPF library abc consists of abc1.bpf.c and abc2.bpf.c. It also has
> > user-space component in abc.c.
> > 2. BPF app uses abs library and has its own app1.bpf.c and app2.bpf.c
> > and app.c for user-space.
> > 3. BPF library author sets up its Makefile to do
> >   a. clang -target bpf -g -O2 -c abc1.bpf.c -o abc1.bpf.o
> >   b. clang -target bpf -g -O2 -c abc2.bpf.c -o abc2.bpf.o
> >   c. bpftool gen lib libabc.bpf.o abc1.bpf.o abc2.bpf.o
> 
> I think we can plug this into bpf2go [1] on our side in the best case,
> which would avoid duplicating the static linker.
> 
> >   d. bpftool gen subskeleton libabc.bpf.o > libabc.subskel.h
> >   e. abc.c (user-space library) is of the form
> >
> > #include "libabc.subskel.h"
> >
> > static struct libabc_bpf *subskel;
> >
> > int libabc__init(struct bpf_object *obj)
> > {
> >     subskel = libabc_bpf__open_subskel(obj);
> >
> >     subskel->data->abc_my_var = 123;
> > }
> >
> > int libabc__attach()
> > {
> >     libabc_bpf__attach(subskel);
> > }
> >
> >   f. cc abc.c into libabc.a and then libabc.a and libabc.bpf.o are
> > distributed to end user
> >
> > 3. Now, from BPF application author side:
> >   a. clang -target bpf -g -O2 -c app1.bpf.c -o app1.bpf.o
> >   b. clang -target bpf -g -O2 -c app2.bpf.c -o app2.bpf.o
> >   c. bpftool gen object app.bpf.o app1.bpf.o app2.bpf.o libabc.bpf.o
> 
> I haven't worked out exactly how things would work, but on the Go side
> it might be possible to distribute libabc.bpf.o plus the Go "library"
> code as a package. So the Go toolchain would never create this merged
> object, but instead do
> 
>     bpftool gen object app.bpf.o app1.bpf.o app2.bpf.o
> 
> and later link app.bpf.o and libabc.bpf.o at runtime. It would be
> simpler from our side if bpftool gen object could link both libraries
> and "programs", maybe we can discuss the details of this during office
> hours.
> 
> 1: https://pkg.go.dev/github.com/cilium/ebpf/cmd/bpf2go

This is really cool! Looks like libbpf C skeleton that embeds the full
elf file as a string into golang object, but it doesn't support
direct data/rodata/bss variable access yet? Looks like only progs and maps
are natively exposed into golang?

Have you seen the light skeleton yet?
https://patchwork.kernel.org/project/netdevbpf/patch/20210512213256.31203-18-alexei.starovoitov@gmail.com/
It's design centered on supporting languages like golang that
cannot take libbpf in C as-is. I think reimplementation of everything
in every other language that doesn't have clean binding to C is going
to hurt BPF ecosystem long term. The light skeleton is designed to address that.
It's libbpf-less. We're going to teach bpftool to emit golang equivalent of .lskel.h
Here is trace_printk.lskel.h example:
https://gist.github.com/4ast/774ea58f8286abac6aa8e3bf3bf3b903
The hex string dumps in there are not elf file anymore.
They're blobs directly interpreted by the kernel.
Note the headers:
#include <bpf/bpf.h>
#include <bpf/skel_internal.h>
The light skeleton is using only three sys_bpf wrappers (map_create, prog_load, test_run).
While the end result looks and feels like existing skeleton.
It doesn't need libelf and doesn't need all of libbpf to load and attach progs.
The plan is to take cilium bpf progs and represent them lskel.h tests in selftests/bpf
to demonstrate the feature richness.

The work on llvm's ld.lld linker is in progress as well:
https://reviews.llvm.org/D101336
The users will be able to:
clang -target bpf -flto -O2 -g -c t1.c -o t1.bc
clang -target bpf -flto -O2 -g -c t2.c -o t2.bc
ld.lld -r t1.bc t2.bc -o final.o

Such llvm linking step can only happen once, since it's an LTO compilation.
To use it with libraries the users would need to:
// compile lib files
clang -target bpf -flto -g -O2 -c abc1.bpf.c -o abc1.bpf.o
clang -target bpf -flto -g -O2 -c abc2.bpf.c -o abc2.bpf.o
// compile app files
clang -target bpf -flto -g -O2 -c app1.bpf.c -o app1.bpf.o
clang -target bpf -flto -g -O2 -c app2.bpf.c -o app2.bpf.o
// link and LTO-compile everything
ld.lld -r abc1.bpf.o abc2.bpf.o app1.bpf.o app2.bpf.o -o final.o

Obviously we still need libbpf static linker for linking true .o-s
when LTO is not suitable.