Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

[Toke Høiland-Jørgensen]

Hi everyone

As you have no doubt noticed, we have started thinking about how to
package eBPF-related applications in distributions. As a part of this,
I've been thinking about what to recommend for applications that ship
pre-compiled BPF byte-code files.

The obvious place to place those would be somewhere in the system
$LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But
since BPF byte code is its own binary format, different from regular
executables, I think having a separate path to put those under makes
sense. So I'm proposing to establish a convention that pre-compiled BPF
programs be installed into /usr/lib{,64}/bpf.

This would let users discover which BPF programs are shipped on their
system, and it could be used to discover which package loaded a
particular BPF program, by walking the directory to find the file a
loaded program came from. It would not work for dynamically-generated
bytecode, of course, but I think at least some applications will end up
shipping pre-compiled bytecode files (we're doing that for xdp-tools,
for instance).

As I said, this would be a convention. We're already using it for
xdp-tools[0], so my plan is to use that as the "first mover", try to get
distributions to establish the path as a part of their filesystem
layout, and then just try to encourage packages to use it. Hopefully it
will catch on.

Does anyone have any objections to this? Do you think it is a complete
waste of time, or is it worth giving it a shot? :)

-Toke

[0] https://github.com/xdp-project/xdp-tools/blob/master/lib/defines.mk#L12

Re: Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

[Alexei Starovoitov]

On Mon, Dec 09, 2019 at 12:29:27PM +0100, Toke Høiland-Jørgensen wrote:
> Hi everyone
> 
> As you have no doubt noticed, we have started thinking about how to
> package eBPF-related applications in distributions. As a part of this,
> I've been thinking about what to recommend for applications that ship
> pre-compiled BPF byte-code files.
> 
> The obvious place to place those would be somewhere in the system
> $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But
> since BPF byte code is its own binary format, different from regular
> executables, I think having a separate path to put those under makes
> sense. So I'm proposing to establish a convention that pre-compiled BPF
> programs be installed into /usr/lib{,64}/bpf.
> 
> This would let users discover which BPF programs are shipped on their
> system, and it could be used to discover which package loaded a
> particular BPF program, by walking the directory to find the file a
> loaded program came from. It would not work for dynamically-generated
> bytecode, of course, but I think at least some applications will end up
> shipping pre-compiled bytecode files (we're doing that for xdp-tools,
> for instance).
> 
> As I said, this would be a convention. We're already using it for
> xdp-tools[0], so my plan is to use that as the "first mover", try to get
> distributions to establish the path as a part of their filesystem
> layout, and then just try to encourage packages to use it. Hopefully it
> will catch on.
> 
> Does anyone have any objections to this? Do you think it is a complete
> waste of time, or is it worth giving it a shot? :)

What will be the name of file/directory ?
What is going to be the mechanism to clean it up?
What will be stored in there? Just .o files ?

libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/
It was useful for debugging. Since TAG is used as directory name
reloading the same bcc script creates the same dir and /var/tmp
periodically gets cleaned by reboot.

Installing bpf .o into common location feels useful. Not sure though
how you can convince folks to follow such convention.
That was the main problem with libbcc. Not everything is using that lib.
So teaching folks who debug bpf in production to look into /var/tmp/bcc
wasn't enough. 'bpftool p s' is still the main mechanism.
Some C++ services embed bpf .o as part of x86 binary and that binary
is installed. They wouldn't want to split bpf .o into separate file
since it will complicate dependency management, risk conflicts, etc.
Just food for thought.

Re: Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

[Jesper Dangaard Brouer]

On Mon, 9 Dec 2019 17:40:19 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Mon, Dec 09, 2019 at 12:29:27PM +0100, Toke Høiland-Jørgensen wrote:
> > Hi everyone
> > 
> > As you have no doubt noticed, we have started thinking about how to
> > package eBPF-related applications in distributions. As a part of this,
> > I've been thinking about what to recommend for applications that ship
> > pre-compiled BPF byte-code files.
> > 
> > The obvious place to place those would be somewhere in the system
> > $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But
> > since BPF byte code is its own binary format, different from regular
> > executables, I think having a separate path to put those under makes
> > sense. So I'm proposing to establish a convention that pre-compiled BPF
> > programs be installed into /usr/lib{,64}/bpf.
> > 
> > This would let users discover which BPF programs are shipped on their
> > system, and it could be used to discover which package loaded a
> > particular BPF program, by walking the directory to find the file a
> > loaded program came from. It would not work for dynamically-generated
> > bytecode, of course, but I think at least some applications will end up
> > shipping pre-compiled bytecode files (we're doing that for xdp-tools,
> > for instance).
> > 
> > As I said, this would be a convention. We're already using it for
> > xdp-tools[0], so my plan is to use that as the "first mover", try to get
> > distributions to establish the path as a part of their filesystem
> > layout, and then just try to encourage packages to use it. Hopefully it
> > will catch on.
> > 
> > Does anyone have any objections to this? Do you think it is a complete
> > waste of time, or is it worth giving it a shot? :)  
> 
> What will be the name of file/directory ?
> What is going to be the mechanism to clean it up?
> What will be stored in there? Just .o files ?
> 
> libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/
> It was useful for debugging. Since TAG is used as directory name
> reloading the same bcc script creates the same dir and /var/tmp
> periodically gets cleaned by reboot.
> 
> Installing bpf .o into common location feels useful. Not sure though
> how you can convince folks to follow such convention.

I imagine the files under /usr/lib{,64}/bpf/ will be pre-compiled
binaries (fairly static file).  These will be delivered together with
the distro RPM file. The distro will detect/enforce that two packages
cannot use the same name for bpf .o files.

I see these files as part of the RPM software package binary files. In
my opinion, this means/imply that the BPF application should not update
these files runtime, because different consistency checksum tools
should be able to verify that this files comes from the original RPM
file.

More below on dynamic files.

> That was the main problem with libbcc. Not everything is using that lib.
> So teaching folks who debug bpf in production to look into /var/tmp/bcc
> wasn't enough. 'bpftool p s' is still the main mechanism.
> Some C++ services embed bpf .o as part of x86 binary and that binary
> is installed. They wouldn't want to split bpf .o into separate file
> since it will complicate dependency management, risk conflicts, etc.
> Just food for thought.

I see three different types of BPF-object files, which belong in
different places (suggestion in parentheses):

 1. Pre-compiled binaries via RPM. (/usr/lib? [1])
 2. Application "startup" compiled "cached" BPF-object (/var/cache? [2]).
 3. Runtime dynamic compiled BPF-objects short lived (/run? [3])

You can follow the links below, to see if match descriptions in
the Filesystem Hierarchy Standard[4].

I think that filetype 1 + 2 makes sense to store in files. For
filetype 3 (the highly dynamic runtime re-compiled files) I'm not
sure it makes sense to store those in any central place.  Applications
could use /run/application-name/, but it will be a pain to deal with
filename-clashes. As Alexei brings up cleanup; /run/ is cleared at the
beginning of the boot process[3].

For fileytpe 2, I suggest /var/cache/bpf/, but with an additional
application name as a subdir, this is to avoid name clashes (which then
becomes the applications responsibility with in its own dir).


Links:

[1] /usr/lib: "Libraries for programming and packages" https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s06.html

[2] /var/cache: "Application cache data" https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html

[3] /run: "Run-time variable data" https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s15.html

[4] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

[Toke Høiland-Jørgensen]

Jesper Dangaard Brouer <brouer@redhat.com> writes:

> On Mon, 9 Dec 2019 17:40:19 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
>> On Mon, Dec 09, 2019 at 12:29:27PM +0100, Toke Høiland-Jørgensen wrote:
>> > Hi everyone
>> > 
>> > As you have no doubt noticed, we have started thinking about how to
>> > package eBPF-related applications in distributions. As a part of this,
>> > I've been thinking about what to recommend for applications that ship
>> > pre-compiled BPF byte-code files.
>> > 
>> > The obvious place to place those would be somewhere in the system
>> > $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But
>> > since BPF byte code is its own binary format, different from regular
>> > executables, I think having a separate path to put those under makes
>> > sense. So I'm proposing to establish a convention that pre-compiled BPF
>> > programs be installed into /usr/lib{,64}/bpf.
>> > 
>> > This would let users discover which BPF programs are shipped on their
>> > system, and it could be used to discover which package loaded a
>> > particular BPF program, by walking the directory to find the file a
>> > loaded program came from. It would not work for dynamically-generated
>> > bytecode, of course, but I think at least some applications will end up
>> > shipping pre-compiled bytecode files (we're doing that for xdp-tools,
>> > for instance).
>> > 
>> > As I said, this would be a convention. We're already using it for
>> > xdp-tools[0], so my plan is to use that as the "first mover", try to get
>> > distributions to establish the path as a part of their filesystem
>> > layout, and then just try to encourage packages to use it. Hopefully it
>> > will catch on.
>> > 
>> > Does anyone have any objections to this? Do you think it is a complete
>> > waste of time, or is it worth giving it a shot? :)  
>> 
>> What will be the name of file/directory ?
>> What is going to be the mechanism to clean it up?
>> What will be stored in there? Just .o files ?
>> 
>> libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/
>> It was useful for debugging. Since TAG is used as directory name
>> reloading the same bcc script creates the same dir and /var/tmp
>> periodically gets cleaned by reboot.
>> 
>> Installing bpf .o into common location feels useful. Not sure though
>> how you can convince folks to follow such convention.
>
> I imagine the files under /usr/lib{,64}/bpf/ will be pre-compiled
> binaries (fairly static file).  These will be delivered together with
> the distro RPM file. The distro will detect/enforce that two packages
> cannot use the same name for bpf .o files.

Yes, that was my intention. Packages can choose whether to create a
subdirectory, or just dump files in /usr/lib{,64}/bpf (this is similar
to /usr/lib).

> I see three different types of BPF-object files, which belong in
> different places (suggestion in parentheses):
>
>  1. Pre-compiled binaries via RPM. (/usr/lib? [1])
>  2. Application "startup" compiled "cached" BPF-object (/var/cache? [2]).
>  3. Runtime dynamic compiled BPF-objects short lived (/run? [3])
>
> You can follow the links below, to see if match descriptions in
> the Filesystem Hierarchy Standard[4].
>
> I think that filetype 1 + 2 makes sense to store in files. For
> filetype 3 (the highly dynamic runtime re-compiled files) I'm not
> sure it makes sense to store those in any central place.  Applications
> could use /run/application-name/, but it will be a pain to deal with
> filename-clashes. As Alexei brings up cleanup; /run/ is cleared at the
> beginning of the boot process[3].
>
> For fileytpe 2, I suggest /var/cache/bpf/, but with an additional
> application name as a subdir, this is to avoid name clashes (which then
> becomes the applications responsibility with in its own dir).

/var/cache/bpf seems reasonable, let's go with that. My plan is to try
to get the directories established in distribution packaging
('filesystem' on Arch and Fedora, 'base-files' on Debian); this will
mean the directories are already there on people's systems, which
hopefully will encourage developers to use them. And then we can try to
provide a bit more nudging through the distribution packaging.

-Toke