libbpf pinning strategies - towards v1

libbpf pinning strategies - towards v1

[Gilad Reti]

As libbpf is heading towards a first major release, we wanted to
discuss libbpf's object pinning strategy.

bpf object pinning has a couple of use cases (feel free to add, there
are more for sure):
1. Sharing specific bpf objects between different processes (for
example, one process loads a bpf skeleton, another one interacts with
it using various bpf maps (for example, for changing configurations
(i.e. dynamic networking rules etc))
2. Preventing bpf objects from destruction upon owning process exit
(i.e. to prevent bpf progs detach upon userspace program crash)

Regarding the first use case, for most cases manually setting the pin
path (both in the loading process and in other processes) will
probably be the best. In such cases, no redesign is required here.

For the second one, something like the bpf_object__pin will be more
appropriate (to allow a complete reuse of the bpf objects). For that
use case, some sensible requirements we can consider are:

1. Paths should be unique:
    a. at the bpf_object level (that is, same pinnable objects that
belong to different bpf_object s should be pinned at different paths).
    b. in the same bpf_object, between different pinnable object types
(i.e. a map and a prog) should always be pinned at different paths.
    c. different objects, belonging to the same bpf_object and of the
same type should be pinned at different paths.
2. Paths should be predictable, given enough information on the
originating bpf_object (that is, adding random UID to ensure
uniqueness is not an option).

All the above should be applied to auto-pinned maps and the
bpf_object__pin function. I am not sure if the
bpf_object__pin_{maps,programs} should conform to those requirements
too. Of course, all paths should be overridable similarly to the
current implementation.

Regarding implementation, 1.c. will already be satisfied by the
current implementation (after the program name pinning path will be
changed, since both map names and function names are unique inside a
single object).
For 1.a and 1.b, I think that bpf_object__pin should produce the
following directory layout:

<obj_name>
├── maps
│      └── <map_name>
└── programs
        └── <program_name>

If we decide that the requirements should apply to the specific
bpf_object__pin_<type>s variants, then each will produce

<obj_name>
└── <type>s (i.e. maps, programs)
        └── <name>

It may be better to put all pinned objects under a objects/ directory
too, I am not sure about that.

As a last point, I think that it will be nice to have a way to pin a
bpf_object_skeleton. This will be an improvement over the current
bpf_object__pin since skeletons keep track of attached links.

There are more use cases I am not familiar with for sure, so I would
like to hear other's opinions and comments.

Re: libbpf pinning strategies - towards v1

[Andrii Nakryiko]

On Sun, Mar 14, 2021 at 8:40 AM Gilad Reti <gilad.reti@gmail.com> wrote:
>
> As libbpf is heading towards a first major release, we wanted to
> discuss libbpf's object pinning strategy.
>
> bpf object pinning has a couple of use cases (feel free to add, there
> are more for sure):
> 1. Sharing specific bpf objects between different processes (for
> example, one process loads a bpf skeleton, another one interacts with
> it using various bpf maps (for example, for changing configurations
> (i.e. dynamic networking rules etc))
> 2. Preventing bpf objects from destruction upon owning process exit
> (i.e. to prevent bpf progs detach upon userspace program crash)
>
> Regarding the first use case, for most cases manually setting the pin
> path (both in the loading process and in other processes) will
> probably be the best. In such cases, no redesign is required here.
>
> For the second one, something like the bpf_object__pin will be more
> appropriate (to allow a complete reuse of the bpf objects). For that
> use case, some sensible requirements we can consider are:
>
> 1. Paths should be unique:
>     a. at the bpf_object level (that is, same pinnable objects that
> belong to different bpf_object s should be pinned at different paths).
>     b. in the same bpf_object, between different pinnable object types
> (i.e. a map and a prog) should always be pinned at different paths.
>     c. different objects, belonging to the same bpf_object and of the
> same type should be pinned at different paths.
> 2. Paths should be predictable, given enough information on the
> originating bpf_object (that is, adding random UID to ensure
> uniqueness is not an option).
>
> All the above should be applied to auto-pinned maps and the
> bpf_object__pin function. I am not sure if the
> bpf_object__pin_{maps,programs} should conform to those requirements
> too. Of course, all paths should be overridable similarly to the
> current implementation.

I actually think that bpf_object__pin_maps and
bpf_object__pin_programs should be removed.
bpf_object__pin()/bpf_object__unpin() and then per-map and per-program
API to control their pinning parameters should be enough to handle all
the cases.

>
> Regarding implementation, 1.c. will already be satisfied by the
> current implementation (after the program name pinning path will be
> changed, since both map names and function names are unique inside a
> single object).

That's going to change with BPF static linking. I'm thinking about
supporting static maps, i.e., maps visible within a single BPF .o
file, but still visible to outside world. At that point, each .o file
should be able to have conflicting map name, just as you'd expect to
have conflicting static variables and static functions. I haven't
thought yet all that is going to be expose  to user-space, though.

> For 1.a and 1.b, I think that bpf_object__pin should produce the
> following directory layout:
>
> <obj_name>
> ├── maps
> │      └── <map_name>
> └── programs
>         └── <program_name>
>
> If we decide that the requirements should apply to the specific
> bpf_object__pin_<type>s variants, then each will produce
>
> <obj_name>
> └── <type>s (i.e. maps, programs)
>         └── <name>
>
> It may be better to put all pinned objects under a objects/ directory
> too, I am not sure about that.

seems a bit of an overkill, first-level directory for an object seems nice

>
> As a last point, I think that it will be nice to have a way to pin a
> bpf_object_skeleton. This will be an improvement over the current
> bpf_object__pin since skeletons keep track of attached links.

Hm.. that's the first time this comes up. You mean that all the
created bpf_links (stored inside skel->links) will be pinned in such a
case? Those links would probably go under <obj_name/links/ directory,
right? Would we then need to generate something like
my_skeleton__load_pinned(), which would be called instead of
my_skeleton__load()?

>
> There are more use cases I am not familiar with for sure, so I would
> like to hear other's opinions and comments.

Yes, absolutely, I'd like to hear some more use cases as well.

I think we need to discuss more on how to manage pinning settings for
maps (including .data, .rodata, etc) and programs. Another aspect that
is rarely discussed but is important is compatibility and
upgradeability. I.e., what if pinned map is not exactly the same as
the one you expect in your BPF code (e.g., map value size increased,
etc). This is especially important for .data, .rodata special maps, as
BPF program code will reference variables through compiled-in offsets.
For such cases we'd need to validate that all expected/used variables
are still at the same place and have the same (or compatible?) sizes.

In short, there is a lot more subtlety to pinning that meets the eye,
which is why I hope that more people will get involved in the
discussion. I personally never had a use for pinning, so for me it's
hard to judge what's important in practice. But I do see a lot of
ambiguity and potential problems with re-using BPF maps and BPF
programs :)

Re: libbpf pinning strategies - towards v1

[Gilad Reti]

On Tue, Mar 16, 2021 at 7:55 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sun, Mar 14, 2021 at 8:40 AM Gilad Reti <gilad.reti@gmail.com> wrote:
> >
> > As libbpf is heading towards a first major release, we wanted to
> > discuss libbpf's object pinning strategy.
> >
> > bpf object pinning has a couple of use cases (feel free to add, there
> > are more for sure):
> > 1. Sharing specific bpf objects between different processes (for
> > example, one process loads a bpf skeleton, another one interacts with
> > it using various bpf maps (for example, for changing configurations
> > (i.e. dynamic networking rules etc))
> > 2. Preventing bpf objects from destruction upon owning process exit
> > (i.e. to prevent bpf progs detach upon userspace program crash)
> >
> > Regarding the first use case, for most cases manually setting the pin
> > path (both in the loading process and in other processes) will
> > probably be the best. In such cases, no redesign is required here.
> >
> > For the second one, something like the bpf_object__pin will be more
> > appropriate (to allow a complete reuse of the bpf objects). For that
> > use case, some sensible requirements we can consider are:
> >
> > 1. Paths should be unique:
> >     a. at the bpf_object level (that is, same pinnable objects that
> > belong to different bpf_object s should be pinned at different paths).
> >     b. in the same bpf_object, between different pinnable object types
> > (i.e. a map and a prog) should always be pinned at different paths.
> >     c. different objects, belonging to the same bpf_object and of the
> > same type should be pinned at different paths.
> > 2. Paths should be predictable, given enough information on the
> > originating bpf_object (that is, adding random UID to ensure
> > uniqueness is not an option).
> >
> > All the above should be applied to auto-pinned maps and the
> > bpf_object__pin function. I am not sure if the
> > bpf_object__pin_{maps,programs} should conform to those requirements
> > too. Of course, all paths should be overridable similarly to the
> > current implementation.
>
> I actually think that bpf_object__pin_maps and
> bpf_object__pin_programs should be removed.
> bpf_object__pin()/bpf_object__unpin() and then per-map and per-program
> API to control their pinning parameters should be enough to handle all
> the cases.
>

Sure. I personally couldn't find any usecase for exposing them, and if
there is no such one then I am totally okay with that.

> >
> > Regarding implementation, 1.c. will already be satisfied by the
> > current implementation (after the program name pinning path will be
> > changed, since both map names and function names are unique inside a
> > single object).
>
> That's going to change with BPF static linking. I'm thinking about
> supporting static maps, i.e., maps visible within a single BPF .o
> file, but still visible to outside world. At that point, each .o file
> should be able to have conflicting map name, just as you'd expect to
> have conflicting static variables and static functions. I haven't
> thought yet all that is going to be expose  to user-space, though.
>
> > For 1.a and 1.b, I think that bpf_object__pin should produce the
> > following directory layout:
> >
> > <obj_name>
> > ├── maps
> > │      └── <map_name>
> > └── programs
> >         └── <program_name>
> >
> > If we decide that the requirements should apply to the specific
> > bpf_object__pin_<type>s variants, then each will produce
> >
> > <obj_name>
> > └── <type>s (i.e. maps, programs)
> >         └── <name>
> >
> > It may be better to put all pinned objects under a objects/ directory
> > too, I am not sure about that.
>
> seems a bit of an overkill, first-level directory for an object seems nice
>
> >
> > As a last point, I think that it will be nice to have a way to pin a
> > bpf_object_skeleton. This will be an improvement over the current
> > bpf_object__pin since skeletons keep track of attached links.
>
> Hm.. that's the first time this comes up. You mean that all the
> created bpf_links (stored inside skel->links) will be pinned in such a
> case? Those links would probably go under <obj_name/links/ directory,
> right? Would we then need to generate something like
> my_skeleton__load_pinned(), which would be called instead of
> my_skeleton__load()?
>

Yes, something like that.

> >
> > There are more use cases I am not familiar with for sure, so I would
> > like to hear other's opinions and comments.
>
> Yes, absolutely, I'd like to hear some more use cases as well.
>
> I think we need to discuss more on how to manage pinning settings for
> maps (including .data, .rodata, etc) and programs. Another aspect that
> is rarely discussed but is important is compatibility and
> upgradeability. I.e., what if pinned map is not exactly the same as
> the one you expect in your BPF code (e.g., map value size increased,
> etc). This is especially important for .data, .rodata special maps, as
> BPF program code will reference variables through compiled-in offsets.
> For such cases we'd need to validate that all expected/used variables
> are still at the same place and have the same (or compatible?) sizes.
>
> In short, there is a lot more subtlety to pinning that meets the eye,
> which is why I hope that more people will get involved in the
> discussion. I personally never had a use for pinning, so for me it's
> hard to judge what's important in practice. But I do see a lot of
> ambiguity and potential problems with re-using BPF maps and BPF
> programs :)