libbpf: the road to v1.0

libbpf: the road to v1.0

[Andrii Nakryiko]

Hi all,

So I've been ruminating on getting libbpf to 1.0 version for a while
now and finally got to write down most of the major (and not so much)
things I wanted to change and/or break, given v1.0 gives an
opportunity to break API/ABI compatibility, where necessary.

I decided to share this with the community in the form of a Google Doc
(check [0]), open to commenting by anyone, because there are many
different things, quite often completely independent from each other.
So email doesn't feel like the right medium to have a discussion given
the amount of people that might be interested about just pieces of the
plan.

The overarching idea is to streamline APIs, make them overall more
consistent throughout, as well as eliminate some very common pitfalls.
Any such changes means potentially more pain for existing users during
the transition period. I realize that, but I hope everyone will keep
in mind the longer term goal of making libbpf easier to use both for
the new and existing users.

My intent is to spend some time in discussions and see what I have
missed and what would be argued to be too drastic/problematic. So the
plan is not set in stone and can/will be adjusted (within reason, of
course, I don't believe everyone is going to converge and be happy
about all the changes, but that's OK). But once decided on the plan, I
(and hopefully others will help) will start implementing changes, will
probably create a wiki page documenting API migration paths, etc, etc.
My current thinking is to do a gradual transition, rather than a big
bang breaking change in 1.0 release. This should give people more time
to find any possible breakages and adopt their code base gradually, so
that by the 1.0 time there isn't much surprise left. But feel free to
argue the other way.

BTW, that document is only describing potentially breaking changes, it
doesn't mean libbpf won't get any other new functionality. I still
plan a bunch of other (new) features to be added before v1.0. E.g.,
stuff like BPF static linking support (merging together multiple BPF
.o files) and declarative PROG_ARRAY map initialization pops to mind
immediately.

So, please check the document, leave comments, make yourself aware of
upcoming changes. Thank you!

  [0] https://docs.google.com/document/d/1UyjTZuPFWiPFyKk1tV5an11_iaRuec6U-ZESZ54nNTY/edit?usp=sharing

-- Andrii

Re: libbpf: the road to v1.0

[Andrii Nakryiko]

On Wed, Feb 24, 2021 at 11:44 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> Hi all,
>
> So I've been ruminating on getting libbpf to 1.0 version for a while
> now and finally got to write down most of the major (and not so much)
> things I wanted to change and/or break, given v1.0 gives an
> opportunity to break API/ABI compatibility, where necessary.
>
> I decided to share this with the community in the form of a Google Doc
> (check [0]), open to commenting by anyone, because there are many
> different things, quite often completely independent from each other.
> So email doesn't feel like the right medium to have a discussion given
> the amount of people that might be interested about just pieces of the
> plan.
>
> The overarching idea is to streamline APIs, make them overall more
> consistent throughout, as well as eliminate some very common pitfalls.
> Any such changes means potentially more pain for existing users during
> the transition period. I realize that, but I hope everyone will keep
> in mind the longer term goal of making libbpf easier to use both for
> the new and existing users.
>
> My intent is to spend some time in discussions and see what I have
> missed and what would be argued to be too drastic/problematic. So the
> plan is not set in stone and can/will be adjusted (within reason, of
> course, I don't believe everyone is going to converge and be happy
> about all the changes, but that's OK). But once decided on the plan, I
> (and hopefully others will help) will start implementing changes, will
> probably create a wiki page documenting API migration paths, etc, etc.
> My current thinking is to do a gradual transition, rather than a big
> bang breaking change in 1.0 release. This should give people more time
> to find any possible breakages and adopt their code base gradually, so
> that by the 1.0 time there isn't much surprise left. But feel free to
> argue the other way.
>
> BTW, that document is only describing potentially breaking changes, it
> doesn't mean libbpf won't get any other new functionality. I still
> plan a bunch of other (new) features to be added before v1.0. E.g.,
> stuff like BPF static linking support (merging together multiple BPF
> .o files) and declarative PROG_ARRAY map initialization pops to mind
> immediately.
>
> So, please check the document, leave comments, make yourself aware of
> upcoming changes. Thank you!
>
>   [0] https://docs.google.com/document/d/1UyjTZuPFWiPFyKk1tV5an11_iaRuec6U-ZESZ54nNTY/edit?usp=sharing
>

Ok, so a bunch of time passed since this email and I have a few
updates and calls to actions.

The above Google doc was transformed into a wiki page ([0]) almost
intact. That should make linking to various sections a bit easier, and
it lives next to libbpf source code on Github.

I've also created a lot of Github issues (42, I believe) for each part
of libbpf 1.0 plan. Those issues are cross-linked from the above
mentioned wiki page. See all the libbpf 1.0-related issues at [1]. If
anyone would like to contribute and help, please check those (and
other, see below) issues, assign them to yourself or just comment on
the issue that you are interested, and start working on the parts of
libbpf 1.0 plan. A good chunk of those issues are marked with "good
first issue" tag and should be a good place to start for those willing
to contribute but are hesitant and/or new to libbpf.

That's not all, though. The first set of changes towards libbpf 1.0
went in recently ([2]) changing and streamlining libbpf API error
returning behaviors. That patch set also introduced a new
libbpf_set_strict_mode() API allowing to have a fine-grained control
over backwards-breaking changes and "simulate" a new libbpf 1.0
behavior. This should give users a way to test and adopt their code
ahead of final libbpf 1.0 release. Furthermore, I've started a "Libbpf
migration guide" wiki ([3]) that is designed to have (relatively)
short instructions on how to adopt pre-libbpf 1.0 code that might
break into libbpf 1.0-compatible code. Often proposed code patterns
will handle both libbpf modes transparently. This migration guide is
supposed to be updated and extended as we deprecate more APIs and/or
implement all the other previously laid out breaking changes. For
deprecated APIs we'll just directly link to specific sections of that
wiki from the LIBBPF_DEPRECATED() annotations.

And last but not least. Since we started utilizing Github issues for
tracking various tasks and features, I've created a few issues for
features and work to be done (help with CI, anyone?) that I had on my
mind for some time. I tried to leave a bit more details, but if you
are eager to help with them and need more info then feel free to
comment on the specific issue and I'll try to elaborate some more. You
can see a pretty short list of those at [4].

  [0] https://github.com/libbpf/libbpf/wiki/Libbpf:-the-road-to-v1.0
  [1] https://github.com/libbpf/libbpf/issues?q=is%3Aissue+is%3Aopen+label%3Alibbpf-1.0
  [2] https://patchwork.kernel.org/project/netdevbpf/list/?series=487853&state=*
  [3] https://github.com/libbpf/libbpf/wiki/Libbpf-1.0-migration-guide
  [4] https://github.com/libbpf/libbpf/issues?q=is%3Aissue+is%3Aopen+-label%3Alibbpf-1.0

> -- Andrii