Bpftool mirror now available

Bpftool mirror now available

[Quentin Monnet]

Hi, I have the pleasure to announce the availability of a mirror for
bpftool on GitHub, at the following URL:

    https://github.com/libbpf/bpftool

This mirror is similar in spirit to the one for libbpf [0], and its
creation was lead by the following motivations.

1. The first goal is to provide a simpler way to build bpftool. So far,
building a binary would require downloading the entire kernel
repository. By contrast, the code in the GitHub mirror is mostly
self-sufficient (it still requires libelf and zlib, and uses libbpf from
its mirror as a git submodule), and offers an easy way to just clone and
compile the tool.

2. Because it is easier to compile and ship, this mirror should
hopefully simplify bpftool packaging for distributions.

3. Another objective was to help other projects build on top of the
existing sources for bpftool. I'm thinking in particular of
eBPF-for-Windows, which has been working on a proof-of-concept port of
the tool for Windows [1]. Bpftool's mirror keeps the minimal amount of
necessary headers, and stripped most of them from the definitions that
are not required in our context, which should make it easier to uncouple
bpftool from Linux.

4. At last, GitHub's automation features should help implement CI checks
for bpftool, very much like libbpf is using today. The recent work
conducted on libbpf's CI and turning some of the checks into reusable
GitHub Actions [2] may help for bpftool.

Just to make it clear, bpftool's mirror does not change the fact that
all bpftool development happens on the kernel mailing-lists (in
particular, the BPF mailing-list), and that the sources hosted in the
kernel repository remain the reference for the tool. At this time the
GitHub repository is just a mirror, and will not accept pull requests on
bpftool's sources.

Regarding synchronisation, the repository contains a script which should
help cherry-pick all commits related to bpftool from the kernel
repository. The idea is to regularly align bpftool on the latest version
from libbpf's mirror (that is to say, to cherry-pick all bpftool-related
commits from bpf-next and bpf trees, up to the commit at which libbpf's
mirror stopped), to avoid any discrepancy between the tool and the
library it relies on.

GitHub was the original home of bpftool, before Jakub moved it to
kernel's tools/ with commit 71bb428fe2c1 ("tools: bpf: add bpftool")
back in 2017. More than four years and five hundred commits later, it is
time to have a stand-alone repository again! But over time, the build
system and the header dependencies got somewhat intertwined, and
extracting the tool again required a few careful steps. Some of them
happened upstream: we addressed the licensing of a few bpftool
dependencies, we switched to libbpf's hash map implementation (instead
of kernel's), we fixed the way bpftool picks up the development headers
from libbpf, and so on. Some other changes happened in the mirror, to
adjust bpftool's Makefile and build system. In the end, I'm rather happy
with the result: the main Makefile in bpftool's mirror only has a few
minor changes with the reference one, and the C sources need no change
at all. The tool you get from the mirror is the same as what you can
compile from tools/bpf/bpftool/.

I hope that this mirror will make it easier to work and develop with
bpftool. Thanks to Andrii for his feedback, and to the folks behind
libbpf's mirroring for their work, some of which I was happy to reuse.

Best regards,
Quentin

[0] https://github.com/libbpf/libbpf
[1] https://github.com/dthaler/bpftool
[2] https://github.com/libbpf/ci

Re: Bpftool mirror now available

[Andrii Nakryiko]

On Wed, Jan 19, 2022 at 6:47 AM Quentin Monnet <quentin@isovalent.com> wrote:
>
> Hi, I have the pleasure to announce the availability of a mirror for
> bpftool on GitHub, at the following URL:
>
>     https://github.com/libbpf/bpftool
>

This is great! Thanks a lot for all the clean ups, fixes, and
improvements to make it possible to mirror bpftool to Github repo!

> This mirror is similar in spirit to the one for libbpf [0], and its
> creation was lead by the following motivations.
>
> 1. The first goal is to provide a simpler way to build bpftool. So far,
> building a binary would require downloading the entire kernel
> repository. By contrast, the code in the GitHub mirror is mostly
> self-sufficient (it still requires libelf and zlib, and uses libbpf from
> its mirror as a git submodule), and offers an easy way to just clone and
> compile the tool.

Yep, libbpf-bootstrap will benefit from this a lot. A bunch of people
already asked about multi-platform support there and the need to
precompile bpftool for each architecture was a big blocker. Now this
blocker is gone as we can just compile bpftool from sources easily.

Same story with libbpf-tools in BCC repo, btw.

>
> 2. Because it is easier to compile and ship, this mirror should
> hopefully simplify bpftool packaging for distributions.

Right, I hope disto packagers will be quick to adopt the new mirror
repo for packaging bpftool. Let's figure out bpftool versioning schema
as a next step. Given bpftool heavily relies on libbpf and isn't
really coupled to kernel versions, it makes sense for bpftool to
reflect libbpf version rather than kernel's. WDYT?

>
> 3. Another objective was to help other projects build on top of the
> existing sources for bpftool. I'm thinking in particular of
> eBPF-for-Windows, which has been working on a proof-of-concept port of
> the tool for Windows [1]. Bpftool's mirror keeps the minimal amount of
> necessary headers, and stripped most of them from the definitions that
> are not required in our context, which should make it easier to uncouple
> bpftool from Linux.
>
> 4. At last, GitHub's automation features should help implement CI checks
> for bpftool, very much like libbpf is using today. The recent work
> conducted on libbpf's CI and turning some of the checks into reusable
> GitHub Actions [2] may help for bpftool.
>
> Just to make it clear, bpftool's mirror does not change the fact that
> all bpftool development happens on the kernel mailing-lists (in
> particular, the BPF mailing-list), and that the sources hosted in the
> kernel repository remain the reference for the tool. At this time the
> GitHub repository is just a mirror, and will not accept pull requests on
> bpftool's sources.
>
> Regarding synchronisation, the repository contains a script which should
> help cherry-pick all commits related to bpftool from the kernel
> repository. The idea is to regularly align bpftool on the latest version
> from libbpf's mirror (that is to say, to cherry-pick all bpftool-related
> commits from bpf-next and bpf trees, up to the commit at which libbpf's
> mirror stopped), to avoid any discrepancy between the tool and the
> library it relies on.
>
> GitHub was the original home of bpftool, before Jakub moved it to
> kernel's tools/ with commit 71bb428fe2c1 ("tools: bpf: add bpftool")
> back in 2017. More than four years and five hundred commits later, it is
> time to have a stand-alone repository again! But over time, the build
> system and the header dependencies got somewhat intertwined, and
> extracting the tool again required a few careful steps. Some of them
> happened upstream: we addressed the licensing of a few bpftool
> dependencies, we switched to libbpf's hash map implementation (instead
> of kernel's), we fixed the way bpftool picks up the development headers
> from libbpf, and so on. Some other changes happened in the mirror, to
> adjust bpftool's Makefile and build system. In the end, I'm rather happy
> with the result: the main Makefile in bpftool's mirror only has a few
> minor changes with the reference one, and the C sources need no change
> at all. The tool you get from the mirror is the same as what you can
> compile from tools/bpf/bpftool/.
>
> I hope that this mirror will make it easier to work and develop with
> bpftool. Thanks to Andrii for his feedback, and to the folks behind
> libbpf's mirroring for their work, some of which I was happy to reuse.
>
> Best regards,
> Quentin
>
> [0] https://github.com/libbpf/libbpf
> [1] https://github.com/dthaler/bpftool
> [2] https://github.com/libbpf/ci

Re: Bpftool mirror now available

[Quentin Monnet]

2022-01-19 22:25 UTC-0800 ~ Andrii Nakryiko <andrii.nakryiko@gmail.com>
> On Wed, Jan 19, 2022 at 6:47 AM Quentin Monnet <quentin@isovalent.com> wrote:

[...]

>> 2. Because it is easier to compile and ship, this mirror should
>> hopefully simplify bpftool packaging for distributions.
> 
> Right, I hope disto packagers will be quick to adopt the new mirror
> repo for packaging bpftool. Let's figure out bpftool versioning schema
> as a next step. Given bpftool heavily relies on libbpf and isn't
> really coupled to kernel versions, it makes sense for bpftool to
> reflect libbpf version rather than kernel's. WDYT?

Personally, I don't mind finding another scheme, as long as we keep it
consistent between the reference sources in the kernel repo and the mirror.

I also agree that it would make sense to align it to libbpf, but that
would mean going backward on the numbers (current version is 5.16.0,
libbpf's is 0.7.0) and this will mess up with every script trying to
compare versions. We could maybe add a prefix to indicate that the
scheme has changed ('l_0.7.0), but similarly, it would break a good
number of tools that expect semantic versioning, I don't think this is
any better.

The other alternative I see would be to pick a different major version
number and arbitrarily declare that bpftool's version is aligned on
libbpf's, but with a difference of 6 for the version number. So we would
start at 6.7.0 and reach 7.0.0 when libbpf 1.0.0 is released. This is
not ideal, but we would keep some consistency, and we can always add the
version of libbpf used for the build to "bpftool version"'s output. How
would you feel about it? Did you have something else in mind?

Quentin

Re: Bpftool mirror now available

[Andrii Nakryiko]

On Thu, Jan 20, 2022 at 4:35 AM Quentin Monnet <quentin@isovalent.com> wrote:
>
> 2022-01-19 22:25 UTC-0800 ~ Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > On Wed, Jan 19, 2022 at 6:47 AM Quentin Monnet <quentin@isovalent.com> wrote:
>
> [...]
>
> >> 2. Because it is easier to compile and ship, this mirror should
> >> hopefully simplify bpftool packaging for distributions.
> >
> > Right, I hope disto packagers will be quick to adopt the new mirror
> > repo for packaging bpftool. Let's figure out bpftool versioning schema
> > as a next step. Given bpftool heavily relies on libbpf and isn't
> > really coupled to kernel versions, it makes sense for bpftool to
> > reflect libbpf version rather than kernel's. WDYT?
>
> Personally, I don't mind finding another scheme, as long as we keep it
> consistent between the reference sources in the kernel repo and the mirror.
>
> I also agree that it would make sense to align it to libbpf, but that
> would mean going backward on the numbers (current version is 5.16.0,
> libbpf's is 0.7.0) and this will mess up with every script trying to
> compare versions. We could maybe add a prefix to indicate that the
> scheme has changed ('l_0.7.0), but similarly, it would break a good
> number of tools that expect semantic versioning, I don't think this is
> any better.
>
> The other alternative I see would be to pick a different major version
> number and arbitrarily declare that bpftool's version is aligned on
> libbpf's, but with a difference of 6 for the version number. So we would
> start at 6.7.0 and reach 7.0.0 when libbpf 1.0.0 is released. This is
> not ideal, but we would keep some consistency, and we can always add the
> version of libbpf used for the build to "bpftool version"'s output. How
> would you feel about it? Did you have something else in mind?

Yeah, this off-by-6 major version difference seems ok-ish to me, I
don't mind that. Another alternative is to have a completely
independent versioning (and report used libbpf version in bpftool
--version output  separately). But I think divorcing it from kernel
version is a must, too much confusion.

>
> Quentin

Re: Bpftool mirror now available

[Quentin Monnet]

2022-01-20 11:07 UTC-0800 ~ Andrii Nakryiko <andrii.nakryiko@gmail.com>
> On Thu, Jan 20, 2022 at 4:35 AM Quentin Monnet <quentin@isovalent.com> wrote:
>>
>> 2022-01-19 22:25 UTC-0800 ~ Andrii Nakryiko <andrii.nakryiko@gmail.com>
>>> On Wed, Jan 19, 2022 at 6:47 AM Quentin Monnet <quentin@isovalent.com> wrote:
>>
>> [...]
>>
>>>> 2. Because it is easier to compile and ship, this mirror should
>>>> hopefully simplify bpftool packaging for distributions.
>>>
>>> Right, I hope disto packagers will be quick to adopt the new mirror
>>> repo for packaging bpftool. Let's figure out bpftool versioning schema
>>> as a next step. Given bpftool heavily relies on libbpf and isn't
>>> really coupled to kernel versions, it makes sense for bpftool to
>>> reflect libbpf version rather than kernel's. WDYT?
>>
>> Personally, I don't mind finding another scheme, as long as we keep it
>> consistent between the reference sources in the kernel repo and the mirror.
>>
>> I also agree that it would make sense to align it to libbpf, but that
>> would mean going backward on the numbers (current version is 5.16.0,
>> libbpf's is 0.7.0) and this will mess up with every script trying to
>> compare versions. We could maybe add a prefix to indicate that the
>> scheme has changed ('l_0.7.0), but similarly, it would break a good
>> number of tools that expect semantic versioning, I don't think this is
>> any better.
>>
>> The other alternative I see would be to pick a different major version
>> number and arbitrarily declare that bpftool's version is aligned on
>> libbpf's, but with a difference of 6 for the version number. So we would
>> start at 6.7.0 and reach 7.0.0 when libbpf 1.0.0 is released. This is
>> not ideal, but we would keep some consistency, and we can always add the
>> version of libbpf used for the build to "bpftool version"'s output. How
>> would you feel about it? Did you have something else in mind?
> 
> Yeah, this off-by-6 major version difference seems ok-ish to me, I
> don't mind that. Another alternative is to have a completely
> independent versioning (and report used libbpf version in bpftool
> --version output  separately). But I think divorcing it from kernel
> version is a must, too much confusion.

Right, let's not tie it to libbpf either, having an independent
versioning scheme is probably the best solution indeed. I'll send a
patchset shortly to update the version and also print the one from libbpf.

Thanks,
Quentin

RE: Bpftool mirror now available

[Dave Thaler]

Quentin Monnet <quentin@isovalent.com> wrote:
> Hi, I have the pleasure to announce the availability of a mirror for bpftool on GitHub, at the following URL:
> [...]
> 3. Another objective was to help other projects build on top of the existing sources for bpftool. I'm thinking in
> particular of eBPF-for-Windows, which has been working on a proof-of-concept port of the tool for Windows [1]. 
> Bpftool's mirror keeps the minimal amount of necessary headers, and stripped most of them from the definitions
> that are not required in our context, which should make it easier to uncouple bpftool from Linux.
> [...]
> Just to make it clear, bpftool's mirror does not change the fact that all bpftool development happens on the
> kernel mailing-lists (in particular, the BPF mailing-list), and that the sources hosted in the kernel repository
> remain the reference for the tool. At this time the GitHub repository is just a mirror, and will not accept pull
> requests on bpftool's sources.

Thanks Quentin, this is a great first step!   I can update the ebpf-for-windows project to use this as a submodule.

Longer term, is the goal to make the mirror be the authoritative reference, or to make the Linux kernel repository
not be Linux-only but accept non-Linux patches to bpftool?

Dave

Re: Bpftool mirror now available

[Quentin Monnet]

2022-01-20 14:19 UTC+0000 ~ Dave Thaler <dthaler@microsoft.com>
> Quentin Monnet <quentin@isovalent.com> wrote:
>> Hi, I have the pleasure to announce the availability of a mirror for bpftool on GitHub, at the following URL:
>> [...]
>> 3. Another objective was to help other projects build on top of the existing sources for bpftool. I'm thinking in
>> particular of eBPF-for-Windows, which has been working on a proof-of-concept port of the tool for Windows [1]. 
>> Bpftool's mirror keeps the minimal amount of necessary headers, and stripped most of them from the definitions
>> that are not required in our context, which should make it easier to uncouple bpftool from Linux.
>> [...]
>> Just to make it clear, bpftool's mirror does not change the fact that all bpftool development happens on the
>> kernel mailing-lists (in particular, the BPF mailing-list), and that the sources hosted in the kernel repository
>> remain the reference for the tool. At this time the GitHub repository is just a mirror, and will not accept pull
>> requests on bpftool's sources.
> 
> Thanks Quentin, this is a great first step!   I can update the ebpf-for-windows project to use this as a submodule.
> 
> Longer term, is the goal to make the mirror be the authoritative reference, or to make the Linux kernel repository
> not be Linux-only but accept non-Linux patches to bpftool?

Hi Dave, longer term goals have not been established yet, and the
discussion about what happens to bpftool next still needs to happen. I
understand that you have been working on making bpftool cross-OS, and
that this raises the question of how to contribute Windows-related
patches upstream.

Moving bpftool out of the kernel and into its own tree (whether on
GitHub or on kernel.org) would make sense to me, although it comes with
a number of things to sort out. First, bpftool is now being used
directly by a number of components in the kernel, for loading programs
or for its ability to generate BPF skeletons for programs. As far as I
can tell, this concerns the following items:

- The kernel itself, when configured with CONFIG_BPF_PRELOAD, requires
bpftool to build, because BPF pre-loaded iterators rely on BPF skeletons
(see kernel/bpf/preload/iterators/Makefile).

- BPF samples and selftests (samples/bpf/Makefile,
tools/bpf/runqslower/Makefile) use BPF for a number of use cases.

- Other tools hosted in the kernel repository, in particular runqslower
(tools/bpf/runqslower/Makefile) and perf (tools/perf/Makefile.perf), use
bpftool to produce BPF skeletons as well.

As far as I can tell, the above do not rely on cutting-edge bpftool
features, and they could maybe be adjusted to consider bpftool as an
external dependency for BPF, somewhat like pahole or clang/LLVM have
been so far.

Another thing to consider is that keeping bpftool next to the kernel
sources has been useful to help keeping the tool in sync, for example
for adding new type names to bpftool's lists when the kernel get new
program/map types. We have recently introduced some CI checks that could
be adjusted to work with an external repo and mitigate this issue, but
still, it is harder to tell people to submit changes to a second
repository when what they want is just to update the kernel. I fear this
would result in a bit more maintenance on bpftool's side (but then
bpftool's requirements in terms of maintenance are not that big when
compared to bigger tools, and maybe some of it could be automated).

Then the other solution, as you mentioned, would be to take
Windows-related patches for bpftool in the Linux repo. For what it's
worth, I don't have any personal objection to it, but it raises the
problems of testing and ownership (who fixes bugs) for these patches.
I'm also unsure what it would mean in terms of development workflow:
would Windows-related contributions be reviewed and tested beforehand,
and treated somewhat like vendor code, or would all the discussions
(Windows-related bug reports, contributions to Windows support but
external to Microsoft, etc.) happen on the BPF mailing list?

If we want bpftool to become fully cross-OS, my feeling is that it would
be maybe more work, but more sensible to move it outside of the kernel
tree (although this does not have to be immediate, obviously - let's see
how the Windows port is doing first). However, this decision is not mine
alone to take, and the maintainers will surely have their say in it
(this could also be a topic for you to raise at the next BSC meeting, I
guess). I hope the considerations above can help for this discussion.

Best regards,
Quentin

RE: Bpftool mirror now available

[Dave Thaler]

Quentin Monnet <quentin@isovalent.com> writes:
> Another thing to consider is that keeping bpftool next to the kernel sources
> has been useful to help keeping the tool in sync, for example for adding new
> type names to bpftool's lists when the kernel get new program/map types.
> We have recently introduced some CI checks that could be adjusted to work
> with an external repo and mitigate this issue, but still, it is harder to tell people
> to submit changes to a second repository when what they want is just to update
> the kernel. I fear this would result in a bit more maintenance on bpftool's side
> (but then bpftool's requirements in terms of maintenance are not that big
> when compared to bigger tools, and maybe some of it could be automated).
>
> Then the other solution, as you mentioned, would be to take Windows-related
> patches for bpftool in the Linux repo. For what it's worth, I don't have any
> personal objection to it, but it raises the problems of testing and ownership
> (who fixes bugs) for these patches.

Personally I would recommend a third approach.   That is, bpftool today 
combines both platform-agnostic code and platform-specific code without
clean factoring between them.  Instead I would want to see it factored such
that there is a clean API between them, where the platform-agnostic code
can be out-of-tree, and the platform-specific code can be in-tree.   This would
allow Windows platform-specific code to similarly be in-tree for the ebpf-for-windows project.  Both the Linux kernel and ebpf-for-windows (and any other
future platforms) can then depend on the out-of-tree code along with their
own platform-specific code needed to build and run on their own platform.
That's roughly the approach that I've taken for some other projects where it
has worked well.

Dave

Re: Bpftool mirror now available

[Daniel Borkmann]

On 1/25/22 4:39 AM, Dave Thaler wrote:
> Quentin Monnet <quentin@isovalent.com> writes:
>> Another thing to consider is that keeping bpftool next to the kernel sources
>> has been useful to help keeping the tool in sync, for example for adding new
>> type names to bpftool's lists when the kernel get new program/map types.
>> We have recently introduced some CI checks that could be adjusted to work
>> with an external repo and mitigate this issue, but still, it is harder to tell people
>> to submit changes to a second repository when what they want is just to update
>> the kernel. I fear this would result in a bit more maintenance on bpftool's side
>> (but then bpftool's requirements in terms of maintenance are not that big
>> when compared to bigger tools, and maybe some of it could be automated).
>>
>> Then the other solution, as you mentioned, would be to take Windows-related
>> patches for bpftool in the Linux repo. For what it's worth, I don't have any
>> personal objection to it, but it raises the problems of testing and ownership
>> (who fixes bugs) for these patches.
> 
> Personally I would recommend a third approach.   That is, bpftool today
> combines both platform-agnostic code and platform-specific code without
> clean factoring between them.  Instead I would want to see it factored such
> that there is a clean API between them, where the platform-agnostic code
> can be out-of-tree, and the platform-specific code can be in-tree.   This would
> allow Windows platform-specific code to similarly be in-tree for the ebpf-for-windows project.  Both the Linux kernel and ebpf-for-windows (and any other
> future platforms) can then depend on the out-of-tree code along with their
> own platform-specific code needed to build and run on their own platform.
> That's roughly the approach that I've taken for some other projects where it
> has worked well.

I wouldn't mind if tools/bpf/bpftool/ would see some refactoring effort to
make it more platform-agnostic, similar as kernel split out arch/ bits vs
generic code. Needed bits should however still be somewhere under bpftool
dir in the tree, at least for Linux, so that patch series touching kernel +
libbpf + bpftool can be run by the existing CI w/o extra detour to first
patch or requiring feature branch on some external out-of-tree dependency.
Perhaps it would be possible to have platform-specific code pluggable via
lib as one of the build options for bpftool..

Thanks,
Daniel