Re: Bpftool mirror now available

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Wed, Jan 19, 2022 at 6:47 AM Quentin Monnet <quentin@isovalent.com> wrote:
>
> Hi, I have the pleasure to announce the availability of a mirror for
> bpftool on GitHub, at the following URL:
>
>     https://github.com/libbpf/bpftool
>

This is great! Thanks a lot for all the clean ups, fixes, and
improvements to make it possible to mirror bpftool to Github repo!

> This mirror is similar in spirit to the one for libbpf [0], and its
> creation was lead by the following motivations.
>
> 1. The first goal is to provide a simpler way to build bpftool. So far,
> building a binary would require downloading the entire kernel
> repository. By contrast, the code in the GitHub mirror is mostly
> self-sufficient (it still requires libelf and zlib, and uses libbpf from
> its mirror as a git submodule), and offers an easy way to just clone and
> compile the tool.

Yep, libbpf-bootstrap will benefit from this a lot. A bunch of people
already asked about multi-platform support there and the need to
precompile bpftool for each architecture was a big blocker. Now this
blocker is gone as we can just compile bpftool from sources easily.

Same story with libbpf-tools in BCC repo, btw.

>
> 2. Because it is easier to compile and ship, this mirror should
> hopefully simplify bpftool packaging for distributions.

Right, I hope disto packagers will be quick to adopt the new mirror
repo for packaging bpftool. Let's figure out bpftool versioning schema
as a next step. Given bpftool heavily relies on libbpf and isn't
really coupled to kernel versions, it makes sense for bpftool to
reflect libbpf version rather than kernel's. WDYT?

>
> 3. Another objective was to help other projects build on top of the
> existing sources for bpftool. I'm thinking in particular of
> eBPF-for-Windows, which has been working on a proof-of-concept port of
> the tool for Windows [1]. Bpftool's mirror keeps the minimal amount of
> necessary headers, and stripped most of them from the definitions that
> are not required in our context, which should make it easier to uncouple
> bpftool from Linux.
>
> 4. At last, GitHub's automation features should help implement CI checks
> for bpftool, very much like libbpf is using today. The recent work
> conducted on libbpf's CI and turning some of the checks into reusable
> GitHub Actions [2] may help for bpftool.
>
> Just to make it clear, bpftool's mirror does not change the fact that
> all bpftool development happens on the kernel mailing-lists (in
> particular, the BPF mailing-list), and that the sources hosted in the
> kernel repository remain the reference for the tool. At this time the
> GitHub repository is just a mirror, and will not accept pull requests on
> bpftool's sources.
>
> Regarding synchronisation, the repository contains a script which should
> help cherry-pick all commits related to bpftool from the kernel
> repository. The idea is to regularly align bpftool on the latest version
> from libbpf's mirror (that is to say, to cherry-pick all bpftool-related
> commits from bpf-next and bpf trees, up to the commit at which libbpf's
> mirror stopped), to avoid any discrepancy between the tool and the
> library it relies on.
>
> GitHub was the original home of bpftool, before Jakub moved it to
> kernel's tools/ with commit 71bb428fe2c1 ("tools: bpf: add bpftool")
> back in 2017. More than four years and five hundred commits later, it is
> time to have a stand-alone repository again! But over time, the build
> system and the header dependencies got somewhat intertwined, and
> extracting the tool again required a few careful steps. Some of them
> happened upstream: we addressed the licensing of a few bpftool
> dependencies, we switched to libbpf's hash map implementation (instead
> of kernel's), we fixed the way bpftool picks up the development headers
> from libbpf, and so on. Some other changes happened in the mirror, to
> adjust bpftool's Makefile and build system. In the end, I'm rather happy
> with the result: the main Makefile in bpftool's mirror only has a few
> minor changes with the reference one, and the C sources need no change
> at all. The tool you get from the mirror is the same as what you can
> compile from tools/bpf/bpftool/.
>
> I hope that this mirror will make it easier to work and develop with
> bpftool. Thanks to Andrii for his feedback, and to the folks behind
> libbpf's mirroring for their work, some of which I was happy to reuse.
>
> Best regards,
> Quentin
>
> [0] https://github.com/libbpf/libbpf
> [1] https://github.com/dthaler/bpftool
> [2] https://github.com/libbpf/ci