Packaging bpftool and libbpf: GitHub or kernel?

Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

Hi, 

I'm considering switch to bpftool's mirror on GitHub for packaging (instead
of using the source found in kernel), but realize that it should goes
hand-in-hand with how libbpf is packaged, which eventually leads these
questions:

  What is the suggested approach for packaging bpftool and libbpf?
  Which source is preferred, GitHub or kernel?
  Does bpftool work on older kernel?

Our current approach is that we (openSUSE/SLES) essentially have two version
of libbpf: a public shared library that uses GitHub mirror as source, which
the general userspace sees and links to; and a private static library built
from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
took similar approach.

This approach means that the version of bpftool and libbpf are _not_ always
in sync[1], which I read may causes problem since libbpf and bpftool depends
on specific version of each other[2].

Using the GitHub mirror of bpftool to package both libbpf and bpftool would
kept their version in sync, and was suggested[3]. Although the same could be
said if we switch back to packaging libbpf from kernel source, an additional
appeal for using GitHub mirrors is that it decouples bpftool from kernel,
making it easily upgradable and with a clearer changelog (the latter is
quite important for enterprise users) like libbpf.

The main concern with using GitHub mirror is that bpftool may be updated far
beyond the version that comes with the runtime kernel. AFAIK bpftool should
work on older kernel since CO-RE is used for built-in BPF iterators and the
underlying libbpf work on older kernel itself. Nonetheless, it would be nice
to get a confirmation from the maintainers.

Are there any other downsides to switching to GitHub mirror for bpftool?

A side note: if we want all userspace visible libbpf to have a coherent
version, perf needs to use the shared libbpf library as well (either
autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to
backport patches to kernel source to keep up with userspace package (libbpf)
changes could be a pain.


Shung-Hsi

1: In practice kernel & bpftool version often falls behind libbpf in
   snapshot distro (e.g. openSUSE Leap, Ubutnu LTS), and the other way
   around on rolling distro (e.g. openSUSE Tumbleweed, Arch).
2: https://lore.kernel.org/bpf/CAADnVQK-arrrNrgtu48_f--WCwR5ki2KGaX=mN2qmW_AcRyb=w@mail.gmail.com/
3: https://lore.kernel.org/bpf/20191204233948.opvlopjkxe5o66lr@ast-mbp.dhcp.thefacebook.com/
4: https://src.fedoraproject.org/rpms/kernel-tools/blob/82960989c918f81fcc6742a6d765afbec5fa4f74/f/kernel-tools.spec#_248

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

On Thu, Apr 13, 2023 at 05:23:16PM +0800, Shung-Hsi Yu wrote:
> Hi, 
> 
> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> of using the source found in kernel), but realize that it should goes
> hand-in-hand with how libbpf is packaged, which eventually leads these
> questions:
> 
>   What is the suggested approach for packaging bpftool and libbpf?
>   Which source is preferred, GitHub or kernel?

An off-topic, yet somewhat related question that I also tried to figure out
is "why the GitHub mirror for libbpf and bpftool exist at the first place?".
It is a non-trivial amount of work for the maintainers after all.

For libbpf, the main uses case for GitHub seem to be for it to be used as
submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.

For bpftool the reason seems to be less clear[2]. From what I can tell right
now its mainly use for CI (this applies to libbpf as well), which is
definitely useful.

But I wonder whether packaging one of the motives to create the mirrors
initially? Can't seem to find anything in this regard.


1: https://github.com/acmel/dwarves/tree/master/lib
2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/

> [snip]

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Toke Høiland-Jørgensen]

Shung-Hsi Yu <shung-hsi.yu@suse.com> writes:

> A side note: if we want all userspace visible libbpf to have a coherent
> version, perf needs to use the shared libbpf library as well (either
> autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to
> backport patches to kernel source to keep up with userspace package (libbpf)
> changes could be a pain.

So basically, this here is the reason we're building libbpf from the
kernel tree for the RHEL package: If we use the github version we'd need
to juggle two different versions of libbpf, one for the in-kernel-tree
users (perf as you mention, but also the BPF selftests), and one for the
userspace packages. Also, having libbpf in the kernel tree means we can
just backport patches to it along with the BPF-related kernel patches
(we do quite extensive BPF backports for each RHEL version). Finally,
building from the kernel tree means we can use the existing
kernel-related procedures for any out of order hotfixes (since AFAIK
none of the github repositories have any concept of stable branches that
receive fixes).

YMMV of course, but figured I'd share our reasoning. To be clear,
building from the kernel tree is not without its own pain points (mostly
related to how the build scripts are structured for our kernel builds).
We've discussed moving to the github version of libbpf multiple times,
but every time we've concluded that it would be more, not less, painful
than having the kernel tree be the single source of truth.

-Toke

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Quentin Monnet]

Hi Shung-Hsi,

On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>
> Hi,
>
> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> of using the source found in kernel), but realize that it should goes
> hand-in-hand with how libbpf is packaged, which eventually leads these
> questions:
>
>   What is the suggested approach for packaging bpftool and libbpf?
>   Which source is preferred, GitHub or kernel?

As you can see from the previous discussions, the suggested approach
would be to package from the GitHub mirror, with libbpf and bpftool in
sync.

My main argument for the mirror is that it keeps things simpler, and
there's no need to deal with the rest of the kernel sources for these
packages. Download from the mirrors, build, ship. But then I have
limited experience at packaging for distros, and I can understand
Toke's point of view, too. So ultimately, the call is yours.

>   Does bpftool work on older kernel?

It should, although it's not perfect. Most features from current
bpftool should work as expected on older kernels. However, if I
remember correctly you would have trouble loading programs on pre-BTF
kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
definitions with BTF info, and attempts to create these maps with BTF,
which fails and blocks the load process.

But we're trying to keep backward-compatibility, so if we're only
talking of kernels recent enough to support BTF, then I'd expect
bpftool to work. If this is not the case, please report on this list.

>
> Our current approach is that we (openSUSE/SLES) essentially have two version
> of libbpf: a public shared library that uses GitHub mirror as source, which
> the general userspace sees and links to; and a private static library built
> from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
> A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
> took similar approach.

I would like them to reconsider this choice eventually. Sounds like
for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
have a real bpftool package instead of having to install
linux-tools-common + linux-tools-generic, or to have distros in
general (Ubuntu/Debian at least) stop compiling out the JIT
disassembler, although this is not strictly related to the location of
the sources. I've not found the time to reach out to package
maintainers yet.

>
> This approach means that the version of bpftool and libbpf are _not_ always
> in sync[1], which I read may causes problem since libbpf and bpftool depends
> on specific version of each other[2].

Whatever source you use, I would strongly recommend finding a way to
keep both in sync. Libbpf has stabilised its API when reaching 1.0,
but bpftool taps into some of the internals of the library. Features
or new definitions are usually added at the same time to libbpf and
bpftool, and if you get a mismatch between the two, you're taking
risks to get build issues.

>
> Using the GitHub mirror of bpftool to package both libbpf and bpftool would
> kept their version in sync, and was suggested[3]. Although the same could be
> said if we switch back to packaging libbpf from kernel source, an additional
> appeal for using GitHub mirrors is that it decouples bpftool from kernel,
> making it easily upgradable and with a clearer changelog (the latter is
> quite important for enterprise users) like libbpf.

Happy to read these changelogs I write are useful to someone :). Yes,
this is my point.

>
> The main concern with using GitHub mirror is that bpftool may be updated far
> beyond the version that comes with the runtime kernel. AFAIK bpftool should
> work on older kernel since CO-RE is used for built-in BPF iterators and the
> underlying libbpf work on older kernel itself. Nonetheless, it would be nice
> to get a confirmation from the maintainers.

As explained above - Mostly, it should work. Otherwise, we can look
into fixing it.

As a side note, I'm open to suggestions/contributions to make life
easier for packaging for the mirror. For example, Mahé and I recently
added GitHub workflows to ship statically-built binaries for amd64 and
arm64 on releases, as well as tarballs with both bpftool+libbpf
sources. If there's something else to make packaging easier, I'm happy
to talk about it.

>
> Are there any other downsides to switching to GitHub mirror for bpftool?

Nothing else comes to mind, but again I'm not packaging for distros.
See Toke's message.

I hope this helps,
Quentin

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Quentin Monnet]

On Thu, 13 Apr 2023 at 10:50, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>
> On Thu, Apr 13, 2023 at 05:23:16PM +0800, Shung-Hsi Yu wrote:
> > Hi,
> >
> > I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > of using the source found in kernel), but realize that it should goes
> > hand-in-hand with how libbpf is packaged, which eventually leads these
> > questions:
> >
> >   What is the suggested approach for packaging bpftool and libbpf?
> >   Which source is preferred, GitHub or kernel?
>
> An off-topic, yet somewhat related question that I also tried to figure out
> is "why the GitHub mirror for libbpf and bpftool exist at the first place?".
> It is a non-trivial amount of work for the maintainers after all.
>
> For libbpf, the main uses case for GitHub seem to be for it to be used as
> submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.

Then it should be enough for bpftool, too :) The bcc repository uses
it as a submodule, for example.

The work is non-trivial, but when compared to libbpf, I managed to
preserve most of the Makefile from the kernel tree and all of the C
code, and bpftool also gets patches less often.

>
> For bpftool the reason seems to be less clear[2]. From what I can tell right
> now its mainly use for CI (this applies to libbpf as well), which is
> definitely useful.

Yes. At the moment, the CI present on bpftool's mirror is more limited
than libbpf's. But it allows me to test some compilation variants:
regular builds, static builds, cross-compiling (to some extent). Some
additional checks that would make little sense to have in the kernel
repo, too. It's mostly for checking that none of these build
configurations break when I sync from the kernel, and helped me find
and fix several issues in the build system on the mirror.

This CI on the mirror doesn't cover bpftool's features, but these
should be tested in the BPF CI itself, so we can catch regressions
before patches are merged. There are some tests already, not many, I'd
like to improve that someday. Anyway.

>
> But I wonder whether packaging one of the motives to create the mirrors
> initially? Can't seem to find anything in this regard.
>
>
> 1: https://github.com/acmel/dwarves/tree/master/lib
> 2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/

It seems like you haven't come across this one?:
https://lore.kernel.org/bpf/267a35a6-a045-c025-c2d9-78afbf6fc325@isovalent.com/t/

Yes, easing packaging was one of the motivations for the mirror. As
mentioned in my other answer, I've not taken the time to reach out to
package maintainers yet, so this hasn't really materialised at this
point.

CI, indeed, was another motivation.

Submodules, or simply making it easier to hack with bpftool's code,
was yet another thing. Microsoft folks intend to make bpftool
compatible with eBPF for Windows. It's quite simpler to work on that
from a repo which is mostly uncoupled from the Linux tree.

Perhaps the most important was to make it easier to just download,
build and use bpftool, for all users who need to get the latest
version, or to patch it, or to create static builds, or to
cross-compile, or whatever reason might cause you to compile it from
the sources. For all those cases, getting the mirror is faster and
requires less space than getting the kernel repo. This makes a nice
difference when periodically rebuilding images in automated workflows,
for example.

Does this answer your questions?
Quentin

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Michal Suchánek]

Hello,

On Fri, Apr 14, 2023 at 02:12:40AM +0100, Quentin Monnet wrote:
> On Thu, 13 Apr 2023 at 10:50, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >
> > On Thu, Apr 13, 2023 at 05:23:16PM +0800, Shung-Hsi Yu wrote:
> > > Hi,
> > >
> > > I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > of using the source found in kernel), but realize that it should goes
> > > hand-in-hand with how libbpf is packaged, which eventually leads these
> > > questions:
> > >
> > >   What is the suggested approach for packaging bpftool and libbpf?
> > >   Which source is preferred, GitHub or kernel?

...

> > But I wonder whether packaging one of the motives to create the mirrors
> > initially? Can't seem to find anything in this regard.
> >
> >
> > 1: https://github.com/acmel/dwarves/tree/master/lib
> > 2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/
> 
> It seems like you haven't come across this one?:
> https://lore.kernel.org/bpf/267a35a6-a045-c025-c2d9-78afbf6fc325@isovalent.com/t/
> 
> Yes, easing packaging was one of the motivations for the mirror. As
> mentioned in my other answer, I've not taken the time to reach out to
> package maintainers yet, so this hasn't really materialised at this
> point.

For me as a package maintainer submodules are a major pain. They always
need special handling, break down, get out of sync between different
projects using them, etc.

Somehow in the past it was possible to build and install development
versions of libraries during development of tools using them, and
release both when a feature was finished.

Arguably using submodules for development may work for some people. For
most it would cause having the wrong submodule code when switching
branches, stray submodule changes in patches, etc.

Using submodules as a general method of distributing dependencies is
hell.

The usual methods of downloading and releasing the source code don't
work, the submodules have to be specifically bundled.

If the dependency in question has a bug each tool author has to be
coaxed to update to a fixed version and re-release, or each tool patched
separately.

Over time API bitrots and is given up, each tool requiring specific and
different release or git SHA of the dependency. We can see that
happening a lot in ecosystems where vendoring is the norm.

Thanks

Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Michal Suchánek]

Hello,

On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> Hi Shung-Hsi,
> 
> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >
> > Hi,
> >
> > I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > of using the source found in kernel), but realize that it should goes
> > hand-in-hand with how libbpf is packaged, which eventually leads these
> > questions:
> >
> >   What is the suggested approach for packaging bpftool and libbpf?
> >   Which source is preferred, GitHub or kernel?
> 
> As you can see from the previous discussions, the suggested approach
> would be to package from the GitHub mirror, with libbpf and bpftool in
> sync.
> 
> My main argument for the mirror is that it keeps things simpler, and
> there's no need to deal with the rest of the kernel sources for these
> packages. Download from the mirrors, build, ship. But then I have
> limited experience at packaging for distros, and I can understand
> Toke's point of view, too. So ultimately, the call is yours.

Things get only ever more complex when submodules are involved.

> >   Does bpftool work on older kernel?
> 
> It should, although it's not perfect. Most features from current
> bpftool should work as expected on older kernels. However, if I
> remember correctly you would have trouble loading programs on pre-BTF
> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
> definitions with BTF info, and attempts to create these maps with BTF,
> which fails and blocks the load process.
> 
> But we're trying to keep backward-compatibility, so if we're only
> talking of kernels recent enough to support BTF, then I'd expect
> bpftool to work. If this is not the case, please report on this list.

It won't build:
https://lore.kernel.org/bpf/20220421003152.339542-3-alobakin@pm.me/

> > Our current approach is that we (openSUSE/SLES) essentially have two version
> > of libbpf: a public shared library that uses GitHub mirror as source, which
> > the general userspace sees and links to; and a private static library built
> > from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
> > A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
> > took similar approach.
> 
> I would like them to reconsider this choice eventually. Sounds like
> for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
> have a real bpftool package instead of having to install
> linux-tools-common + linux-tools-generic, or to have distros in
> general (Ubuntu/Debian at least) stop compiling out the JIT
> disassembler, although this is not strictly related to the location of
> the sources. I've not found the time to reach out to package
> maintainers yet.
> 
> >
> > This approach means that the version of bpftool and libbpf are _not_ always
> > in sync[1], which I read may causes problem since libbpf and bpftool depends
> > on specific version of each other[2].
> 
> Whatever source you use, I would strongly recommend finding a way to
> keep both in sync. Libbpf has stabilised its API when reaching 1.0,
> but bpftool taps into some of the internals of the library. Features
> or new definitions are usually added at the same time to libbpf and
> bpftool, and if you get a mismatch between the two, you're taking
> risks to get build issues.

In other words no API exists.

> > Using the GitHub mirror of bpftool to package both libbpf and bpftool would
> > kept their version in sync, and was suggested[3]. Although the same could be
> > said if we switch back to packaging libbpf from kernel source, an additional
> > appeal for using GitHub mirrors is that it decouples bpftool from kernel,
> > making it easily upgradable and with a clearer changelog (the latter is
> > quite important for enterprise users) like libbpf.
> 
> Happy to read these changelogs I write are useful to someone :). Yes,
> this is my point.

Yes, publishing the changelog in a usable way relieves others of the
need to write thier own, usually with less understanding of the changes
in question. That's generally the idea of opensource - not endlessly
repeating what has already been done :)

> > The main concern with using GitHub mirror is that bpftool may be updated far
> > beyond the version that comes with the runtime kernel. AFAIK bpftool should
> > work on older kernel since CO-RE is used for built-in BPF iterators and the
> > underlying libbpf work on older kernel itself. Nonetheless, it would be nice
> > to get a confirmation from the maintainers.
> 
> As explained above - Mostly, it should work. Otherwise, we can look
> into fixing it.
> 
> As a side note, I'm open to suggestions/contributions to make life
> easier for packaging for the mirror. For example, Mahé and I recently
> added GitHub workflows to ship statically-built binaries for amd64 and
> arm64 on releases, as well as tarballs with both bpftool+libbpf
> sources. If there's something else to make packaging easier, I'm happy
> to talk about it.

Make it possible to build with system-installed libbpf. If it's released
it should have versioned dependency on a libbpf release, and libbpf from
that version on should be good enough to build it.

I tried copying those 'private' headers into a separate directory, and
link against static libbpf, and it seems to work. Of course, having
an actual API would be much better.

Thanks

Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Quentin Monnet]

2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> Hello,
> 
> On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
>> Hi Shung-Hsi,
>>
>> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>>>
>>> Hi,
>>>
>>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
>>> of using the source found in kernel), but realize that it should goes
>>> hand-in-hand with how libbpf is packaged, which eventually leads these
>>> questions:
>>>
>>>   What is the suggested approach for packaging bpftool and libbpf?
>>>   Which source is preferred, GitHub or kernel?
>>
>> As you can see from the previous discussions, the suggested approach
>> would be to package from the GitHub mirror, with libbpf and bpftool in
>> sync.
>>
>> My main argument for the mirror is that it keeps things simpler, and
>> there's no need to deal with the rest of the kernel sources for these
>> packages. Download from the mirrors, build, ship. But then I have
>> limited experience at packaging for distros, and I can understand
>> Toke's point of view, too. So ultimately, the call is yours.
> 
> Things get only ever more complex when submodules are involved.

I understand the generic pain points from your other email. But could
you be more specific for the case of bpftool? It's not like we're
shipping all lib dependencies as submodules. Sync-ups are specifically
aligned to the same commit used to sync the libbpf mirror, so that it's
pretty much as if we had the right version of the library shipped in the
repository - only, it's one --recurse-submodules away.

> 
>>>   Does bpftool work on older kernel?
>>
>> It should, although it's not perfect. Most features from current
>> bpftool should work as expected on older kernels. However, if I
>> remember correctly you would have trouble loading programs on pre-BTF
>> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
>> definitions with BTF info, and attempts to create these maps with BTF,
>> which fails and blocks the load process.
>>
>> But we're trying to keep backward-compatibility, so if we're only
>> talking of kernels recent enough to support BTF, then I'd expect
>> bpftool to work. If this is not the case, please report on this list.
> 
> It won't build:
> https://lore.kernel.org/bpf/20220421003152.339542-3-alobakin@pm.me/

True in this case, and this is something that needs to get fixed. Thanks
for reopening that thread! Are you building bpftool on kernels older
than 5.15? (genuine curiosity)

> 
>>> Our current approach is that we (openSUSE/SLES) essentially have two version
>>> of libbpf: a public shared library that uses GitHub mirror as source, which
>>> the general userspace sees and links to; and a private static library built
>>> from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
>>> A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
>>> took similar approach.
>>
>> I would like them to reconsider this choice eventually. Sounds like
>> for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
>> have a real bpftool package instead of having to install
>> linux-tools-common + linux-tools-generic, or to have distros in
>> general (Ubuntu/Debian at least) stop compiling out the JIT
>> disassembler, although this is not strictly related to the location of
>> the sources. I've not found the time to reach out to package
>> maintainers yet.
>>
>>>
>>> This approach means that the version of bpftool and libbpf are _not_ always
>>> in sync[1], which I read may causes problem since libbpf and bpftool depends
>>> on specific version of each other[2].
>>
>> Whatever source you use, I would strongly recommend finding a way to
>> keep both in sync. Libbpf has stabilised its API when reaching 1.0,
>> but bpftool taps into some of the internals of the library. Features
>> or new definitions are usually added at the same time to libbpf and
>> bpftool, and if you get a mismatch between the two, you're taking
>> risks to get build issues.
> 
> In other words no API exists.

Of course it does. Libbpf exposes a specific set of functions to user
applications.

But correct, from bpftool's perspective, there are a few locations where
we accept to derogate and to access to the internals directly, making it
more dependent on a specific version, or commit, of libbpf, and blurring
the notion of API.

This special relationship is nothing new though, and it has been
discussed before. It derives from both tools being developed in the same
repository, and bpftool being so tightly linked to libbpf - it has been
qualified of command-line interface for libbpf in the past. Bpftool's
version number itself is aligned on libbpf's. (As a side note, bpftool
used to pull libbpf's headers directly from libbpf's dir instead of
installing them locally, which facilitated this mix-in for
public/internal headers in the first place.)

I know you advocated making the required functions part of the API,
given that some users (such as bpftool) need them. These functions are
not exposed, by choice. They are not judged relevant to generic user
space application (I'm sure libbpf's maintainers are opened to
discussion if use cases come up). Some of the internals we get from
libbpf are also mostly to avoid re-implementing things, such as netlink
attributes processing, or implementing hash maps. These have nothing to
do in libbpf's API.

> 
>>> Using the GitHub mirror of bpftool to package both libbpf and bpftool would
>>> kept their version in sync, and was suggested[3]. Although the same could be
>>> said if we switch back to packaging libbpf from kernel source, an additional
>>> appeal for using GitHub mirrors is that it decouples bpftool from kernel,
>>> making it easily upgradable and with a clearer changelog (the latter is
>>> quite important for enterprise users) like libbpf.
>>
>> Happy to read these changelogs I write are useful to someone :). Yes,
>> this is my point.
> 
> Yes, publishing the changelog in a usable way relieves others of the
> need to write thier own, usually with less understanding of the changes
> in question. That's generally the idea of opensource - not endlessly
> repeating what has already been done :)

Not discussing that - My point is that I've received little feedback on
the mirror so far, so it's hard for me to figure out who's using it or
whether anyone has been reading the changelogs.

> 
>>> The main concern with using GitHub mirror is that bpftool may be updated far
>>> beyond the version that comes with the runtime kernel. AFAIK bpftool should
>>> work on older kernel since CO-RE is used for built-in BPF iterators and the
>>> underlying libbpf work on older kernel itself. Nonetheless, it would be nice
>>> to get a confirmation from the maintainers.
>>
>> As explained above - Mostly, it should work. Otherwise, we can look
>> into fixing it.
>>
>> As a side note, I'm open to suggestions/contributions to make life
>> easier for packaging for the mirror. For example, Mahé and I recently
>> added GitHub workflows to ship statically-built binaries for amd64 and
>> arm64 on releases, as well as tarballs with both bpftool+libbpf
>> sources. If there's something else to make packaging easier, I'm happy
>> to talk about it.
> 
> Make it possible to build with system-installed libbpf. If it's released
> it should have versioned dependency on a libbpf release, and libbpf from
> that version on should be good enough to build it.
> 
> I tried copying those 'private' headers into a separate directory, and
> link against static libbpf, and it seems to work. Of course, having
> an actual API would be much better.

Just as you said yourself, the missing stability is in the way. I don't
see this happening as long as bpftool is using libbpf's internals. I do
expect builds to work most of the time by copying the headers as you
did, but as soon as something changes and it no longer does, everyone
will start filing issues on GitHub instead of using the version that
works, and I don't want that.

As for decoupling from the internal: making the functions part of the
API is not an option. One option could be to move this code into further
dependencies shared between libbpf and bpftool - although I guess libbpf
developers will have little appetite for that. We could also duplicate
the necessary code in bpftool, which doesn't sound optimal, but might be
one solution. Other options have been discussed before, such as moving
bpftool into libbpf's directory/mirror and shipping both together, but
there was no consensus at the time, and I don't expect libbpf to ship
with bpftool any time soon.

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Michal Suchánek]

On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > Hello,
> > 
> > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> >> Hi Shung-Hsi,
> >>
> >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> >>> of using the source found in kernel), but realize that it should goes
> >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> >>> questions:
> >>>
> >>>   What is the suggested approach for packaging bpftool and libbpf?
> >>>   Which source is preferred, GitHub or kernel?
> >>
> >> As you can see from the previous discussions, the suggested approach
> >> would be to package from the GitHub mirror, with libbpf and bpftool in
> >> sync.
> >>
> >> My main argument for the mirror is that it keeps things simpler, and
> >> there's no need to deal with the rest of the kernel sources for these
> >> packages. Download from the mirrors, build, ship. But then I have
> >> limited experience at packaging for distros, and I can understand
> >> Toke's point of view, too. So ultimately, the call is yours.
> > 
> > Things get only ever more complex when submodules are involved.
> 
> I understand the generic pain points from your other email. But could
> you be more specific for the case of bpftool? It's not like we're
> shipping all lib dependencies as submodules. Sync-ups are specifically
> aligned to the same commit used to sync the libbpf mirror, so that it's
> pretty much as if we had the right version of the library shipped in the
> repository - only, it's one --recurse-submodules away.

It's so in every project that uses submodules. Except git does not
recurse into submodules by default, you have to fix it up by hand.
Forges don't support submodules so you will not get the submodule when
downloading the project archive, and won't see it the the project tree.

After previous experience with submodules I did not even try, I just
patched the makefile to use system libbpf before attempting anything
else.

> >>>   Does bpftool work on older kernel?
> >>
> >> It should, although it's not perfect. Most features from current
> >> bpftool should work as expected on older kernels. However, if I
> >> remember correctly you would have trouble loading programs on pre-BTF
> >> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
> >> definitions with BTF info, and attempts to create these maps with BTF,
> >> which fails and blocks the load process.
> >>
> >> But we're trying to keep backward-compatibility, so if we're only
> >> talking of kernels recent enough to support BTF, then I'd expect
> >> bpftool to work. If this is not the case, please report on this list.
> > 
> > It won't build:
> > https://lore.kernel.org/bpf/20220421003152.339542-3-alobakin@pm.me/
> 
> True in this case, and this is something that needs to get fixed. Thanks
> for reopening that thread! Are you building bpftool on kernels older
> than 5.15? (genuine curiosity)

Yes, 5.14 and 5.3. I would not be able to notice this particular
breakage otherwise.

> >>> Our current approach is that we (openSUSE/SLES) essentially have two version
> >>> of libbpf: a public shared library that uses GitHub mirror as source, which
> >>> the general userspace sees and links to; and a private static library built
> >>> from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
> >>> A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
> >>> took similar approach.
> >>
> >> I would like them to reconsider this choice eventually. Sounds like
> >> for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
> >> have a real bpftool package instead of having to install
> >> linux-tools-common + linux-tools-generic, or to have distros in
> >> general (Ubuntu/Debian at least) stop compiling out the JIT
> >> disassembler, although this is not strictly related to the location of
> >> the sources. I've not found the time to reach out to package
> >> maintainers yet.
> >>
> >>>
> >>> This approach means that the version of bpftool and libbpf are _not_ always
> >>> in sync[1], which I read may causes problem since libbpf and bpftool depends
> >>> on specific version of each other[2].
> >>
> >> Whatever source you use, I would strongly recommend finding a way to
> >> keep both in sync. Libbpf has stabilised its API when reaching 1.0,
> >> but bpftool taps into some of the internals of the library. Features
> >> or new definitions are usually added at the same time to libbpf and
> >> bpftool, and if you get a mismatch between the two, you're taking
> >> risks to get build issues.
> > 
> > In other words no API exists.
> 
> Of course it does. Libbpf exposes a specific set of functions to user
> applications.
> 
> But correct, from bpftool's perspective, there are a few locations where
> we accept to derogate and to access to the internals directly, making it
> more dependent on a specific version, or commit, of libbpf, and blurring
> the notion of API.
> 
> This special relationship is nothing new though, and it has been
> discussed before. It derives from both tools being developed in the same
> repository, and bpftool being so tightly linked to libbpf - it has been
> qualified of command-line interface for libbpf in the past. Bpftool's

If bpftool is a commandline interface for libbpf maybe the best choice
is to just dump both into the same repository, and provide make targets
for building one, the other, and both.

> version number itself is aligned on libbpf's. (As a side note, bpftool
> used to pull libbpf's headers directly from libbpf's dir instead of
> installing them locally, which facilitated this mix-in for
> public/internal headers in the first place.)
> 
> I know you advocated making the required functions part of the API,
> given that some users (such as bpftool) need them. These functions are
> not exposed, by choice. They are not judged relevant to generic user
> space application (I'm sure libbpf's maintainers are opened to
> discussion if use cases come up). Some of the internals we get from
> libbpf are also mostly to avoid re-implementing things, such as netlink
> attributes processing, or implementing hash maps. These have nothing to
> do in libbpf's API.

And we do not have microframeworks for implementing reusable hashmaps or
netlink parsers. I am sure that bpftool is not the first nor last tool
that needs a hashmap or parse netlink but the ecosystem for small
single-purpose C libraries never really took off.

> >>> The main concern with using GitHub mirror is that bpftool may be updated far
> >>> beyond the version that comes with the runtime kernel. AFAIK bpftool should
> >>> work on older kernel since CO-RE is used for built-in BPF iterators and the
> >>> underlying libbpf work on older kernel itself. Nonetheless, it would be nice
> >>> to get a confirmation from the maintainers.
> >>
> >> As explained above - Mostly, it should work. Otherwise, we can look
> >> into fixing it.
> >>
> >> As a side note, I'm open to suggestions/contributions to make life
> >> easier for packaging for the mirror. For example, Mahé and I recently
> >> added GitHub workflows to ship statically-built binaries for amd64 and
> >> arm64 on releases, as well as tarballs with both bpftool+libbpf
> >> sources. If there's something else to make packaging easier, I'm happy
> >> to talk about it.
> > 
> > Make it possible to build with system-installed libbpf. If it's released
> > it should have versioned dependency on a libbpf release, and libbpf from
> > that version on should be good enough to build it.
> > 
> > I tried copying those 'private' headers into a separate directory, and
> > link against static libbpf, and it seems to work. Of course, having
> > an actual API would be much better.
> 
> Just as you said yourself, the missing stability is in the way. I don't
> see this happening as long as bpftool is using libbpf's internals. I do
> expect builds to work most of the time by copying the headers as you
> did, but as soon as something changes and it no longer does, everyone
> will start filing issues on GitHub instead of using the version that
> works, and I don't want that.

So these are not really separate projects, they should ship together.

> As for decoupling from the internal: making the functions part of the
> API is not an option. One option could be to move this code into further
> dependencies shared between libbpf and bpftool - although I guess libbpf
> developers will have little appetite for that. We could also duplicate
> the necessary code in bpftool, which doesn't sound optimal, but might be
> one solution. Other options have been discussed before, such as moving
> bpftool into libbpf's directory/mirror and shipping both together, but
> there was no consensus at the time, and I don't expect libbpf to ship
> with bpftool any time soon.

Why is that? Is there any speciffic reason for the sepaaration if
bpftool needs specific libbpf anyway?

Thanks

Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Thu, Apr 13, 2023 at 5:35 PM Quentin Monnet <quentin@isovalent.com> wrote:
>
> Hi Shung-Hsi,
>
> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >
> > Hi,
> >
> > I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > of using the source found in kernel), but realize that it should goes
> > hand-in-hand with how libbpf is packaged, which eventually leads these
> > questions:
> >
> >   What is the suggested approach for packaging bpftool and libbpf?
> >   Which source is preferred, GitHub or kernel?
>
> As you can see from the previous discussions, the suggested approach
> would be to package from the GitHub mirror, with libbpf and bpftool in
> sync.
>
> My main argument for the mirror is that it keeps things simpler, and
> there's no need to deal with the rest of the kernel sources for these
> packages. Download from the mirrors, build, ship. But then I have
> limited experience at packaging for distros, and I can understand
> Toke's point of view, too. So ultimately, the call is yours.
>
> >   Does bpftool work on older kernel?
>
> It should, although it's not perfect. Most features from current
> bpftool should work as expected on older kernels. However, if I
> remember correctly you would have trouble loading programs on pre-BTF
> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
> definitions with BTF info, and attempts to create these maps with BTF,
> which fails and blocks the load process.

Not really. Libbpf won't upload BTF if the kernel doesn't support it
and it's not required for correct BPF map (e.g., local storage) or BPF
program functioning.

>
> But we're trying to keep backward-compatibility, so if we're only
> talking of kernels recent enough to support BTF, then I'd expect
> bpftool to work. If this is not the case, please report on this list.
>
> >
> > Our current approach is that we (openSUSE/SLES) essentially have two version
> > of libbpf: a public shared library that uses GitHub mirror as source, which
> > the general userspace sees and links to; and a private static library built
> > from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
> > A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
> > took similar approach.
>
> I would like them to reconsider this choice eventually. Sounds like
> for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
> have a real bpftool package instead of having to install
> linux-tools-common + linux-tools-generic, or to have distros in
> general (Ubuntu/Debian at least) stop compiling out the JIT
> disassembler, although this is not strictly related to the location of
> the sources. I've not found the time to reach out to package
> maintainers yet.
>
> >
> > This approach means that the version of bpftool and libbpf are _not_ always
> > in sync[1], which I read may causes problem since libbpf and bpftool depends
> > on specific version of each other[2].
>
> Whatever source you use, I would strongly recommend finding a way to
> keep both in sync. Libbpf has stabilised its API when reaching 1.0,
> but bpftool taps into some of the internals of the library. Features
> or new definitions are usually added at the same time to libbpf and
> bpftool, and if you get a mismatch between the two, you're taking
> risks to get build issues.
>
> >
> > Using the GitHub mirror of bpftool to package both libbpf and bpftool would
> > kept their version in sync, and was suggested[3]. Although the same could be
> > said if we switch back to packaging libbpf from kernel source, an additional
> > appeal for using GitHub mirrors is that it decouples bpftool from kernel,
> > making it easily upgradable and with a clearer changelog (the latter is
> > quite important for enterprise users) like libbpf.
>
> Happy to read these changelogs I write are useful to someone :). Yes,
> this is my point.
>
> >
> > The main concern with using GitHub mirror is that bpftool may be updated far
> > beyond the version that comes with the runtime kernel. AFAIK bpftool should
> > work on older kernel since CO-RE is used for built-in BPF iterators and the
> > underlying libbpf work on older kernel itself. Nonetheless, it would be nice
> > to get a confirmation from the maintainers.
>
> As explained above - Mostly, it should work. Otherwise, we can look
> into fixing it.
>
> As a side note, I'm open to suggestions/contributions to make life
> easier for packaging for the mirror. For example, Mahé and I recently
> added GitHub workflows to ship statically-built binaries for amd64 and
> arm64 on releases, as well as tarballs with both bpftool+libbpf
> sources. If there's something else to make packaging easier, I'm happy
> to talk about it.

Please contribute this for veristat ([0]) and retsnoop ([1]). I've
been packaging submodule sources for them using a simple script ([2]),
but if this can be automated, that would be awesome. Thanks!

  [0] https://github.com/libbpf/veristat/
  [1] https://github.com/anakryiko/retsnoop/
  [2] https://github.com/anakryiko/retsnoop/blob/master/scripts/archive-srcs-full.sh

>
> >
> > Are there any other downsides to switching to GitHub mirror for bpftool?
>
> Nothing else comes to mind, but again I'm not packaging for distros.

+1.

Libbpf, bpftool, and other libbpf-dependent tools are meant to be
backwards compatible with old kernels as much as possible. Bugs do
happen, but those should be reported and fixed. Otherwise if some
newer feature is requiring a newer kernel, we always try to develop it
in a way that will fail gracefully. So there is no reason to not
switch to Github-based mirrors, for both libbpf and bpftool.

> See Toke's message.
>
> I hope this helps,
> Quentin

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
>
> On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > Hello,
> > >
> > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > >> Hi Shung-Hsi,
> > >>
> > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > >>> of using the source found in kernel), but realize that it should goes
> > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > >>> questions:
> > >>>
> > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > >>>   Which source is preferred, GitHub or kernel?
> > >>
> > >> As you can see from the previous discussions, the suggested approach
> > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > >> sync.
> > >>
> > >> My main argument for the mirror is that it keeps things simpler, and
> > >> there's no need to deal with the rest of the kernel sources for these
> > >> packages. Download from the mirrors, build, ship. But then I have
> > >> limited experience at packaging for distros, and I can understand
> > >> Toke's point of view, too. So ultimately, the call is yours.
> > >
> > > Things get only ever more complex when submodules are involved.
> >
> > I understand the generic pain points from your other email. But could
> > you be more specific for the case of bpftool? It's not like we're
> > shipping all lib dependencies as submodules. Sync-ups are specifically
> > aligned to the same commit used to sync the libbpf mirror, so that it's
> > pretty much as if we had the right version of the library shipped in the
> > repository - only, it's one --recurse-submodules away.
>
> It's so in every project that uses submodules. Except git does not
> recurse into submodules by default, you have to fix it up by hand.
> Forges don't support submodules so you will not get the submodule when
> downloading the project archive, and won't see it the the project tree.

git submodule update --init --recursive didn't work?

>
> After previous experience with submodules I did not even try, I just
> patched the makefile to use system libbpf before attempting anything
> else.

Quentin mentioned that he's packaging (or will package) libbpf sources
as part of bpftool release on Github. I've been this for other
libbpf-using tools as well, and it works pretty well (at least for
Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])

By switching up actual libbpf used to compile with bpftool, you are
potentially introducing subtle problems that your users will be quite
unhappy about, if they run into them. Let's work together to make it
easier for you to package bpftool properly. We can't switch bpftool to
reliably use system-wide libbpf (either static or shared, doesn't
matter) because of dependency on internal functionality.


  [0] https://github.com/libbpf/veristat/releases/tag/v0.1

>
> > >>>   Does bpftool work on older kernel?
> > >>
> > >> It should, although it's not perfect. Most features from current
> > >> bpftool should work as expected on older kernels. However, if I
> > >> remember correctly you would have trouble loading programs on pre-BTF
> > >> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
> > >> definitions with BTF info, and attempts to create these maps with BTF,
> > >> which fails and blocks the load process.
> > >>
> > >> But we're trying to keep backward-compatibility, so if we're only
> > >> talking of kernels recent enough to support BTF, then I'd expect
> > >> bpftool to work. If this is not the case, please report on this list.
> > >
> > > It won't build:
> > > https://lore.kernel.org/bpf/20220421003152.339542-3-alobakin@pm.me/
> >
> > True in this case, and this is something that needs to get fixed. Thanks
> > for reopening that thread! Are you building bpftool on kernels older
> > than 5.15? (genuine curiosity)
>
> Yes, 5.14 and 5.3. I would not be able to notice this particular
> breakage otherwise.
>
> > >>> Our current approach is that we (openSUSE/SLES) essentially have two version
> > >>> of libbpf: a public shared library that uses GitHub mirror as source, which
> > >>> the general userspace sees and links to; and a private static library built
> > >>> from kernel source used by bpftool, perf, resolve_btfids, selftests, etc.
> > >>> A survey of other distros (Arch, Debian, Fedora, Ubuntu) suggest that they
> > >>> took similar approach.
> > >>
> > >> I would like them to reconsider this choice eventually. Sounds like
> > >> for RHEL, this will be a tough sell :). At least, I'd love Ubuntu to
> > >> have a real bpftool package instead of having to install
> > >> linux-tools-common + linux-tools-generic, or to have distros in
> > >> general (Ubuntu/Debian at least) stop compiling out the JIT
> > >> disassembler, although this is not strictly related to the location of
> > >> the sources. I've not found the time to reach out to package
> > >> maintainers yet.
> > >>
> > >>>
> > >>> This approach means that the version of bpftool and libbpf are _not_ always
> > >>> in sync[1], which I read may causes problem since libbpf and bpftool depends
> > >>> on specific version of each other[2].
> > >>
> > >> Whatever source you use, I would strongly recommend finding a way to
> > >> keep both in sync. Libbpf has stabilised its API when reaching 1.0,
> > >> but bpftool taps into some of the internals of the library. Features
> > >> or new definitions are usually added at the same time to libbpf and
> > >> bpftool, and if you get a mismatch between the two, you're taking
> > >> risks to get build issues.
> > >
> > > In other words no API exists.
> >
> > Of course it does. Libbpf exposes a specific set of functions to user
> > applications.
> >
> > But correct, from bpftool's perspective, there are a few locations where
> > we accept to derogate and to access to the internals directly, making it
> > more dependent on a specific version, or commit, of libbpf, and blurring
> > the notion of API.
> >
> > This special relationship is nothing new though, and it has been
> > discussed before. It derives from both tools being developed in the same
> > repository, and bpftool being so tightly linked to libbpf - it has been
> > qualified of command-line interface for libbpf in the past. Bpftool's
>
> If bpftool is a commandline interface for libbpf maybe the best choice
> is to just dump both into the same repository, and provide make targets
> for building one, the other, and both.

bpftool is way more than just an interface to libbpf, so it doesn't
make sense to bundle them in one repo. Yes, we take advantage of them
being developed together to give it more and better features, which
otherwise we probably just wouldn't add to either libbpf or bpftool
(like BTFgen, for example).

This whole submodule vs shared library has been discussed so many
times. We are open to making your life easier as a packager where
possible, if you can be specific about the issues that are hurting
you. But this relationship between bpftool and libbpf and use of
internal APIs was a conscious decision and it has its big benefits,
which definitely outweigh git submodule inconvenience. We are not
changing it, it's pretty fundamental for bpftool.

>
> > version number itself is aligned on libbpf's. (As a side note, bpftool
> > used to pull libbpf's headers directly from libbpf's dir instead of
> > installing them locally, which facilitated this mix-in for
> > public/internal headers in the first place.)
> >
> > I know you advocated making the required functions part of the API,
> > given that some users (such as bpftool) need them. These functions are
> > not exposed, by choice. They are not judged relevant to generic user
> > space application (I'm sure libbpf's maintainers are opened to
> > discussion if use cases come up). Some of the internals we get from
> > libbpf are also mostly to avoid re-implementing things, such as netlink
> > attributes processing, or implementing hash maps. These have nothing to
> > do in libbpf's API.
>
> And we do not have microframeworks for implementing reusable hashmaps or
> netlink parsers. I am sure that bpftool is not the first nor last tool
> that needs a hashmap or parse netlink but the ecosystem for small
> single-purpose C libraries never really took off.
>
> > >>> The main concern with using GitHub mirror is that bpftool may be updated far
> > >>> beyond the version that comes with the runtime kernel. AFAIK bpftool should
> > >>> work on older kernel since CO-RE is used for built-in BPF iterators and the
> > >>> underlying libbpf work on older kernel itself. Nonetheless, it would be nice
> > >>> to get a confirmation from the maintainers.
> > >>
> > >> As explained above - Mostly, it should work. Otherwise, we can look
> > >> into fixing it.
> > >>
> > >> As a side note, I'm open to suggestions/contributions to make life
> > >> easier for packaging for the mirror. For example, Mahé and I recently
> > >> added GitHub workflows to ship statically-built binaries for amd64 and
> > >> arm64 on releases, as well as tarballs with both bpftool+libbpf
> > >> sources. If there's something else to make packaging easier, I'm happy
> > >> to talk about it.
> > >
> > > Make it possible to build with system-installed libbpf. If it's released
> > > it should have versioned dependency on a libbpf release, and libbpf from
> > > that version on should be good enough to build it.
> > >
> > > I tried copying those 'private' headers into a separate directory, and
> > > link against static libbpf, and it seems to work. Of course, having
> > > an actual API would be much better.
> >
> > Just as you said yourself, the missing stability is in the way. I don't
> > see this happening as long as bpftool is using libbpf's internals. I do
> > expect builds to work most of the time by copying the headers as you
> > did, but as soon as something changes and it no longer does, everyone
> > will start filing issues on GitHub instead of using the version that
> > works, and I don't want that.
>
> So these are not really separate projects, they should ship together.
>
> > As for decoupling from the internal: making the functions part of the
> > API is not an option. One option could be to move this code into further
> > dependencies shared between libbpf and bpftool - although I guess libbpf
> > developers will have little appetite for that. We could also duplicate
> > the necessary code in bpftool, which doesn't sound optimal, but might be
> > one solution. Other options have been discussed before, such as moving
> > bpftool into libbpf's directory/mirror and shipping both together, but
> > there was no consensus at the time, and I don't expect libbpf to ship
> > with bpftool any time soon.
>
> Why is that? Is there any speciffic reason for the sepaaration if
> bpftool needs specific libbpf anyway?
>
> Thanks
>
> Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

On Fri, Apr 14, 2023 at 02:12:40AM +0100, Quentin Monnet wrote:
> On Thu, 13 Apr 2023 at 10:50, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > On Thu, Apr 13, 2023 at 05:23:16PM +0800, Shung-Hsi Yu wrote:
> > > Hi,
> > >
> > > I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > of using the source found in kernel), but realize that it should goes
> > > hand-in-hand with how libbpf is packaged, which eventually leads these
> > > questions:
> > >
> > >   What is the suggested approach for packaging bpftool and libbpf?
> > >   Which source is preferred, GitHub or kernel?
> >
> > An off-topic, yet somewhat related question that I also tried to figure out
> > is "why the GitHub mirror for libbpf and bpftool exist at the first place?".
> > It is a non-trivial amount of work for the maintainers after all.
> >
> > For libbpf, the main uses case for GitHub seem to be for it to be used as
> > submodule for other projects (e.g. pahole[1]), and that alone seem to suffice.
> 
> Then it should be enough for bpftool, too :) The bcc repository uses
> it as a submodule, for example.

Hmm... bcc having bpftool as a submodule is a news to me. bcc having libbpf
as a submodule is quite straight forward, but I didn't know bcc depends on
bpftool as well.

> The work is non-trivial, but when compared to libbpf, I managed to
> preserve most of the Makefile from the kernel tree and all of the C
> code, and bpftool also gets patches less often.
> 
> >
> > For bpftool the reason seems to be less clear[2]. From what I can tell right
> > now its mainly use for CI (this applies to libbpf as well), which is
> > definitely useful.
> 
> Yes. At the moment, the CI present on bpftool's mirror is more limited
> than libbpf's. But it allows me to test some compilation variants:
> regular builds, static builds, cross-compiling (to some extent). Some
> additional checks that would make little sense to have in the kernel
> repo, too. It's mostly for checking that none of these build
> configurations break when I sync from the kernel, and helped me find
> and fix several issues in the build system on the mirror.
> 
> This CI on the mirror doesn't cover bpftool's features, but these
> should be tested in the BPF CI itself, so we can catch regressions
> before patches are merged. There are some tests already, not many, I'd
> like to improve that someday. Anyway.
> 
> >
> > But I wonder whether packaging one of the motives to create the mirrors
> > initially? Can't seem to find anything in this regard.
> >
> >
> > 1: https://github.com/acmel/dwarves/tree/master/lib
> > 2: https://lore.kernel.org/bpf/CAEf4BzZ+0XpH_zJ0P78vjzmFAH3kGZ21w3-LcSEG=B=+ZQWJ=w@mail.gmail.com/
> 
> It seems like you haven't come across this one?:
> https://lore.kernel.org/bpf/267a35a6-a045-c025-c2d9-78afbf6fc325@isovalent.com/t/

That has all I wanted to know and more, not sure how I was able to missed
that, my Xapian searching skill really needs more work :)

> Yes, easing packaging was one of the motivations for the mirror. As
> mentioned in my other answer, I've not taken the time to reach out to
> package maintainers yet, so this hasn't really materialised at this
> point.
> 
> CI, indeed, was another motivation.
> 
> Submodules, or simply making it easier to hack with bpftool's code,
> was yet another thing. Microsoft folks intend to make bpftool
> compatible with eBPF for Windows. It's quite simpler to work on that
> from a repo which is mostly uncoupled from the Linux tree.
> 
> Perhaps the most important was to make it easier to just download,
> build and use bpftool, for all users who need to get the latest
> version, or to patch it, or to create static builds, or to
> cross-compile, or whatever reason might cause you to compile it from
> the sources. For all those cases, getting the mirror is faster and
> requires less space than getting the kernel repo. This makes a nice
> difference when periodically rebuilding images in automated workflows,
> for example.

While for distro packaging the net benefit is somewhat unclear (after all
that the point of this thread), I very much agree with the above, this does
give users much greater control over the bpftool they can choose to use.

> Does this answer your questions?

Yes, thanks!

> Quentin

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Michal Suchánek]

On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> >
> > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > Hello,
> > > >
> > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > >> Hi Shung-Hsi,
> > > >>
> > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > >>> of using the source found in kernel), but realize that it should goes
> > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > >>> questions:
> > > >>>
> > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > >>>   Which source is preferred, GitHub or kernel?
> > > >>
> > > >> As you can see from the previous discussions, the suggested approach
> > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > >> sync.
> > > >>
> > > >> My main argument for the mirror is that it keeps things simpler, and
> > > >> there's no need to deal with the rest of the kernel sources for these
> > > >> packages. Download from the mirrors, build, ship. But then I have
> > > >> limited experience at packaging for distros, and I can understand
> > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > >
> > > > Things get only ever more complex when submodules are involved.
> > >
> > > I understand the generic pain points from your other email. But could
> > > you be more specific for the case of bpftool? It's not like we're
> > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > pretty much as if we had the right version of the library shipped in the
> > > repository - only, it's one --recurse-submodules away.
> >
> > It's so in every project that uses submodules. Except git does not
> > recurse into submodules by default, you have to fix it up by hand.
> > Forges don't support submodules so you will not get the submodule when
> > downloading the project archive, and won't see it the the project tree.
> 
> git submodule update --init --recursive didn't work?

That's one part of the manual fixup.

The other part is after each git operation that could possibly cause the
submodules to go out of sync, basically any operation that changes the
checked-out commit.

Of course, you can make some shell aliases that append whatever submodule
chicanery to whatever git command you might issue, and tell everyone
else to do that, and then it will work in that one shell, and not in any
other shell nor any tool that invokes git directly.

> > After previous experience with submodules I did not even try, I just
> > patched the makefile to use system libbpf before attempting anything
> > else.
> 
> Quentin mentioned that he's packaging (or will package) libbpf sources
> as part of bpftool release on Github. I've been this for other
> libbpf-using tools as well, and it works pretty well (at least for
> Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
> 
> By switching up actual libbpf used to compile with bpftool, you are
> potentially introducing subtle problems that your users will be quite
> unhappy about, if they run into them. Let's work together to make it
> easier for you to package bpftool properly. We can't switch bpftool to
> reliably use system-wide libbpf (either static or shared, doesn't
> matter) because of dependency on internal functionality.
> 
> 
>   [0] https://github.com/libbpf/veristat/releases/tag/v0.1

So how many copies of libbpf do I need for having a CO-RE toolchain?

Will different tools have different view of the kernel because they each
use different private copy of libbpf with different features?

When there is a bug in libbpf how many places need to be patched to fix
it?

Thanks

Michal

RE: Packaging bpftool and libbpf: GitHub or kernel?

[Dave Thaler]

Quentin Monnet <quentin@isovalent.com> wrote:
> My point is that I've received little feedback on the mirror
> so far, so it's hard for me to figure out who's using it or whether anyone has
> been reading the changelogs.

FWIW, the ebpf-for-windows project uses both bpftool and libbpf as git submodules.
Both of them are, today, very Linux-centric and not usable directly cross-platform (something I'd like to see change in the future as I've previously discussed but not
had enough cycles to drive much).

So today the libbpf mirror is used as a submodule just to get the header files,
where ebpf-for-windows currently uses its own C implementation of the same
prototypes, to map to Windows ioctls instead of Linux syscalls.

And today bpftool is used via a github fork of the mirror, where various ifdefs
and Windows-specific alternatives are in the fork.  The intent is to clean it up
and upstream it, but I just haven't gotten to that yet.  But ebpf-for-windows
does not currently support BTF so it is like an older Linux kernel in that sense.

So we haven't updated either to commits that rely on libbpf >= 1.0 since
the APIs we support were removed.

> >>>   Does bpftool work on older kernel?
> >>
> >> It should, although it's not perfect. Most features from current
> >> bpftool should work as expected on older kernels. However, if I
> >> remember correctly you would have trouble loading programs on pre-BTF
> >> kernels, because bpftool relies on libbpf >= 1.0 and only accepts map
> >> definitions with BTF info, and attempts to create these maps with
> >> BTF, which fails and blocks the load process.
> >>
> >> But we're trying to keep backward-compatibility, so if we're only
> >> talking of kernels recent enough to support BTF, then I'd expect
> >> bpftool to work. If this is not the case, please report on this list.

Yeah, unless/until we support BTF in the future, we cannot use the current
HEAD of the libbpf or bpftool mirrors, only older commits we sync to.

In that sense, the removal of pre-1.0 APIs made the mirrors more
Linux-only and less cross-platform.

Dave

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
>
> On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > >
> > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > > Hello,
> > > > >
> > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > > >> Hi Shung-Hsi,
> > > > >>
> > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > > >>>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > > >>> of using the source found in kernel), but realize that it should goes
> > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > > >>> questions:
> > > > >>>
> > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > > >>>   Which source is preferred, GitHub or kernel?
> > > > >>
> > > > >> As you can see from the previous discussions, the suggested approach
> > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > > >> sync.
> > > > >>
> > > > >> My main argument for the mirror is that it keeps things simpler, and
> > > > >> there's no need to deal with the rest of the kernel sources for these
> > > > >> packages. Download from the mirrors, build, ship. But then I have
> > > > >> limited experience at packaging for distros, and I can understand
> > > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > > >
> > > > > Things get only ever more complex when submodules are involved.
> > > >
> > > > I understand the generic pain points from your other email. But could
> > > > you be more specific for the case of bpftool? It's not like we're
> > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > > pretty much as if we had the right version of the library shipped in the
> > > > repository - only, it's one --recurse-submodules away.
> > >
> > > It's so in every project that uses submodules. Except git does not
> > > recurse into submodules by default, you have to fix it up by hand.
> > > Forges don't support submodules so you will not get the submodule when
> > > downloading the project archive, and won't see it the the project tree.
> >
> > git submodule update --init --recursive didn't work?
>
> That's one part of the manual fixup.
>
> The other part is after each git operation that could possibly cause the
> submodules to go out of sync, basically any operation that changes the
> checked-out commit.
>
> Of course, you can make some shell aliases that append whatever submodule
> chicanery to whatever git command you might issue, and tell everyone
> else to do that, and then it will work in that one shell, and not in any
> other shell nor any tool that invokes git directly.

Are we discussing a *standard* Git submodule feature and argue that
because it might be cumbersome or unfamiliar to some engineers that
projects should avoid using Git submodules?

For one, I don't have any special aliases for dealing with Git
submodules and it works fine. If I jump between branches or tags which
update Git submodule reference, I do above `git submodule update
--init --recursive` explicitly if I see that Git status shows
out-of-sync Git submodule state. If I want to update a Git submodule,
I update the submodule's Git repo, and then git add it in the repo
that uses this submodule. I haven't run into any other issues with
this.

Having said that, I do realize that some build systems have more
troubles with Git submodules (this was a complaint from Fedora
packagers), and I empathize with this, which is why I do the archiving
of submodule sources when cutting releases. If you'd prefer some other
way of dealing with this, please let's have a constructive discussion
about that. Suggesting projects to stop using Git submodules isn't
that, IMO.

>
> > > After previous experience with submodules I did not even try, I just
> > > patched the makefile to use system libbpf before attempting anything
> > > else.
> >
> > Quentin mentioned that he's packaging (or will package) libbpf sources
> > as part of bpftool release on Github. I've been this for other
> > libbpf-using tools as well, and it works pretty well (at least for
> > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
> >
> > By switching up actual libbpf used to compile with bpftool, you are
> > potentially introducing subtle problems that your users will be quite
> > unhappy about, if they run into them. Let's work together to make it
> > easier for you to package bpftool properly. We can't switch bpftool to
> > reliably use system-wide libbpf (either static or shared, doesn't
> > matter) because of dependency on internal functionality.
> >
> >
> >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
>
> So how many copies of libbpf do I need for having a CO-RE toolchain?

What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
etc are tools. The fact they are using statically linked libbpf
through Git submodule is irrelevant to end users. You need one libbpf
in the system (for those who link dynamically against libbpf), the
rest are just tools.

>
> Will different tools have different view of the kernel because they each
> use different private copy of libbpf with different features?

That's up to tools, not libbpf. You are over pivoting on libbpf here.
There is one view of the kernel, it depends on what features the
kernel supports. If the tool requires some specific functionality of
libbpf, it will update its Git submodule reference to get a version of
libbpf that provides that feature. That's the point, an
application/tool is in control of what kind of features it gets from
libbpf.

>
> When there is a bug in libbpf how many places need to be patched to fix
> it?

That's up to tools, again. If the bug is affecting them, they should
cut a new version of their *tool*, using a patched version of libbpf
from Github. If it doesn't affect them, then it doesn't matter *to
them*.

>
> Thanks
>
> Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Michal Suchánek]

On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
> On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
> >
> > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > > >
> > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > > > Hello,
> > > > > >
> > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > > > >> Hi Shung-Hsi,
> > > > > >>
> > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > > > >>>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > > > >>> of using the source found in kernel), but realize that it should goes
> > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > > > >>> questions:
> > > > > >>>
> > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > > > >>>   Which source is preferred, GitHub or kernel?
> > > > > >>
> > > > > >> As you can see from the previous discussions, the suggested approach
> > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > > > >> sync.
> > > > > >>
> > > > > >> My main argument for the mirror is that it keeps things simpler, and
> > > > > >> there's no need to deal with the rest of the kernel sources for these
> > > > > >> packages. Download from the mirrors, build, ship. But then I have
> > > > > >> limited experience at packaging for distros, and I can understand
> > > > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > > > >
> > > > > > Things get only ever more complex when submodules are involved.
> > > > >
> > > > > I understand the generic pain points from your other email. But could
> > > > > you be more specific for the case of bpftool? It's not like we're
> > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > > > pretty much as if we had the right version of the library shipped in the
> > > > > repository - only, it's one --recurse-submodules away.
> > > >
> > > > It's so in every project that uses submodules. Except git does not
> > > > recurse into submodules by default, you have to fix it up by hand.
> > > > Forges don't support submodules so you will not get the submodule when
> > > > downloading the project archive, and won't see it the the project tree.
> > >
> > > git submodule update --init --recursive didn't work?
> >
> > That's one part of the manual fixup.
> >
> > The other part is after each git operation that could possibly cause the
> > submodules to go out of sync, basically any operation that changes the
> > checked-out commit.
> >
> > Of course, you can make some shell aliases that append whatever submodule
> > chicanery to whatever git command you might issue, and tell everyone
> > else to do that, and then it will work in that one shell, and not in any
> > other shell nor any tool that invokes git directly.
> 
> Are we discussing a *standard* Git submodule feature and argue that
> because it might be cumbersome or unfamiliar to some engineers that
> projects should avoid using Git submodules?

As far as I am aware they are unfamiliar to *most* engineers, and for
good reasons.

> For one, I don't have any special aliases for dealing with Git
> submodules and it works fine. If I jump between branches or tags which
> update Git submodule reference, I do above `git submodule update
> --init --recursive` explicitly if I see that Git status shows
> out-of-sync Git submodule state. If I want to update a Git submodule,
> I update the submodule's Git repo, and then git add it in the repo
> that uses this submodule. I haven't run into any other issues with
> this.

You know, git could just handle submodules automagically. As you say,
it's not rocket science. For historical reasons it does not.

With that working with submodules is cumbersome, and it's additional
thing that can break down that the engineer needs to be constantly aware
of increasing the mental overhead of working with such projects.

It may not be much of a problem for people who work with such projects
daily but not everyone does. Those who don't need to do the mental
switch whenever submodules are encountered, and are prone to getting
issues when they forget that they have to go that extra mile for this
specific project.

> > > > After previous experience with submodules I did not even try, I just
> > > > patched the makefile to use system libbpf before attempting anything
> > > > else.
> > >
> > > Quentin mentioned that he's packaging (or will package) libbpf sources
> > > as part of bpftool release on Github. I've been this for other
> > > libbpf-using tools as well, and it works pretty well (at least for
> > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
> > >
> > > By switching up actual libbpf used to compile with bpftool, you are
> > > potentially introducing subtle problems that your users will be quite
> > > unhappy about, if they run into them. Let's work together to make it
> > > easier for you to package bpftool properly. We can't switch bpftool to
> > > reliably use system-wide libbpf (either static or shared, doesn't
> > > matter) because of dependency on internal functionality.
> > >
> > >
> > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> >
> > So how many copies of libbpf do I need for having a CO-RE toolchain?
> 
> What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> etc are tools. The fact they are using statically linked libbpf
> through Git submodule is irrelevant to end users. You need one libbpf
> in the system (for those who link dynamically against libbpf), the
> rest are just tools.
> 
> >
> > Will different tools have different view of the kernel because they each
> > use different private copy of libbpf with different features?
> 
> That's up to tools, not libbpf. You are over pivoting on libbpf here.
> There is one view of the kernel, it depends on what features the
> kernel supports. If the tool requires some specific functionality of
> libbpf, it will update its Git submodule reference to get a version of
> libbpf that provides that feature. That's the point, an
> application/tool is in control of what kind of features it gets from
> libbpf.
> 
> >
> > When there is a bug in libbpf how many places need to be patched to fix
> > it?
> 
> That's up to tools, again. If the bug is affecting them, they should
> cut a new version of their *tool*, using a patched version of libbpf
> from Github. If it doesn't affect them, then it doesn't matter *to
> them*.

I don't share your optimism about this happening in the real world.

For one the issue that the github tarballs do not contain the submodule
and thus cannot be built was raised nearly two months ago, and while a
test snapshot that does include the submodule is released, a release
does not exist yet.

For people to make use of the repository without a release cut they need
to replicate that submodule support - that is add support for submodules
in their development tooling. Otherwise you personally cutting a release
becomes a single point of failure.

Because there is no API it's not really advisable to just apply patches
on top of the last release either. Applying patches may cause the main
project and the submodule to go out of sync, the submodule would not get
updated by applying a patch to the main project, and the other way
around.

Suppose a severe security bug that requires patching libbpf is found.
Now there is a number of tools that are each tied to one specific
version of libbpf, and cannot be upgraded to up-to-date fixed version
because that would break them. I would hope that never happens.
Nonetheless, libbpf is used to generate code, and if the code is
generated wrong worst case it can have severe security implications.

Thanks

Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

Thanks for sharing! I though I'd expands on what you said to draw a clearer
picture of the challenges.

On Thu, Apr 13, 2023 at 01:00:20PM +0200, Toke Høiland-Jørgensen wrote:
> Shung-Hsi Yu <shung-hsi.yu@suse.com> writes:
> 
> > A side note: if we want all userspace visible libbpf to have a coherent
> > version, perf needs to use the shared libbpf library as well (either
> > autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to
> > backport patches to kernel source to keep up with userspace package (libbpf)
> > changes could be a pain.

Here some more context for completeness. Kernel source changes are published
at a much slower pace than userspace. When an application in the kernel
source (e.g. perf) depends on the userspace library, it's kind of like
trying to catchup a car on a bike, which is doable, as evident by the
plethora of userspace libraries perf already depends on. While I don't
having experience maintaining perf, judging by tools/perf/Makefile.config
that does not seem like an easy feat.

For perf to use libbpf in kernel would mean that it's just depending on
something that moves at the same pace.

That said, maybe perf won't need additional backport to keep up with libbpf
as long as we keep it within that same major version (and disable
deprecation warning)? @Andrii

Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.

> So basically, this here is the reason we're building libbpf from the
> kernel tree for the RHEL package: If we use the github version we'd need
> to juggle two different versions of libbpf, one for the in-kernel-tree
> users (perf as you mention, but also the BPF selftests), and one for the
> userspace packages. Also, having libbpf in the kernel tree means we can
> just backport patches to it along with the BPF-related kernel patches
> (we do quite extensive BPF backports for each RHEL version).

> Finally, building from the kernel tree means we can use the existing
> kernel-related procedures for any out of order hotfixes (since AFAIK none
> of the github repositories have any concept of stable branches that
> receive fixes).

+1

Got something similar in place as well and being able to stick with existing
procedure is appealing. 

> YMMV of course, but figured I'd share our reasoning. To be clear,
> building from the kernel tree is not without its own pain points (mostly
> related to how the build scripts are structured for our kernel builds).
> We've discussed moving to the github version of libbpf multiple times,
> but every time we've concluded that it would be more, not less, painful
> than having the kernel tree be the single source of truth.

We package maintainer are certainly quite hard to please :)

Just having an individual package easy to work with is not enough, we want
it to be easier for most packages before jumping on the bandwagon, which is
why this email ended up talking about perf despite it started as a
discussion on packaging libbpf and bpftool.

I suppose the mileage depends on the build system & scripts in use and how
much backporting is done; the more kernel backporting (along with more
established processes in place), the more painful it'd be to move to the
GitHub version. My gut feeling is that SLES do less backporting compared to
RHEL when it comes to BPF, and that probably placed us closer to the middle
ground.

Thanks,
Shung-Hsi

> -Toke
> 

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

On Tue, Apr 18, 2023 at 07:41:32PM +0200, Michal Suchánek wrote:
> On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
> > On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > >
> > > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> > > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > > > > Hello,
> > > > > > >
> > > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > > > > >> Hi Shung-Hsi,
> > > > > > >>
> > > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > > > > >>> of using the source found in kernel), but realize that it should goes
> > > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > > > > >>> questions:
> > > > > > >>>
> > > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > > > > >>>   Which source is preferred, GitHub or kernel?
> > > > > > >>
> > > > > > >> As you can see from the previous discussions, the suggested approach
> > > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > > > > >> sync.
> > > > > > >>
> > > > > > >> My main argument for the mirror is that it keeps things simpler, and
> > > > > > >> there's no need to deal with the rest of the kernel sources for these
> > > > > > >> packages. Download from the mirrors, build, ship. But then I have
> > > > > > >> limited experience at packaging for distros, and I can understand
> > > > > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > > > > >
> > > > > > > Things get only ever more complex when submodules are involved.
> > > > > >
> > > > > > I understand the generic pain points from your other email. But could
> > > > > > you be more specific for the case of bpftool? It's not like we're
> > > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > > > > pretty much as if we had the right version of the library shipped in the
> > > > > > repository - only, it's one --recurse-submodules away.
> > > > >
> > > > > It's so in every project that uses submodules. Except git does not
> > > > > recurse into submodules by default, you have to fix it up by hand.
> > > > > Forges don't support submodules so you will not get the submodule when
> > > > > downloading the project archive, and won't see it the the project tree.
> > > >
> > > > git submodule update --init --recursive didn't work?
> > >
> > > That's one part of the manual fixup.
> > >
> > > The other part is after each git operation that could possibly cause the
> > > submodules to go out of sync, basically any operation that changes the
> > > checked-out commit.
> > >
> > > Of course, you can make some shell aliases that append whatever submodule
> > > chicanery to whatever git command you might issue, and tell everyone
> > > else to do that, and then it will work in that one shell, and not in any
> > > other shell nor any tool that invokes git directly.
> > 
> > Are we discussing a *standard* Git submodule feature and argue that
> > because it might be cumbersome or unfamiliar to some engineers that
> > projects should avoid using Git submodules?
> 
> As far as I am aware they are unfamiliar to *most* engineers, and for
> good reasons.
> 
> > For one, I don't have any special aliases for dealing with Git
> > submodules and it works fine. If I jump between branches or tags which
> > update Git submodule reference, I do above `git submodule update
> > --init --recursive` explicitly if I see that Git status shows
> > out-of-sync Git submodule state. If I want to update a Git submodule,
> > I update the submodule's Git repo, and then git add it in the repo
> > that uses this submodule. I haven't run into any other issues with
> > this.
> 
> You know, git could just handle submodules automagically. As you say,
> it's not rocket science. For historical reasons it does not.
> 
> With that working with submodules is cumbersome, and it's additional
> thing that can break down that the engineer needs to be constantly aware
> of increasing the mental overhead of working with such projects.
> 
> It may not be much of a problem for people who work with such projects
> daily but not everyone does. Those who don't need to do the mental
> switch whenever submodules are encountered, and are prone to getting
> issues when they forget that they have to go that extra mile for this
> specific project.

For me it's less about having to go through the extra loop. It's that
submodules would require git to be installed, network access, which all adds
extra moving parts compared to a tarball...

> > > > > After previous experience with submodules I did not even try, I just
> > > > > patched the makefile to use system libbpf before attempting anything
> > > > > else.
> > > >
> > > > Quentin mentioned that he's packaging (or will package) libbpf sources
> > > > as part of bpftool release on Github. I've been this for other
> > > > libbpf-using tools as well, and it works pretty well (at least for
> > > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])

and having libbpf included in bpftool release means the complain above no
longer holds. Though I have yet test build the mirror version of libbpf and
bpftool like Michal has done.

> > > > By switching up actual libbpf used to compile with bpftool, you are
> > > > potentially introducing subtle problems that your users will be quite
> > > > unhappy about, if they run into them. Let's work together to make it
> > > > easier for you to package bpftool properly. We can't switch bpftool to
> > > > reliably use system-wide libbpf (either static or shared, doesn't
> > > > matter) because of dependency on internal functionality.
> > > >
> > > >
> > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> > >
> > > So how many copies of libbpf do I need for having a CO-RE toolchain?
> > 
> > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> > etc are tools. The fact they are using statically linked libbpf
> > through Git submodule is irrelevant to end users. You need one libbpf
> > in the system (for those who link dynamically against libbpf), the
> > rest are just tools.
> > 
> > >
> > > Will different tools have different view of the kernel because they each
> > > use different private copy of libbpf with different features?
> > 
> > That's up to tools, not libbpf. You are over pivoting on libbpf here.
> > There is one view of the kernel, it depends on what features the
> > kernel supports. If the tool requires some specific functionality of
> > libbpf, it will update its Git submodule reference to get a version of
> > libbpf that provides that feature. That's the point, an
> > application/tool is in control of what kind of features it gets from
> > libbpf.

Since libbpf has a stable API & ABI, is it theoretically possible for
bpftool, veristat, retsnoop, etc. all share the same version of libbpf?

What I'd like to do it build libbpf and bpftool out of bpftool GitHub
mirror's release tarball (w/ submodule included, which exists now for
snapshot). For the rest of the tool that does not depends on libbpf private
function, have them dynamically link to the libbpf built from bpftool's
source, just like how libelf is dynamically linked.

I'm not saying that those tools should not have libbpf as submodule; as
submodule do look useful. But for packaging I really would like to have the
option of choosing the exact version of libbpf being used.

> > > When there is a bug in libbpf how many places need to be patched to fix
> > > it?
> > 
> > That's up to tools, again. If the bug is affecting them, they should
> > cut a new version of their *tool*, using a patched version of libbpf
> > from Github. If it doesn't affect them, then it doesn't matter *to
> > them*.
> 
> I don't share your optimism about this happening in the real world.
> 
> For one the issue that the github tarballs do not contain the submodule
> and thus cannot be built was raised nearly two months ago, and while a
> test snapshot that does include the submodule is released, a release
> does not exist yet.
> 
> For people to make use of the repository without a release cut they need
> to replicate that submodule support - that is add support for submodules
> in their development tooling. Otherwise you personally cutting a release
> becomes a single point of failure.
> 
> Because there is no API it's not really advisable to just apply patches
> on top of the last release either. Applying patches may cause the main
> project and the submodule to go out of sync, the submodule would not get
> updated by applying a patch to the main project, and the other way
> around.
> 
> Suppose a severe security bug that requires patching libbpf is found.
> Now there is a number of tools that are each tied to one specific
> version of libbpf, and cannot be upgraded to up-to-date fixed version
> because that would break them. I would hope that never happens.
> Nonetheless, libbpf is used to generate code, and if the code is
> generated wrong worst case it can have severe security implications.
> 
> Thanks
> 
> Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Tue, Apr 18, 2023 at 10:41 AM Michal Suchánek <msuchanek@suse.de> wrote:
>
> On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
> > On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > >
> > > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> > > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > > > >
> > > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > > > > Hello,
> > > > > > >
> > > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > > > > >> Hi Shung-Hsi,
> > > > > > >>
> > > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > > > > >>> of using the source found in kernel), but realize that it should goes
> > > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > > > > >>> questions:
> > > > > > >>>
> > > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > > > > >>>   Which source is preferred, GitHub or kernel?
> > > > > > >>
> > > > > > >> As you can see from the previous discussions, the suggested approach
> > > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > > > > >> sync.
> > > > > > >>
> > > > > > >> My main argument for the mirror is that it keeps things simpler, and
> > > > > > >> there's no need to deal with the rest of the kernel sources for these
> > > > > > >> packages. Download from the mirrors, build, ship. But then I have
> > > > > > >> limited experience at packaging for distros, and I can understand
> > > > > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > > > > >
> > > > > > > Things get only ever more complex when submodules are involved.
> > > > > >
> > > > > > I understand the generic pain points from your other email. But could
> > > > > > you be more specific for the case of bpftool? It's not like we're
> > > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > > > > pretty much as if we had the right version of the library shipped in the
> > > > > > repository - only, it's one --recurse-submodules away.
> > > > >
> > > > > It's so in every project that uses submodules. Except git does not
> > > > > recurse into submodules by default, you have to fix it up by hand.
> > > > > Forges don't support submodules so you will not get the submodule when
> > > > > downloading the project archive, and won't see it the the project tree.
> > > >
> > > > git submodule update --init --recursive didn't work?
> > >
> > > That's one part of the manual fixup.
> > >
> > > The other part is after each git operation that could possibly cause the
> > > submodules to go out of sync, basically any operation that changes the
> > > checked-out commit.
> > >
> > > Of course, you can make some shell aliases that append whatever submodule
> > > chicanery to whatever git command you might issue, and tell everyone
> > > else to do that, and then it will work in that one shell, and not in any
> > > other shell nor any tool that invokes git directly.
> >
> > Are we discussing a *standard* Git submodule feature and argue that
> > because it might be cumbersome or unfamiliar to some engineers that
> > projects should avoid using Git submodules?
>
> As far as I am aware they are unfamiliar to *most* engineers, and for
> good reasons.
>
> > For one, I don't have any special aliases for dealing with Git
> > submodules and it works fine. If I jump between branches or tags which
> > update Git submodule reference, I do above `git submodule update
> > --init --recursive` explicitly if I see that Git status shows
> > out-of-sync Git submodule state. If I want to update a Git submodule,
> > I update the submodule's Git repo, and then git add it in the repo
> > that uses this submodule. I haven't run into any other issues with
> > this.
>
> You know, git could just handle submodules automagically. As you say,
> it's not rocket science. For historical reasons it does not.
>
> With that working with submodules is cumbersome, and it's additional
> thing that can break down that the engineer needs to be constantly aware
> of increasing the mental overhead of working with such projects.
>
> It may not be much of a problem for people who work with such projects
> daily but not everyone does. Those who don't need to do the mental
> switch whenever submodules are encountered, and are prone to getting
> issues when they forget that they have to go that extra mile for this
> specific project.

I agree with you that Git submodules are not the most straightforward
feature usability-wise. Where I disagree is that this is enough reason
to not use Git submodules. They do provide a lot of value.

>
> > > > > After previous experience with submodules I did not even try, I just
> > > > > patched the makefile to use system libbpf before attempting anything
> > > > > else.
> > > >
> > > > Quentin mentioned that he's packaging (or will package) libbpf sources
> > > > as part of bpftool release on Github. I've been this for other
> > > > libbpf-using tools as well, and it works pretty well (at least for
> > > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
> > > >
> > > > By switching up actual libbpf used to compile with bpftool, you are
> > > > potentially introducing subtle problems that your users will be quite
> > > > unhappy about, if they run into them. Let's work together to make it
> > > > easier for you to package bpftool properly. We can't switch bpftool to
> > > > reliably use system-wide libbpf (either static or shared, doesn't
> > > > matter) because of dependency on internal functionality.
> > > >
> > > >
> > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> > >
> > > So how many copies of libbpf do I need for having a CO-RE toolchain?
> >
> > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> > etc are tools. The fact they are using statically linked libbpf
> > through Git submodule is irrelevant to end users. You need one libbpf
> > in the system (for those who link dynamically against libbpf), the
> > rest are just tools.
> >
> > >
> > > Will different tools have different view of the kernel because they each
> > > use different private copy of libbpf with different features?
> >
> > That's up to tools, not libbpf. You are over pivoting on libbpf here.
> > There is one view of the kernel, it depends on what features the
> > kernel supports. If the tool requires some specific functionality of
> > libbpf, it will update its Git submodule reference to get a version of
> > libbpf that provides that feature. That's the point, an
> > application/tool is in control of what kind of features it gets from
> > libbpf.
> >
> > >
> > > When there is a bug in libbpf how many places need to be patched to fix
> > > it?
> >
> > That's up to tools, again. If the bug is affecting them, they should
> > cut a new version of their *tool*, using a patched version of libbpf
> > from Github. If it doesn't affect them, then it doesn't matter *to
> > them*.
>
> I don't share your optimism about this happening in the real world.
>
> For one the issue that the github tarballs do not contain the submodule
> and thus cannot be built was raised nearly two months ago, and while a
> test snapshot that does include the submodule is released, a release
> does not exist yet.

Because no one raised this issue earlier (for bpftool). Fedora
packagers raised it for retsnoop, so retsnoop has it. That's how
development works, one can't anticipate all the possible issues, they
need to be reported and worked on.

>
> For people to make use of the repository without a release cut they need
> to replicate that submodule support - that is add support for submodules
> in their development tooling. Otherwise you personally cutting a release
> becomes a single point of failure.

I hope distros won't be packaging an unreleased version, though?

>
> Because there is no API it's not really advisable to just apply patches
> on top of the last release either. Applying patches may cause the main
> project and the submodule to go out of sync, the submodule would not get
> updated by applying a patch to the main project, and the other way
> around.

I'm not sure where we are going with this overall discussion at this point, tbh.

>
> Suppose a severe security bug that requires patching libbpf is found.
> Now there is a number of tools that are each tied to one specific
> version of libbpf, and cannot be upgraded to up-to-date fixed version
> because that would break them. I would hope that never happens.
> Nonetheless, libbpf is used to generate code, and if the code is
> generated wrong worst case it can have severe security implications.
>

Yes, I hear this argument from packagers all the time. Yet, somehow
it's been fine for the last few years. Please realize that there are
many reasons why we do want submodules and static linking of libbpf,
and accept that projects do decide to stick to submodules. It might
not be perfect, but the benefits of such an approach outweigh the
hypothetical issues you brought up.

This has all been discussed multiple times, as I said, I don't think
any of us added anything new to previous discussions. So if you are
interested, please work with us to improve your life as a packager,
but also accept the way this project is set up on Github.

> Thanks
>
> Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Wed, Apr 19, 2023 at 3:55 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>
> Thanks for sharing! I though I'd expands on what you said to draw a clearer
> picture of the challenges.
>
> On Thu, Apr 13, 2023 at 01:00:20PM +0200, Toke Høiland-Jørgensen wrote:
> > Shung-Hsi Yu <shung-hsi.yu@suse.com> writes:
> >
> > > A side note: if we want all userspace visible libbpf to have a coherent
> > > version, perf needs to use the shared libbpf library as well (either
> > > autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to
> > > backport patches to kernel source to keep up with userspace package (libbpf)
> > > changes could be a pain.
>
> Here some more context for completeness. Kernel source changes are published
> at a much slower pace than userspace. When an application in the kernel
> source (e.g. perf) depends on the userspace library, it's kind of like
> trying to catchup a car on a bike, which is doable, as evident by the
> plethora of userspace libraries perf already depends on. While I don't
> having experience maintaining perf, judging by tools/perf/Makefile.config
> that does not seem like an easy feat.
>
> For perf to use libbpf in kernel would mean that it's just depending on
> something that moves at the same pace.
>
> That said, maybe perf won't need additional backport to keep up with libbpf
> as long as we keep it within that same major version (and disable
> deprecation warning)? @Andrii
>
> Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.

I'm not sure what the proposal is, but I'll delegate to Arnaldo.

>
> > So basically, this here is the reason we're building libbpf from the
> > kernel tree for the RHEL package: If we use the github version we'd need
> > to juggle two different versions of libbpf, one for the in-kernel-tree
> > users (perf as you mention, but also the BPF selftests), and one for the
> > userspace packages. Also, having libbpf in the kernel tree means we can
> > just backport patches to it along with the BPF-related kernel patches
> > (we do quite extensive BPF backports for each RHEL version).
>
> > Finally, building from the kernel tree means we can use the existing
> > kernel-related procedures for any out of order hotfixes (since AFAIK none
> > of the github repositories have any concept of stable branches that
> > receive fixes).
>
> +1
>
> Got something similar in place as well and being able to stick with existing
> procedure is appealing.
>
> > YMMV of course, but figured I'd share our reasoning. To be clear,
> > building from the kernel tree is not without its own pain points (mostly
> > related to how the build scripts are structured for our kernel builds).
> > We've discussed moving to the github version of libbpf multiple times,
> > but every time we've concluded that it would be more, not less, painful
> > than having the kernel tree be the single source of truth.
>
> We package maintainer are certainly quite hard to please :)
>
> Just having an individual package easy to work with is not enough, we want
> it to be easier for most packages before jumping on the bandwagon, which is
> why this email ended up talking about perf despite it started as a
> discussion on packaging libbpf and bpftool.
>
> I suppose the mileage depends on the build system & scripts in use and how
> much backporting is done; the more kernel backporting (along with more
> established processes in place), the more painful it'd be to move to the
> GitHub version. My gut feeling is that SLES do less backporting compared to
> RHEL when it comes to BPF, and that probably placed us closer to the middle
> ground.

Even though libbpf source is developed in kernel repo, it's not tied
to specific kernel. So any kernel backports have no relevance to
libbpf itself. It's yet another reason to switch to Github mirror.
Github is merging libbpf-related fixes from both bpf and bpf-next
trees during sync, and is meant to always be the latest and best
version with all fixes included.

I won't claim anything for perf, maybe Arnaldo can clarify, but I
suspect that perf is also meant to be relatively independent from
specific kernel and work on wide variety of kernels.

As for stable branches. For libbpf, we don't have it because we didn't
need it yet. We did have bug fix patch releases that seem to be
working out fine, though.

>
> Thanks,
> Shung-Hsi
>
> > -Toke
> >

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Wed, Apr 19, 2023 at 7:14 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>
> On Tue, Apr 18, 2023 at 07:41:32PM +0200, Michal Suchánek wrote:
> > On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
> > > On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > > >
> > > > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> > > > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> > > > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> > > > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> > > > > > > >> Hi Shung-Hsi,
> > > > > > > >>
> > > > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> > > > > > > >>>
> > > > > > > >>> Hi,
> > > > > > > >>>
> > > > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> > > > > > > >>> of using the source found in kernel), but realize that it should goes
> > > > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> > > > > > > >>> questions:
> > > > > > > >>>
> > > > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> > > > > > > >>>   Which source is preferred, GitHub or kernel?
> > > > > > > >>
> > > > > > > >> As you can see from the previous discussions, the suggested approach
> > > > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> > > > > > > >> sync.
> > > > > > > >>
> > > > > > > >> My main argument for the mirror is that it keeps things simpler, and
> > > > > > > >> there's no need to deal with the rest of the kernel sources for these
> > > > > > > >> packages. Download from the mirrors, build, ship. But then I have
> > > > > > > >> limited experience at packaging for distros, and I can understand
> > > > > > > >> Toke's point of view, too. So ultimately, the call is yours.
> > > > > > > >
> > > > > > > > Things get only ever more complex when submodules are involved.
> > > > > > >
> > > > > > > I understand the generic pain points from your other email. But could
> > > > > > > you be more specific for the case of bpftool? It's not like we're
> > > > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> > > > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> > > > > > > pretty much as if we had the right version of the library shipped in the
> > > > > > > repository - only, it's one --recurse-submodules away.
> > > > > >
> > > > > > It's so in every project that uses submodules. Except git does not
> > > > > > recurse into submodules by default, you have to fix it up by hand.
> > > > > > Forges don't support submodules so you will not get the submodule when
> > > > > > downloading the project archive, and won't see it the the project tree.
> > > > >
> > > > > git submodule update --init --recursive didn't work?
> > > >
> > > > That's one part of the manual fixup.
> > > >
> > > > The other part is after each git operation that could possibly cause the
> > > > submodules to go out of sync, basically any operation that changes the
> > > > checked-out commit.
> > > >
> > > > Of course, you can make some shell aliases that append whatever submodule
> > > > chicanery to whatever git command you might issue, and tell everyone
> > > > else to do that, and then it will work in that one shell, and not in any
> > > > other shell nor any tool that invokes git directly.
> > >
> > > Are we discussing a *standard* Git submodule feature and argue that
> > > because it might be cumbersome or unfamiliar to some engineers that
> > > projects should avoid using Git submodules?
> >
> > As far as I am aware they are unfamiliar to *most* engineers, and for
> > good reasons.
> >
> > > For one, I don't have any special aliases for dealing with Git
> > > submodules and it works fine. If I jump between branches or tags which
> > > update Git submodule reference, I do above `git submodule update
> > > --init --recursive` explicitly if I see that Git status shows
> > > out-of-sync Git submodule state. If I want to update a Git submodule,
> > > I update the submodule's Git repo, and then git add it in the repo
> > > that uses this submodule. I haven't run into any other issues with
> > > this.
> >
> > You know, git could just handle submodules automagically. As you say,
> > it's not rocket science. For historical reasons it does not.
> >
> > With that working with submodules is cumbersome, and it's additional
> > thing that can break down that the engineer needs to be constantly aware
> > of increasing the mental overhead of working with such projects.
> >
> > It may not be much of a problem for people who work with such projects
> > daily but not everyone does. Those who don't need to do the mental
> > switch whenever submodules are encountered, and are prone to getting
> > issues when they forget that they have to go that extra mile for this
> > specific project.
>
> For me it's less about having to go through the extra loop. It's that
> submodules would require git to be installed, network access, which all adds
> extra moving parts compared to a tarball...
>
> > > > > > After previous experience with submodules I did not even try, I just
> > > > > > patched the makefile to use system libbpf before attempting anything
> > > > > > else.
> > > > >
> > > > > Quentin mentioned that he's packaging (or will package) libbpf sources
> > > > > as part of bpftool release on Github. I've been this for other
> > > > > libbpf-using tools as well, and it works pretty well (at least for
> > > > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
>
> and having libbpf included in bpftool release means the complain above no
> longer holds. Though I have yet test build the mirror version of libbpf and
> bpftool like Michal has done.

Great. This seems to work well for other tools that use libbpf through
submodule (anakryiko/retsnoop and libbpf/veristat on Github)

>
> > > > > By switching up actual libbpf used to compile with bpftool, you are
> > > > > potentially introducing subtle problems that your users will be quite
> > > > > unhappy about, if they run into them. Let's work together to make it
> > > > > easier for you to package bpftool properly. We can't switch bpftool to
> > > > > reliably use system-wide libbpf (either static or shared, doesn't
> > > > > matter) because of dependency on internal functionality.
> > > > >
> > > > >
> > > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> > > >
> > > > So how many copies of libbpf do I need for having a CO-RE toolchain?
> > >
> > > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> > > etc are tools. The fact they are using statically linked libbpf
> > > through Git submodule is irrelevant to end users. You need one libbpf
> > > in the system (for those who link dynamically against libbpf), the
> > > rest are just tools.
> > >
> > > >
> > > > Will different tools have different view of the kernel because they each
> > > > use different private copy of libbpf with different features?
> > >
> > > That's up to tools, not libbpf. You are over pivoting on libbpf here.
> > > There is one view of the kernel, it depends on what features the
> > > kernel supports. If the tool requires some specific functionality of
> > > libbpf, it will update its Git submodule reference to get a version of
> > > libbpf that provides that feature. That's the point, an
> > > application/tool is in control of what kind of features it gets from
> > > libbpf.
>
> Since libbpf has a stable API & ABI, is it theoretically possible for
> bpftool, veristat, retsnoop, etc. all share the same version of libbpf?

No, because libbpf is not just a set of APIs. Newer libbpf versions
support more BPF-side features, more kernel features, etc, etc. Libbpf
is not a typical user-space library, it is a BPF loader, and even if
user-visible API doesn't change, libbpf's support for various BPF-side
features is extended. Which is important for tools like bpftool,
retsnoop, veristat which rely on loading and working with BPF object
files.

>
> What I'd like to do it build libbpf and bpftool out of bpftool GitHub
> mirror's release tarball (w/ submodule included, which exists now for
> snapshot). For the rest of the tool that does not depends on libbpf private
> function, have them dynamically link to the libbpf built from bpftool's
> source, just like how libelf is dynamically linked.

Please don't do it, let applications control which libbpf versions
they are using. It's not just about user space APIs, I can't emphasize
this enough. Don't think you know better than developers of respective
applications, don't try to dictate how those applications should be
organized and developed.

One good example is iproute2, which chose to link (or not) with libbpf
dynamically. Now users periodically report various issues where their
BPF object files are not loaded, and it often comes down to unexpected
version of libbpf (or lack of libbpf support altogether) which which
iproute2 was built/deployed. This is just putting a burden on iproute2
users, and accidentally libbpf maintainers, for no good reason.

>
> I'm not saying that those tools should not have libbpf as submodule; as
> submodule do look useful. But for packaging I really would like to have the
> option of choosing the exact version of libbpf being used.

The exact version of libbpf used by bpftool, retsnoop, veristat, etc
*is not relevant* to you as a packager. If you want happy users, use
*exact* version of libbpf from submodule to build them, with which
application was developed, tested, and advertised supported BPF
features. There is no reuse to be done here, they all can be on
different (and sometimes not yet released) libbpf version. For good
reasons, which are outside of your control as a packager.

>
> > > > When there is a bug in libbpf how many places need to be patched to fix
> > > > it?
> > >
> > > That's up to tools, again. If the bug is affecting them, they should
> > > cut a new version of their *tool*, using a patched version of libbpf
> > > from Github. If it doesn't affect them, then it doesn't matter *to
> > > them*.
> >
> > I don't share your optimism about this happening in the real world.
> >
> > For one the issue that the github tarballs do not contain the submodule
> > and thus cannot be built was raised nearly two months ago, and while a
> > test snapshot that does include the submodule is released, a release
> > does not exist yet.
> >
> > For people to make use of the repository without a release cut they need
> > to replicate that submodule support - that is add support for submodules
> > in their development tooling. Otherwise you personally cutting a release
> > becomes a single point of failure.
> >
> > Because there is no API it's not really advisable to just apply patches
> > on top of the last release either. Applying patches may cause the main
> > project and the submodule to go out of sync, the submodule would not get
> > updated by applying a patch to the main project, and the other way
> > around.
> >
> > Suppose a severe security bug that requires patching libbpf is found.
> > Now there is a number of tools that are each tied to one specific
> > version of libbpf, and cannot be upgraded to up-to-date fixed version
> > because that would break them. I would hope that never happens.
> > Nonetheless, libbpf is used to generate code, and if the code is
> > generated wrong worst case it can have severe security implications.
> >
> > Thanks
> >
> > Michal

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Toke Høiland-Jørgensen]

Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:

> On Wed, Apr 19, 2023 at 7:14 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>>
>> On Tue, Apr 18, 2023 at 07:41:32PM +0200, Michal Suchánek wrote:
>> > On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
>> > > On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
>> > > >
>> > > > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
>> > > > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
>> > > > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
>> > > > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
>> > > > > > > > Hello,
>> > > > > > > >
>> > > > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
>> > > > > > > >> Hi Shung-Hsi,
>> > > > > > > >>
>> > > > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>> > > > > > > >>>
>> > > > > > > >>> Hi,
>> > > > > > > >>>
>> > > > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
>> > > > > > > >>> of using the source found in kernel), but realize that it should goes
>> > > > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
>> > > > > > > >>> questions:
>> > > > > > > >>>
>> > > > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
>> > > > > > > >>>   Which source is preferred, GitHub or kernel?
>> > > > > > > >>
>> > > > > > > >> As you can see from the previous discussions, the suggested approach
>> > > > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
>> > > > > > > >> sync.
>> > > > > > > >>
>> > > > > > > >> My main argument for the mirror is that it keeps things simpler, and
>> > > > > > > >> there's no need to deal with the rest of the kernel sources for these
>> > > > > > > >> packages. Download from the mirrors, build, ship. But then I have
>> > > > > > > >> limited experience at packaging for distros, and I can understand
>> > > > > > > >> Toke's point of view, too. So ultimately, the call is yours.
>> > > > > > > >
>> > > > > > > > Things get only ever more complex when submodules are involved.
>> > > > > > >
>> > > > > > > I understand the generic pain points from your other email. But could
>> > > > > > > you be more specific for the case of bpftool? It's not like we're
>> > > > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
>> > > > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
>> > > > > > > pretty much as if we had the right version of the library shipped in the
>> > > > > > > repository - only, it's one --recurse-submodules away.
>> > > > > >
>> > > > > > It's so in every project that uses submodules. Except git does not
>> > > > > > recurse into submodules by default, you have to fix it up by hand.
>> > > > > > Forges don't support submodules so you will not get the submodule when
>> > > > > > downloading the project archive, and won't see it the the project tree.
>> > > > >
>> > > > > git submodule update --init --recursive didn't work?
>> > > >
>> > > > That's one part of the manual fixup.
>> > > >
>> > > > The other part is after each git operation that could possibly cause the
>> > > > submodules to go out of sync, basically any operation that changes the
>> > > > checked-out commit.
>> > > >
>> > > > Of course, you can make some shell aliases that append whatever submodule
>> > > > chicanery to whatever git command you might issue, and tell everyone
>> > > > else to do that, and then it will work in that one shell, and not in any
>> > > > other shell nor any tool that invokes git directly.
>> > >
>> > > Are we discussing a *standard* Git submodule feature and argue that
>> > > because it might be cumbersome or unfamiliar to some engineers that
>> > > projects should avoid using Git submodules?
>> >
>> > As far as I am aware they are unfamiliar to *most* engineers, and for
>> > good reasons.
>> >
>> > > For one, I don't have any special aliases for dealing with Git
>> > > submodules and it works fine. If I jump between branches or tags which
>> > > update Git submodule reference, I do above `git submodule update
>> > > --init --recursive` explicitly if I see that Git status shows
>> > > out-of-sync Git submodule state. If I want to update a Git submodule,
>> > > I update the submodule's Git repo, and then git add it in the repo
>> > > that uses this submodule. I haven't run into any other issues with
>> > > this.
>> >
>> > You know, git could just handle submodules automagically. As you say,
>> > it's not rocket science. For historical reasons it does not.
>> >
>> > With that working with submodules is cumbersome, and it's additional
>> > thing that can break down that the engineer needs to be constantly aware
>> > of increasing the mental overhead of working with such projects.
>> >
>> > It may not be much of a problem for people who work with such projects
>> > daily but not everyone does. Those who don't need to do the mental
>> > switch whenever submodules are encountered, and are prone to getting
>> > issues when they forget that they have to go that extra mile for this
>> > specific project.
>>
>> For me it's less about having to go through the extra loop. It's that
>> submodules would require git to be installed, network access, which all adds
>> extra moving parts compared to a tarball...
>>
>> > > > > > After previous experience with submodules I did not even try, I just
>> > > > > > patched the makefile to use system libbpf before attempting anything
>> > > > > > else.
>> > > > >
>> > > > > Quentin mentioned that he's packaging (or will package) libbpf sources
>> > > > > as part of bpftool release on Github. I've been this for other
>> > > > > libbpf-using tools as well, and it works pretty well (at least for
>> > > > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
>>
>> and having libbpf included in bpftool release means the complain above no
>> longer holds. Though I have yet test build the mirror version of libbpf and
>> bpftool like Michal has done.
>
> Great. This seems to work well for other tools that use libbpf through
> submodule (anakryiko/retsnoop and libbpf/veristat on Github)
>
>>
>> > > > > By switching up actual libbpf used to compile with bpftool, you are
>> > > > > potentially introducing subtle problems that your users will be quite
>> > > > > unhappy about, if they run into them. Let's work together to make it
>> > > > > easier for you to package bpftool properly. We can't switch bpftool to
>> > > > > reliably use system-wide libbpf (either static or shared, doesn't
>> > > > > matter) because of dependency on internal functionality.
>> > > > >
>> > > > >
>> > > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
>> > > >
>> > > > So how many copies of libbpf do I need for having a CO-RE toolchain?
>> > >
>> > > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
>> > > etc are tools. The fact they are using statically linked libbpf
>> > > through Git submodule is irrelevant to end users. You need one libbpf
>> > > in the system (for those who link dynamically against libbpf), the
>> > > rest are just tools.
>> > >
>> > > >
>> > > > Will different tools have different view of the kernel because they each
>> > > > use different private copy of libbpf with different features?
>> > >
>> > > That's up to tools, not libbpf. You are over pivoting on libbpf here.
>> > > There is one view of the kernel, it depends on what features the
>> > > kernel supports. If the tool requires some specific functionality of
>> > > libbpf, it will update its Git submodule reference to get a version of
>> > > libbpf that provides that feature. That's the point, an
>> > > application/tool is in control of what kind of features it gets from
>> > > libbpf.
>>
>> Since libbpf has a stable API & ABI, is it theoretically possible for
>> bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
>
> No, because libbpf is not just a set of APIs. Newer libbpf versions
> support more BPF-side features, more kernel features, etc, etc. Libbpf
> is not a typical user-space library, it is a BPF loader, and even if
> user-visible API doesn't change, libbpf's support for various BPF-side
> features is extended. Which is important for tools like bpftool,
> retsnoop, veristat which rely on loading and working with BPF object
> files.

The converse of this is also true: if your system is upgraded to a new
kernel version with new BPF features, the libbpf version should follow
it, and all applications linked against it will automatically take
advantage of any bugfixes regardless without having to wait for each
application to be updated.

Libbpf is really no different from any other library here, and I really
don't get why you keep insisting it's "special"...

>> What I'd like to do it build libbpf and bpftool out of bpftool GitHub
>> mirror's release tarball (w/ submodule included, which exists now for
>> snapshot). For the rest of the tool that does not depends on libbpf private
>> function, have them dynamically link to the libbpf built from bpftool's
>> source, just like how libelf is dynamically linked.
>
> Please don't do it, let applications control which libbpf versions
> they are using. It's not just about user space APIs, I can't emphasize
> this enough. Don't think you know better than developers of respective
> applications, don't try to dictate how those applications should be
> organized and developed.

A well-behaved application will detect which features are available in
the system version of the libraries they use, and if something is
missing that it needs, either work around it or refuse to build. We do
this with libbpf in xdp-tools and the only issues we've had with it has
been the changing API in pre-1.0 libbpf...

> One good example is iproute2, which chose to link (or not) with libbpf
> dynamically. Now users periodically report various issues where their
> BPF object files are not loaded, and it often comes down to unexpected
> version of libbpf (or lack of libbpf support altogether) which which
> iproute2 was built/deployed. This is just putting a burden on iproute2
> users, and accidentally libbpf maintainers, for no good reason.

How would this have been any different if iproute2 was statically linked
against libbpf?

>> I'm not saying that those tools should not have libbpf as submodule; as
>> submodule do look useful. But for packaging I really would like to have the
>> option of choosing the exact version of libbpf being used.
>
> The exact version of libbpf used by bpftool, retsnoop, veristat, etc
> *is not relevant* to you as a packager. If you want happy users, use
> *exact* version of libbpf from submodule to build them, with which
> application was developed, tested, and advertised supported BPF
> features. There is no reuse to be done here, they all can be on
> different (and sometimes not yet released) libbpf version. For good
> reasons, which are outside of your control as a packager.

This is... just not how distributions work. As a user I trust my
distribution to provide me with a coherent system where critical system
libraries are maintained and receive timely updates. And I absolutely
trust the distribution more to do this over application developers who
just vendor in some version as a submodule and leave it there until they
need a new feature...

-Toke

Re: Packaging bpftool and libbpf: GitHub or kernel?

[tonyj]

On Wed, Apr 19, 2023 at 12:42:39PM -0700, Andrii Nakryiko wrote:

> I won't claim anything for perf, maybe Arnaldo can clarify, but I
> suspect that perf is also meant to be relatively independent from
> specific kernel and work on wide variety of kernels.

Supposedly it is though I've never been able to convince myself to
decouple (and ship the latest perf) for SLE. One reason is documented 
userspace functionality that has no backing kernel support which for 
an Enterprise distro opens us up to bug reports.

For a while Vince was maintaining a page documenting breakage in the 
perf_event ABI but last I checked it ended at V4.0.  Not sure if he 
just stopped updating, or that's where breakage ended :)

I know how Fedora maintains perf but I keep meaning to look at how
RHEL maintains it.  As Michal said our code base is V5.14 for SLE15*
and due to the frequent code-refactoring in perf, backporting perf
userspace fixes for SLE is a real chore unless we agressively maintain
forward porting ("forklifting").

Tony

-- 
Tony Jones
SUSE Kernel Performance Team

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Wed, Apr 19, 2023 at 2:23 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:
>
> > On Wed, Apr 19, 2023 at 7:14 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >>
> >> On Tue, Apr 18, 2023 at 07:41:32PM +0200, Michal Suchánek wrote:
> >> > On Tue, Apr 18, 2023 at 09:38:20AM -0700, Andrii Nakryiko wrote:
> >> > > On Tue, Apr 18, 2023 at 4:24 AM Michal Suchánek <msuchanek@suse.de> wrote:
> >> > > >
> >> > > > On Mon, Apr 17, 2023 at 05:20:03PM -0700, Andrii Nakryiko wrote:
> >> > > > > On Fri, Apr 14, 2023 at 9:15 AM Michal Suchánek <msuchanek@suse.de> wrote:
> >> > > > > > On Fri, Apr 14, 2023 at 01:30:02PM +0100, Quentin Monnet wrote:
> >> > > > > > > 2023-04-14 11:50 UTC+0200 ~ Michal Suchánek <msuchanek@suse.de>
> >> > > > > > > > Hello,
> >> > > > > > > >
> >> > > > > > > > On Fri, Apr 14, 2023 at 01:35:20AM +0100, Quentin Monnet wrote:
> >> > > > > > > >> Hi Shung-Hsi,
> >> > > > > > > >>
> >> > > > > > > >> On Thu, 13 Apr 2023 at 10:23, Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >> > > > > > > >>>
> >> > > > > > > >>> Hi,
> >> > > > > > > >>>
> >> > > > > > > >>> I'm considering switch to bpftool's mirror on GitHub for packaging (instead
> >> > > > > > > >>> of using the source found in kernel), but realize that it should goes
> >> > > > > > > >>> hand-in-hand with how libbpf is packaged, which eventually leads these
> >> > > > > > > >>> questions:
> >> > > > > > > >>>
> >> > > > > > > >>>   What is the suggested approach for packaging bpftool and libbpf?
> >> > > > > > > >>>   Which source is preferred, GitHub or kernel?
> >> > > > > > > >>
> >> > > > > > > >> As you can see from the previous discussions, the suggested approach
> >> > > > > > > >> would be to package from the GitHub mirror, with libbpf and bpftool in
> >> > > > > > > >> sync.
> >> > > > > > > >>
> >> > > > > > > >> My main argument for the mirror is that it keeps things simpler, and
> >> > > > > > > >> there's no need to deal with the rest of the kernel sources for these
> >> > > > > > > >> packages. Download from the mirrors, build, ship. But then I have
> >> > > > > > > >> limited experience at packaging for distros, and I can understand
> >> > > > > > > >> Toke's point of view, too. So ultimately, the call is yours.
> >> > > > > > > >
> >> > > > > > > > Things get only ever more complex when submodules are involved.
> >> > > > > > >
> >> > > > > > > I understand the generic pain points from your other email. But could
> >> > > > > > > you be more specific for the case of bpftool? It's not like we're
> >> > > > > > > shipping all lib dependencies as submodules. Sync-ups are specifically
> >> > > > > > > aligned to the same commit used to sync the libbpf mirror, so that it's
> >> > > > > > > pretty much as if we had the right version of the library shipped in the
> >> > > > > > > repository - only, it's one --recurse-submodules away.
> >> > > > > >
> >> > > > > > It's so in every project that uses submodules. Except git does not
> >> > > > > > recurse into submodules by default, you have to fix it up by hand.
> >> > > > > > Forges don't support submodules so you will not get the submodule when
> >> > > > > > downloading the project archive, and won't see it the the project tree.
> >> > > > >
> >> > > > > git submodule update --init --recursive didn't work?
> >> > > >
> >> > > > That's one part of the manual fixup.
> >> > > >
> >> > > > The other part is after each git operation that could possibly cause the
> >> > > > submodules to go out of sync, basically any operation that changes the
> >> > > > checked-out commit.
> >> > > >
> >> > > > Of course, you can make some shell aliases that append whatever submodule
> >> > > > chicanery to whatever git command you might issue, and tell everyone
> >> > > > else to do that, and then it will work in that one shell, and not in any
> >> > > > other shell nor any tool that invokes git directly.
> >> > >
> >> > > Are we discussing a *standard* Git submodule feature and argue that
> >> > > because it might be cumbersome or unfamiliar to some engineers that
> >> > > projects should avoid using Git submodules?
> >> >
> >> > As far as I am aware they are unfamiliar to *most* engineers, and for
> >> > good reasons.
> >> >
> >> > > For one, I don't have any special aliases for dealing with Git
> >> > > submodules and it works fine. If I jump between branches or tags which
> >> > > update Git submodule reference, I do above `git submodule update
> >> > > --init --recursive` explicitly if I see that Git status shows
> >> > > out-of-sync Git submodule state. If I want to update a Git submodule,
> >> > > I update the submodule's Git repo, and then git add it in the repo
> >> > > that uses this submodule. I haven't run into any other issues with
> >> > > this.
> >> >
> >> > You know, git could just handle submodules automagically. As you say,
> >> > it's not rocket science. For historical reasons it does not.
> >> >
> >> > With that working with submodules is cumbersome, and it's additional
> >> > thing that can break down that the engineer needs to be constantly aware
> >> > of increasing the mental overhead of working with such projects.
> >> >
> >> > It may not be much of a problem for people who work with such projects
> >> > daily but not everyone does. Those who don't need to do the mental
> >> > switch whenever submodules are encountered, and are prone to getting
> >> > issues when they forget that they have to go that extra mile for this
> >> > specific project.
> >>
> >> For me it's less about having to go through the extra loop. It's that
> >> submodules would require git to be installed, network access, which all adds
> >> extra moving parts compared to a tarball...
> >>
> >> > > > > > After previous experience with submodules I did not even try, I just
> >> > > > > > patched the makefile to use system libbpf before attempting anything
> >> > > > > > else.
> >> > > > >
> >> > > > > Quentin mentioned that he's packaging (or will package) libbpf sources
> >> > > > > as part of bpftool release on Github. I've been this for other
> >> > > > > libbpf-using tools as well, and it works pretty well (at least for
> >> > > > > Fedora and ArchLinux). E.g., srcs-full-* archives for veristat ([0])
> >>
> >> and having libbpf included in bpftool release means the complain above no
> >> longer holds. Though I have yet test build the mirror version of libbpf and
> >> bpftool like Michal has done.
> >
> > Great. This seems to work well for other tools that use libbpf through
> > submodule (anakryiko/retsnoop and libbpf/veristat on Github)
> >
> >>
> >> > > > > By switching up actual libbpf used to compile with bpftool, you are
> >> > > > > potentially introducing subtle problems that your users will be quite
> >> > > > > unhappy about, if they run into them. Let's work together to make it
> >> > > > > easier for you to package bpftool properly. We can't switch bpftool to
> >> > > > > reliably use system-wide libbpf (either static or shared, doesn't
> >> > > > > matter) because of dependency on internal functionality.
> >> > > > >
> >> > > > >
> >> > > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> >> > > >
> >> > > > So how many copies of libbpf do I need for having a CO-RE toolchain?
> >> > >
> >> > > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> >> > > etc are tools. The fact they are using statically linked libbpf
> >> > > through Git submodule is irrelevant to end users. You need one libbpf
> >> > > in the system (for those who link dynamically against libbpf), the
> >> > > rest are just tools.
> >> > >
> >> > > >
> >> > > > Will different tools have different view of the kernel because they each
> >> > > > use different private copy of libbpf with different features?
> >> > >
> >> > > That's up to tools, not libbpf. You are over pivoting on libbpf here.
> >> > > There is one view of the kernel, it depends on what features the
> >> > > kernel supports. If the tool requires some specific functionality of
> >> > > libbpf, it will update its Git submodule reference to get a version of
> >> > > libbpf that provides that feature. That's the point, an
> >> > > application/tool is in control of what kind of features it gets from
> >> > > libbpf.
> >>
> >> Since libbpf has a stable API & ABI, is it theoretically possible for
> >> bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
> >
> > No, because libbpf is not just a set of APIs. Newer libbpf versions
> > support more BPF-side features, more kernel features, etc, etc. Libbpf
> > is not a typical user-space library, it is a BPF loader, and even if
> > user-visible API doesn't change, libbpf's support for various BPF-side
> > features is extended. Which is important for tools like bpftool,
> > retsnoop, veristat which rely on loading and working with BPF object
> > files.
>
> The converse of this is also true: if your system is upgraded to a new
> kernel version with new BPF features, the libbpf version should follow
> it, and all applications linked against it will automatically take
> advantage of any bugfixes regardless without having to wait for each
> application to be updated.

No, if my application was not developed to take advantage of a new
kernel feature, newer libbpf will do nothing for me. If my application
wants to support that feature, I'll update my application and
correspondingly update libbpf embedded in it. If my application is
affected by some bug fix, I'll update libbpf even faster than distros
will get to it.

I've heard all such arguments over the last few years. They are not
convincing and my own practical experience shows irrelevance of the
above argument.

>
> Libbpf is really no different from any other library here, and I really
> don't get why you keep insisting it's "special"...

It's special in the sense that it provides two sets of APIs -- for
user-space (typical libraries) and BPF object files. Besides that, for
BPF-side it's not even a set of APIs (headers, helpers, etc), it also
provides some set of functionality that can improve or be extended
over time. E.g., libbpf used to not support non-inlined BPF
subprograms, and then it started supporting them. In terms of API/ABI
-- nothing changed. Yet the change is very important.

Now, I build a tool that is using libbpf and some BPF functionality,
e.g., retsnoop. Libbpf just got SEC("ksyscall") support. Retsnoop
wants to take advantage of it. I just go and use SEC("ksyscall")
programs in .bpf.c files that are embedded inside retsnoop. I don't
have to *and don't want to* do feature detection of whether a
particular libbpf version that happens to be installed/packaged on the
system supports this version. I *know* it does, because I control it,
through a submodule. That's what I care about.

Whether some distro insists on libbpf being shared across any
libbpf-using application or not is none of my concern. Libbpf is an
implementation detail of my application (retsnoop), it's not for the
packager to decide how I develop and structure my tool.

>
> >> What I'd like to do it build libbpf and bpftool out of bpftool GitHub
> >> mirror's release tarball (w/ submodule included, which exists now for
> >> snapshot). For the rest of the tool that does not depends on libbpf private
> >> function, have them dynamically link to the libbpf built from bpftool's
> >> source, just like how libelf is dynamically linked.
> >
> > Please don't do it, let applications control which libbpf versions
> > they are using. It's not just about user space APIs, I can't emphasize
> > this enough. Don't think you know better than developers of respective
> > applications, don't try to dictate how those applications should be
> > organized and developed.
>
> A well-behaved application will detect which features are available in

No, a well-behaved application will provide a reliable functionality
without necessarily paying maintenance and development cost of a maze
of #ifdef-ery just to satisfy arbitrary distro requirements of linking
with some shared library ("because security").

> the system version of the libraries they use, and if something is
> missing that it needs, either work around it or refuse to build. We do
> this with libbpf in xdp-tools and the only issues we've had with it has
> been the changing API in pre-1.0 libbpf...
>
> > One good example is iproute2, which chose to link (or not) with libbpf
> > dynamically. Now users periodically report various issues where their
> > BPF object files are not loaded, and it often comes down to unexpected
> > version of libbpf (or lack of libbpf support altogether) which which
> > iproute2 was built/deployed. This is just putting a burden on iproute2
> > users, and accidentally libbpf maintainers, for no good reason.
>
> How would this have been any different if iproute2 was statically linked
> against libbpf?

iproute2 version would determine what BPF features are supported, and
it would be consistent across distros and end user systems, regardless
of what libbpf shared library happens to be packaged and installed.
And users would know that starting from version X iproute2 is
libbpf-1.0+ compatible in what sort of BPF object file features are
supported by iproute2 when loading BPF programs.

>
> >> I'm not saying that those tools should not have libbpf as submodule; as
> >> submodule do look useful. But for packaging I really would like to have the
> >> option of choosing the exact version of libbpf being used.
> >
> > The exact version of libbpf used by bpftool, retsnoop, veristat, etc
> > *is not relevant* to you as a packager. If you want happy users, use
> > *exact* version of libbpf from submodule to build them, with which
> > application was developed, tested, and advertised supported BPF
> > features. There is no reuse to be done here, they all can be on
> > different (and sometimes not yet released) libbpf version. For good
> > reasons, which are outside of your control as a packager.
>
> This is... just not how distributions work. As a user I trust my
> distribution to provide me with a coherent system where critical system
> libraries are maintained and receive timely updates. And I absolutely
> trust the distribution more to do this over application developers who
> just vendor in some version as a submodule and leave it there until they
> need a new feature...

Ok.

>
> -Toke
>

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Toke Høiland-Jørgensen]

Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:

>> >> > > > > By switching up actual libbpf used to compile with bpftool, you are
>> >> > > > > potentially introducing subtle problems that your users will be quite
>> >> > > > > unhappy about, if they run into them. Let's work together to make it
>> >> > > > > easier for you to package bpftool properly. We can't switch bpftool to
>> >> > > > > reliably use system-wide libbpf (either static or shared, doesn't
>> >> > > > > matter) because of dependency on internal functionality.
>> >> > > > >
>> >> > > > >
>> >> > > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
>> >> > > >
>> >> > > > So how many copies of libbpf do I need for having a CO-RE toolchain?
>> >> > >
>> >> > > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
>> >> > > etc are tools. The fact they are using statically linked libbpf
>> >> > > through Git submodule is irrelevant to end users. You need one libbpf
>> >> > > in the system (for those who link dynamically against libbpf), the
>> >> > > rest are just tools.
>> >> > >
>> >> > > >
>> >> > > > Will different tools have different view of the kernel because they each
>> >> > > > use different private copy of libbpf with different features?
>> >> > >
>> >> > > That's up to tools, not libbpf. You are over pivoting on libbpf here.
>> >> > > There is one view of the kernel, it depends on what features the
>> >> > > kernel supports. If the tool requires some specific functionality of
>> >> > > libbpf, it will update its Git submodule reference to get a version of
>> >> > > libbpf that provides that feature. That's the point, an
>> >> > > application/tool is in control of what kind of features it gets from
>> >> > > libbpf.
>> >>
>> >> Since libbpf has a stable API & ABI, is it theoretically possible for
>> >> bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
>> >
>> > No, because libbpf is not just a set of APIs. Newer libbpf versions
>> > support more BPF-side features, more kernel features, etc, etc. Libbpf
>> > is not a typical user-space library, it is a BPF loader, and even if
>> > user-visible API doesn't change, libbpf's support for various BPF-side
>> > features is extended. Which is important for tools like bpftool,
>> > retsnoop, veristat which rely on loading and working with BPF object
>> > files.
>>
>> The converse of this is also true: if your system is upgraded to a new
>> kernel version with new BPF features, the libbpf version should follow
>> it, and all applications linked against it will automatically take
>> advantage of any bugfixes regardless without having to wait for each
>> application to be updated.
>
> No, if my application was not developed to take advantage of a new
> kernel feature, newer libbpf will do nothing for me. If my application
> wants to support that feature, I'll update my application and
> correspondingly update libbpf embedded in it. If my application is
> affected by some bug fix, I'll update libbpf even faster than distros
> will get to it.

You may do that, but you're also someone who is following the
development of libbpf closely and pay attention to when bugs appear. Not
all applications developers have the same vigilance for all the
libraries they rely on. Which is the reason distros generally take on
the responsibility of ensuring their users receive timely library
updates.

> I've heard all such arguments over the last few years. They are not
> convincing and my own practical experience shows irrelevance of the
> above argument.

I don't doubt your personal experience, I'm just objecting to you
dismissing other points of view just because you haven't experienced
them yourself.

>> Libbpf is really no different from any other library here, and I really
>> don't get why you keep insisting it's "special"...
>
> It's special in the sense that it provides two sets of APIs -- for
> user-space (typical libraries) and BPF object files. Besides that, for
> BPF-side it's not even a set of APIs (headers, helpers, etc), it also
> provides some set of functionality that can improve or be extended
> over time. E.g., libbpf used to not support non-inlined BPF
> subprograms, and then it started supporting them. In terms of API/ABI
> -- nothing changed. Yet the change is very important.

Lots of libraries do that. File format libraries support new format
features without changing their API, networking libraries support new
protocol features, etc. So again, libbpf is not special in this
respect.

> Now, I build a tool that is using libbpf and some BPF functionality,
> e.g., retsnoop. Libbpf just got SEC("ksyscall") support. Retsnoop
> wants to take advantage of it. I just go and use SEC("ksyscall")
> programs in .bpf.c files that are embedded inside retsnoop.
> I don't have to *and don't want to* do feature detection of whether a
> particular libbpf version that happens to be installed/packaged on the
> system supports this version. I *know* it does, because I control it,
> through a submodule. That's what I care about.

Right, so just require a minimum version of the library where the API
you want to use is available. That is pretty standard and distros deal
with this all the time. This is not an argument for static linking or
vendoring...

> Whether some distro insists on libbpf being shared across any
> libbpf-using application or not is none of my concern. Libbpf is an
> implementation detail of my application (retsnoop), it's not for the
> packager to decide how I develop and structure my tool.

Right, well, you don't *have* to be cooperative with the wider
ecosystem, of course. Just as packagers don't have to follow your
recommendations if they have good reasons not to. I believe we've had
this discussion before, and I don't think we're going to agree this time
around either, so let's not waste any more virtual ink on rehashing it :)

-Toke

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Arnaldo Carvalho de Melo]

Em Wed, Apr 19, 2023 at 12:42:39PM -0700, Andrii Nakryiko escreveu:
> On Wed, Apr 19, 2023 at 3:55 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
> >
> > Thanks for sharing! I though I'd expands on what you said to draw a clearer
> > picture of the challenges.
> >
> > On Thu, Apr 13, 2023 at 01:00:20PM +0200, Toke Høiland-Jørgensen wrote:
> > > Shung-Hsi Yu <shung-hsi.yu@suse.com> writes:
> > >
> > > > A side note: if we want all userspace visible libbpf to have a coherent
> > > > version, perf needs to use the shared libbpf library as well (either
> > > > autodetected or forced with LIBBPF_DYNAMIC=1 like Fedora[4]). But having to
> > > > backport patches to kernel source to keep up with userspace package (libbpf)
> > > > changes could be a pain.
> >
> > Here some more context for completeness. Kernel source changes are published
> > at a much slower pace than userspace. When an application in the kernel
> > source (e.g. perf) depends on the userspace library, it's kind of like
> > trying to catchup a car on a bike, which is doable, as evident by the

That is why perf is continuously (at least) build tested against lots of
distros, see the last test output:

[perfbuilder@five ~]$ export BUILD_TARBALL=http://192.168.86.10/perf/perf-6.3.0-rc1.tar.xz
[perfbuilder@five ~]$ time dm
   1   151.45 almalinux:8                   : Ok   gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16) , clang version 14.0.6 (Red Hat 14.0.6-1.module_el8.7.0+3277+b822483f)
   2   150.54 almalinux:9                   : Ok   gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 14.0.6 (Red Hat 14.0.6-4.el9_1)
   3   159.04 alpine:3.15                   : Ok   gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027 , Alpine clang version 12.0.1
   4   153.81 alpine:3.16                   : Ok   gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219 , Alpine clang version 13.0.1
   5   137.88 alpine:3.17                   : Ok   gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924 , Alpine clang version 15.0.7
   6   139.32 alpine:edge                   : Ok   gcc (Alpine 12.2.1_git20220924-r9) 12.2.1 20220924 , Alpine clang version 16.0.0
   7   109.51 alt:p9                        : Ok   x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0
   8   104.99 alt:p10                       : Ok   x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1
   9    85.35 alt:sisyphus                  : Ok   x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt2) , ALT Linux Team clang version 13.0.1
  10   111.22 amazonlinux:2                 : Ok   gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15) , clang version 11.1.0 (Amazon Linux 2 11.1.0-1.amzn2.0.2)
  11   127.05 amazonlinux:2023              : Ok   gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4) , clang version 15.0.6 (Amazon Linux 15.0.6-3.amzn2023.0.2)
  12   126.56 amazonlinux:devel             : Ok   gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4) , clang version 15.0.6 (Amazon Linux 15.0.6-3.amzn2023.0.2)
  13   154.76 archlinux:base                : Ok   gcc (GCC) 12.2.0 , clang version 14.0.6
  14   135.39 centos:stream                 : Ok   gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18) , clang version 15.0.7 (Red Hat 15.0.7-1.module_el8.8.0+1258+af79b238)
  15    40.01 clearlinux:latest             : Ok   gcc (Clear Linux OS for Intel Architecture) 12.2.1 20230412 releases/gcc-12.2.0-699-g43ab94d20e
  16    95.89 debian:10                     : Ok   gcc (Debian 8.3.0-6) 8.3.0 , Debian clang version 11.0.1-2~deb10u1
  17   121.95 debian:11                     : Ok   gcc (Debian 10.2.1-6) 10.2.1 20210110 , Debian clang version 11.0.1-2
  18   136.49 debian:experimental           : Ok   gcc (Debian 12.2.0-14) 12.2.0 , Debian clang version 14.0.6
  19    29.78 debian:experimental-x-arm64   : Ok   aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0
  20    22.96 debian:experimental-x-mips    : Ok   mips-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0
  21     3.37 debian:experimental-x-mips64  : FAIL gcc version 10.2.1 20210110 (Debian 10.2.1-6)
  22    11.29 debian:experimental-x-mipsel  : FAIL gcc version 12.2.0 (Debian 12.2.0-14)
  23    33.28 fedora:26                     : Ok   gcc (GCC) 7.3.1 20180130 (Red Hat 7.3.1-2)
  24    33.29 fedora:27                     : Ok   gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6)
  25    29.77 fedora:28                     : Ok   gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
  26    32.27 fedora:29                     : Ok   gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
  27    34.18 fedora:30                     : Ok   gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
  28   139.98 fedora:31                     : Ok   gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2) , clang version 9.0.1 (Fedora 9.0.1-4.fc31)
  29   119.93 fedora:32                     : Ok   gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) , clang version 10.0.1 (Fedora 10.0.1-3.fc32)
  30   123.87 fedora:33                     : Ok   gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) , clang version 11.0.0 (Fedora 11.0.0-3.fc33)
  31   188.77 fedora:34                     : Ok   gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 12.0.1 (Fedora 12.0.1-1.fc34)
  32    22.44 fedora:34-x-ARC-glibc         : Ok   arc-linux-gcc (ARC HS GNU/Linux glibc toolchain 2019.03-rc1) 8.3.1 20190225
  33    20.33 fedora:34-x-ARC-uClibc        : Ok   arc-linux-gcc (ARCv2 ISA Linux uClibc toolchain 2019.03-rc1) 8.3.1 20190225
  34   172.51 fedora:35                     : Ok   gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-3) , clang version 13.0.1 (Fedora 13.0.1-1.fc35)
  35   138.29 fedora:36                     : Ok   gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4) , clang version 14.0.5 (Fedora 14.0.5-2.fc36)
  36   139.99 fedora:37                     : Ok   gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4) , clang version 15.0.7 (Fedora 15.0.7-1.fc37)
  37   144.73 fedora:38                     : Ok   gcc (GCC) 13.0.1 20230401 (Red Hat 13.0.1-0) , clang version 16.0.0 (Fedora 16.0.0-2.fc38)
  38   191.37 fedora:39                     : Ok   gcc (GCC) 13.0.1 20230404 (Red Hat 13.0.1-0) , clang version 16.0.1 (Fedora 16.0.1-1.fc39)
  39   187.13 fedora:rawhide                : Ok   gcc (GCC) 13.0.1 20230404 (Red Hat 13.0.1-0) , clang version 16.0.1 (Fedora 16.0.1-1.fc39)
  40    13.08 gentoo:stage3                 : FAIL gcc version 12.2.1 20230121 (Gentoo 12.2.1_p20230121-r1 p10)
  41   152.25 manjaro:base                  : Ok   gcc (GCC) 12.2.1 20230201 , clang version 15.0.7
  42   148.30 opensuse:15.4                 : Ok   gcc (SUSE Linux) 7.5.0 , clang version 13.0.1
  43   152.82 opensuse:15.5                 : Ok   gcc (SUSE Linux) 7.5.0 , clang version 15.0.7
  44   187.75 opensuse:tumbleweed           : Ok   gcc (SUSE Linux) 12.2.1 20221020 [revision 0aaef83351473e8f4eb774f8f999bbe87a4866d7] , clang version 15.0.6
  45   133.67 oraclelinux:8                 : Ok   gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16.0.2) , clang version 14.0.6 (Red Hat 14.0.6-1.0.1.module+el8.7.0+20823+214a699d)
  46   132.26 oraclelinux:9                 : Ok   gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2.1.0.2) , clang version 14.0.6 (Red Hat 14.0.6-4.0.1.el9_1)
  47   150.02 rockylinux:8                  : Ok   gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16) , clang version 14.0.6 (Red Hat 14.0.6-1.module+el8.7.0+1080+d88dc670)
  48   133.78 rockylinux:9                  : Ok   gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2) , clang version 14.0.6 (Red Hat 14.0.6-4.el9_1)
  49    28.97 ubuntu:18.04                  : Ok   gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  50    25.06 ubuntu:18.04-x-arm            : Ok   arm-linux-gnueabihf-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
  51    25.06 ubuntu:18.04-x-arm64          : Ok   aarch64-linux-gnu-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
  52    20.54 ubuntu:18.04-x-m68k           : Ok   m68k-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  53    24.56 ubuntu:18.04-x-powerpc        : Ok   powerpc-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  54    26.07 ubuntu:18.04-x-powerpc64      : Ok   powerpc64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  55    26.46 ubuntu:18.04-x-powerpc64el    : Ok   powerpc64le-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  56   113.72 ubuntu:18.04-x-riscv64        : Ok   riscv64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  57    22.55 ubuntu:18.04-x-s390           : Ok   s390x-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  58    24.35 ubuntu:18.04-x-sh4            : Ok   sh4-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  59    22.54 ubuntu:18.04-x-sparc64        : Ok   sparc64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
  60    31.87 ubuntu:20.04                  : Ok   gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
  61   179.64 ubuntu:22.04                  : Ok   gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 , Ubuntu clang version 14.0.0-1ubuntu1
  62     1.26 ubuntu:22.04-x-riscv64        : FAIL gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
  63    87.25 ubuntu:22.10                  : Ok   gcc (Ubuntu 12.2.0-3ubuntu1) 12.2.0 , Ubuntu clang version 15.0.7
  64    86.96 ubuntu:23.04                  : Ok   gcc (Ubuntu 12.2.0-17ubuntu1) 12.2.0 , Ubuntu clang version 15.0.7
BUILD_TARBALL_HEAD=f8b04f975d2c3d7c8e8cb53155744c20a41813ac
65 6011.52

real	101m23.391s
user	0m51.216s
sys	0m40.407s
[perfbuilder@five ~]$

> > plethora of userspace libraries perf already depends on. While I don't
> > having experience maintaining perf, judging by tools/perf/Makefile.config
> > that does not seem like an easy feat.

Not that bad having the tests we have in place :-)

> > For perf to use libbpf in kernel would mean that it's just depending on
> > something that moves at the same pace.

More tests to check build both with the in-kernel libbpf and with the
one in the distro:

⬢[acme@toolbox perf-tools-next]$ grep LIBBPF_DYNAMIC tools/perf/tests/make
make_libbpf_dynamic := LIBBPF_DYNAMIC=1
⬢[acme@toolbox perf-tools-next]$


> > That said, maybe perf won't need additional backport to keep up with libbpf
> > as long as we keep it within that same major version (and disable
> > deprecation warning)? @Andrii
> >
> > Now that We've got pass libbpf 1.0 it seems like a good time to reconsider.
> 
> I'm not sure what the proposal is, but I'll delegate to Arnaldo.
 
> > > So basically, this here is the reason we're building libbpf from the
> > > kernel tree for the RHEL package: If we use the github version we'd need
> > > to juggle two different versions of libbpf, one for the in-kernel-tree
> > > users (perf as you mention, but also the BPF selftests), and one for the

Have you tried with LIBBPF_DYNAMIC=1?

> > > userspace packages. Also, having libbpf in the kernel tree means we can
> > > just backport patches to it along with the BPF-related kernel patches
> > > (we do quite extensive BPF backports for each RHEL version).
> >
> > > Finally, building from the kernel tree means we can use the existing
> > > kernel-related procedures for any out of order hotfixes (since AFAIK none
> > > of the github repositories have any concept of stable branches that
> > > receive fixes).
> >
> > +1
> >
> > Got something similar in place as well and being able to stick with existing
> > procedure is appealing.
> >
> > > YMMV of course, but figured I'd share our reasoning. To be clear,
> > > building from the kernel tree is not without its own pain points (mostly
> > > related to how the build scripts are structured for our kernel builds).
> > > We've discussed moving to the github version of libbpf multiple times,
> > > but every time we've concluded that it would be more, not less, painful
> > > than having the kernel tree be the single source of truth.
> >
> > We package maintainer are certainly quite hard to please :)
> >
> > Just having an individual package easy to work with is not enough, we want
> > it to be easier for most packages before jumping on the bandwagon, which is
> > why this email ended up talking about perf despite it started as a
> > discussion on packaging libbpf and bpftool.
> >
> > I suppose the mileage depends on the build system & scripts in use and how
> > much backporting is done; the more kernel backporting (along with more
> > established processes in place), the more painful it'd be to move to the
> > GitHub version. My gut feeling is that SLES do less backporting compared to
> > RHEL when it comes to BPF, and that probably placed us closer to the middle
> > ground.
> 
> Even though libbpf source is developed in kernel repo, it's not tied
> to specific kernel. So any kernel backports have no relevance to
> libbpf itself. It's yet another reason to switch to Github mirror.
> Github is merging libbpf-related fixes from both bpf and bpf-next
> trees during sync, and is meant to always be the latest and best
> version with all fixes included.
> 
> I won't claim anything for perf, maybe Arnaldo can clarify, but I
> suspect that perf is also meant to be relatively independent from
> specific kernel and work on wide variety of kernels.
> 
> As for stable branches. For libbpf, we don't have it because we didn't
> need it yet. We did have bug fix patch releases that seem to be
> working out fine, though.
> 
> >
> > Thanks,
> > Shung-Hsi
> >
> > > -Toke
> > >

-- 

- Arnaldo

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Thu, Apr 20, 2023 at 7:46 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:
>
> >> >> > > > > By switching up actual libbpf used to compile with bpftool, you are
> >> >> > > > > potentially introducing subtle problems that your users will be quite
> >> >> > > > > unhappy about, if they run into them. Let's work together to make it
> >> >> > > > > easier for you to package bpftool properly. We can't switch bpftool to
> >> >> > > > > reliably use system-wide libbpf (either static or shared, doesn't
> >> >> > > > > matter) because of dependency on internal functionality.
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >   [0] https://github.com/libbpf/veristat/releases/tag/v0.1
> >> >> > > >
> >> >> > > > So how many copies of libbpf do I need for having a CO-RE toolchain?
> >> >> > >
> >> >> > > What do you mean by "CO-RE toolchain"? bpftool, veristat, retsnoop,
> >> >> > > etc are tools. The fact they are using statically linked libbpf
> >> >> > > through Git submodule is irrelevant to end users. You need one libbpf
> >> >> > > in the system (for those who link dynamically against libbpf), the
> >> >> > > rest are just tools.
> >> >> > >
> >> >> > > >
> >> >> > > > Will different tools have different view of the kernel because they each
> >> >> > > > use different private copy of libbpf with different features?
> >> >> > >
> >> >> > > That's up to tools, not libbpf. You are over pivoting on libbpf here.
> >> >> > > There is one view of the kernel, it depends on what features the
> >> >> > > kernel supports. If the tool requires some specific functionality of
> >> >> > > libbpf, it will update its Git submodule reference to get a version of
> >> >> > > libbpf that provides that feature. That's the point, an
> >> >> > > application/tool is in control of what kind of features it gets from
> >> >> > > libbpf.
> >> >>
> >> >> Since libbpf has a stable API & ABI, is it theoretically possible for
> >> >> bpftool, veristat, retsnoop, etc. all share the same version of libbpf?
> >> >
> >> > No, because libbpf is not just a set of APIs. Newer libbpf versions
> >> > support more BPF-side features, more kernel features, etc, etc. Libbpf
> >> > is not a typical user-space library, it is a BPF loader, and even if
> >> > user-visible API doesn't change, libbpf's support for various BPF-side
> >> > features is extended. Which is important for tools like bpftool,
> >> > retsnoop, veristat which rely on loading and working with BPF object
> >> > files.
> >>
> >> The converse of this is also true: if your system is upgraded to a new
> >> kernel version with new BPF features, the libbpf version should follow
> >> it, and all applications linked against it will automatically take
> >> advantage of any bugfixes regardless without having to wait for each
> >> application to be updated.
> >
> > No, if my application was not developed to take advantage of a new
> > kernel feature, newer libbpf will do nothing for me. If my application
> > wants to support that feature, I'll update my application and
> > correspondingly update libbpf embedded in it. If my application is
> > affected by some bug fix, I'll update libbpf even faster than distros
> > will get to it.
>
> You may do that, but you're also someone who is following the
> development of libbpf closely and pay attention to when bugs appear. Not
> all applications developers have the same vigilance for all the
> libraries they rely on. Which is the reason distros generally take on
> the responsibility of ensuring their users receive timely library
> updates.

The discussion was about bpftool, veristat, and retsnoop. "You may do
that" applies to all of them. I'm not forcing anyone else to follow
the same approach (e.g., I'm not forcing perf to vendor libbpf, for
example), I'm just opposing someone else forcing us (bpftool,
veristat, retsnoop) to not vendor libbpf.

>
> > I've heard all such arguments over the last few years. They are not
> > convincing and my own practical experience shows irrelevance of the
> > above argument.
>
> I don't doubt your personal experience, I'm just objecting to you
> dismissing other points of view just because you haven't experienced
> them yourself.

I acknowledged the security argument. And disagreeing and dismissing
are two big differences. So yes, I don't think the security argument
outweighs all the downsides of not controlling the exact libbpf
version my application relies on. And specifically for libbpf-related
use cases, libbpf-using applications are running under root. They are
not supposed to accept untrusted ELF files. So even if there is some
bug leading to crashes or some malfunction, I don't buy the
"emergency, we need to update all libbpf-using apps ASAP" argument,
sorry.

>
> >> Libbpf is really no different from any other library here, and I really
> >> don't get why you keep insisting it's "special"...
> >
> > It's special in the sense that it provides two sets of APIs -- for
> > user-space (typical libraries) and BPF object files. Besides that, for
> > BPF-side it's not even a set of APIs (headers, helpers, etc), it also
> > provides some set of functionality that can improve or be extended
> > over time. E.g., libbpf used to not support non-inlined BPF
> > subprograms, and then it started supporting them. In terms of API/ABI
> > -- nothing changed. Yet the change is very important.
>
> Lots of libraries do that. File format libraries support new format
> features without changing their API, networking libraries support new
> protocol features, etc. So again, libbpf is not special in this
> respect.

I didn't mean to start a "which library is the most special" contest.
The original point was about libbpf 1.0 stability of API, and that was
my objection, because it's not just about available APIs and their
stability. And libbpf is not used for the sake of using libbpf, it is
used with (usually embedded) BPF object file(s) that application is
expected to work, including any new SEC() support, global variables,
etc, etc.

>
> > Now, I build a tool that is using libbpf and some BPF functionality,
> > e.g., retsnoop. Libbpf just got SEC("ksyscall") support. Retsnoop
> > wants to take advantage of it. I just go and use SEC("ksyscall")
> > programs in .bpf.c files that are embedded inside retsnoop.
> > I don't have to *and don't want to* do feature detection of whether a
> > particular libbpf version that happens to be installed/packaged on the
> > system supports this version. I *know* it does, because I control it,
> > through a submodule. That's what I care about.
>
> Right, so just require a minimum version of the library where the API
> you want to use is available. That is pretty standard and distros deal
> with this all the time. This is not an argument for static linking or
> vendoring...

bpftool can't do that due to use of internal APIs. For others
(retsnoop, veristat), I don't want to be restricted by the released
libbpf version, or by expecting that the target system has new enough
libbpf installed.

>
> > Whether some distro insists on libbpf being shared across any
> > libbpf-using application or not is none of my concern. Libbpf is an
> > implementation detail of my application (retsnoop), it's not for the
> > packager to decide how I develop and structure my tool.
>
> Right, well, you don't *have* to be cooperative with the wider
> ecosystem, of course. Just as packagers don't have to follow your
> recommendations if they have good reasons not to. I believe we've had
> this discussion before, and I don't think we're going to agree this time
> around either, so let's not waste any more virtual ink on rehashing it :)

Exactly, so I'm not sure why we are even having this conversation all
over again. I agree on not wasting virtual ink anymore. I'm not
forcing anyone to follow my advice, I expect others to not force me to
follow theirs.

>
> -Toke
>

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Vince Weaver]

On Wed, 19 Apr 2023, tonyj@suse.de wrote:
> 
> For a while Vince was maintaining a page documenting breakage in the 
> perf_event ABI but last I checked it ended at V4.0.  Not sure if he 
> just stopped updating, or that's where breakage ended :)

I actually still try to keep an eye on things.  I originally tracked 
things because we had to work around a lot of those breakages in PAPI.  

There have been a few breakages since then, mostly in rdpmc support and 
also some weird things on ARM64.  I stopped documenting things as 
thoroughly in part because despite the common wisdom I don't think the 
kernel devs care much about ABI breaks at least with the perf interface.

Vince

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Shung-Hsi Yu]

On Thu, Apr 20, 2023 at 02:39:17PM -0700, Andrii Nakryiko wrote:
> On Thu, Apr 20, 2023 at 7:46 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> > [snip]
> > Right, well, you don't *have* to be cooperative with the wider
> > ecosystem, of course. Just as packagers don't have to follow your
> > recommendations if they have good reasons not to. I believe we've had
> > this discussion before, and I don't think we're going to agree this time
> > around either, so let's not waste any more virtual ink on rehashing it :)
> 
> Exactly, so I'm not sure why we are even having this conversation all
> over again. I agree on not wasting virtual ink anymore. I'm not
> forcing anyone to follow my advice, I expect others to not force me to
> follow theirs.

Thanks for still going through the reasoning.

I don't have anything to add to the discussion, so instead here's an attempt
to summarize the thread thus far, reading between the lines here and there
to keep it terse but complete; feel free to point out where I misunderstood.


# Packaging bpftool and libbpf

- bpftool and libbpf version should be kept in sync
  - interdependency is by design
  - bpftool uses private functionality of libbpf
  - bpftool generated file is tie to specific libbpf (?)

- the GitHub mirror is the recommended source 

- benefits of using the GitHub mirror includes
  - ease of upgrade
  - maintainer crafted changelog

- downsides of using the GitHub mirror has to do with kernel backporting

- git submodule requires extra work for distros to package
  - offsetted if the source of submodules are released along

- bpftool releases will (have a file that) includes submodules' source along
  going forward

- bpftool and libbpf both should work on earlier kernel (if not it's a bug)


# Other

- motivations for GitHub mirror
  - ease of distribution, packaging, build
  - CI, to be used as submodule, Window support, etc.

- libbpf interface stability
  - stable API and ABI (within major version)
  - BPF object format is not considered stable

- libbpf is not opinionated in how it's used as a library, either
  - statically or dynamically linked
  - a tagged release or a random commit

- on statically linking libbpf
  - reasoning
    - full control of implementation detail, decouples from distro package
  - against
    - difficulty in applying fixes


Shung-Hsi

> >
> > -Toke
> >

Re: Packaging bpftool and libbpf: GitHub or kernel?

[Andrii Nakryiko]

On Fri, Apr 28, 2023 at 2:13 AM Shung-Hsi Yu <shung-hsi.yu@suse.com> wrote:
>
> On Thu, Apr 20, 2023 at 02:39:17PM -0700, Andrii Nakryiko wrote:
> > On Thu, Apr 20, 2023 at 7:46 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> > > [snip]
> > > Right, well, you don't *have* to be cooperative with the wider
> > > ecosystem, of course. Just as packagers don't have to follow your
> > > recommendations if they have good reasons not to. I believe we've had
> > > this discussion before, and I don't think we're going to agree this time
> > > around either, so let's not waste any more virtual ink on rehashing it :)
> >
> > Exactly, so I'm not sure why we are even having this conversation all
> > over again. I agree on not wasting virtual ink anymore. I'm not
> > forcing anyone to follow my advice, I expect others to not force me to
> > follow theirs.
>
> Thanks for still going through the reasoning.
>
> I don't have anything to add to the discussion, so instead here's an attempt
> to summarize the thread thus far, reading between the lines here and there
> to keep it terse but complete; feel free to point out where I misunderstood.
>
>
> # Packaging bpftool and libbpf
>
> - bpftool and libbpf version should be kept in sync
>   - interdependency is by design
>   - bpftool uses private functionality of libbpf
>   - bpftool generated file is tie to specific libbpf (?)

this bullet point is not true, generated BPF skeleton or statically
linked BPF object file should work across wide range of libbpf
versions

>
> - the GitHub mirror is the recommended source
>
> - benefits of using the GitHub mirror includes
>   - ease of upgrade
>   - maintainer crafted changelog

I'd also say consistency between all distros (assuming everyone use
Github mirror). Because release X means exactly the same set of
commits.

>
> - downsides of using the GitHub mirror has to do with kernel backporting
>
> - git submodule requires extra work for distros to package
>   - offsetted if the source of submodules are released along
>
> - bpftool releases will (have a file that) includes submodules' source along
>   going forward
>
> - bpftool and libbpf both should work on earlier kernel (if not it's a bug)
>
>
> # Other
>
> - motivations for GitHub mirror
>   - ease of distribution, packaging, build
>   - CI, to be used as submodule, Window support, etc.
>
> - libbpf interface stability
>   - stable API and ABI (within major version)
>   - BPF object format is not considered stable

BPF object file format is stable, but not frozen, it can keep evolving

>
> - libbpf is not opinionated in how it's used as a library, either
>   - statically or dynamically linked
>   - a tagged release or a random commit
>
> - on statically linking libbpf
>   - reasoning
>     - full control of implementation detail, decouples from distro package
>   - against
>     - difficulty in applying fixes
>

Looks good to me apart from things I pointed out above. Thanks for summarizing!


>
> Shung-Hsi
>
> > >
> > > -Toke
> > >