Re: [PATCH bpf-next 00/17] BPF static linker: support externs

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Wed, Apr 14, 2021 at 1:02 PM Andrii Nakryiko <andrii@kernel.org> wrote:
>
> Add BPF static linker support for extern resolution of global variables,
> functions, and BTF-defined maps.
>
> This patch set consists of 4 parts:
>   - few patches are extending bpftool to simplify working with BTF dump;
>   - libbpf object loading logic is extended to support __hidden functions and
>     overriden (unused) __weak functions; also BTF-defined map parsing logic is
>     refactored to be re-used by linker;
>   - the crux of the patch set is BPF static linker logic extension to perform
>     extern resolution for three categories: global variables, BPF
>     sub-programs, BTF-defined maps;
>   - a set of selftests that validate that all the combinations of
>     extern/weak/__hidden are working as expected.
>
> See respective patches for more details.
>
> One aspect hasn't been addressed yet and is going to be resolved in the next
> patch set, but is worth mentioning. With BPF static linking of multiple .o
> files, dealing with static everything becomes more problematic for BPF
> skeleton and in general for any by name look up APIs. This is due to static
> entities are allowed to have non-unique name. Historically this was never
> a problem due to BPF programs were always confined to a single C file. That
> changes now and needs to be addressed. The thinking so far is for BPF static
> linker to prepend filename to each static variable and static map (which is
> currently not supported by libbpf, btw), so that they can be unambiguously
> resolved by (mostly) unique name. Mostly, because even filenames can be
> duplicated, but that should be rare and easy to address by wiser choice of
> filenames by users. Fortunately, static BPF subprograms don't suffer from this
> issues, as they are not independent entities and are neither exposed in BPF
> skeleton, nor is lookup-able by any of libbpf APIs (and there is little reason
> to do that anyways).
>
> This and few other things will be the topic of the next set of patches.
>

I forgot to mention that selftests are relying on Clang fix [0], which
is not yet in nightly builds, so kernel-patches CI is currently
failing a test. So, when/if testing locally, please make sure to build
Clang from latest main.

  [0] https://reviews.llvm.org/D100362

> Andrii Nakryiko (17):
>   bpftool: support dumping BTF VAR's "extern" linkage
>   bpftool: dump more info about DATASEC members
>   libbpf: suppress compiler warning when using SEC() macro with externs
>   libbpf: mark BPF subprogs with hidden visibility as static for BPF
>     verifier
>   libbpf: allow gaps in BPF program sections to support overriden weak
>     functions
>   libbpf: refactor BTF map definition parsing
>   libbpf: factor out symtab and relos sanity checks
>   libbpf: make few internal helpers available outside of libbpf.c
>   libbpf: extend sanity checking ELF symbols with externs validation
>   libbpf: tighten BTF type ID rewriting with error checking
>   libbpf: add linker extern resolution support for functions and global
>     variables
>   libbpf: support extern resolution for BTF-defined maps in .maps
>     section
>   selftests/bpf: use -O0 instead of -Og in selftests builds
>   selftests/bpf: omit skeleton generation for multi-linked BPF object
>     files
>   selftests/bpf: add function linking selftest
>   selftests/bpf: add global variables linking selftest
>   sleftests/bpf: add map linking selftest
>
>  tools/bpf/bpftool/btf.c                       |   30 +-
>  tools/lib/bpf/bpf_helpers.h                   |   19 +-
>  tools/lib/bpf/btf.c                           |    5 -
>  tools/lib/bpf/libbpf.c                        |  370 +++--
>  tools/lib/bpf/libbpf_internal.h               |   45 +
>  tools/lib/bpf/linker.c                        | 1292 ++++++++++++++---
>  tools/testing/selftests/bpf/Makefile          |   18 +-
>  .../selftests/bpf/prog_tests/linked_funcs.c   |   42 +
>  .../selftests/bpf/prog_tests/linked_maps.c    |   33 +
>  .../selftests/bpf/prog_tests/linked_vars.c    |   43 +
>  .../selftests/bpf/progs/linked_funcs1.c       |   73 +
>  .../selftests/bpf/progs/linked_funcs2.c       |   73 +
>  .../selftests/bpf/progs/linked_maps1.c        |  102 ++
>  .../selftests/bpf/progs/linked_maps2.c        |  112 ++
>  .../selftests/bpf/progs/linked_vars1.c        |   60 +
>  .../selftests/bpf/progs/linked_vars2.c        |   61 +
>  16 files changed, 2028 insertions(+), 350 deletions(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_funcs.c
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_maps.c
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_vars.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_funcs1.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_funcs2.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_maps1.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_maps2.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_vars1.c
>  create mode 100644 tools/testing/selftests/bpf/progs/linked_vars2.c
>
> --
> 2.30.2
>