All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 bpf-next 00/17] BPF static linker: support externs
@ 2021-04-16 20:23 Andrii Nakryiko
  2021-04-16 20:23 ` [PATCH v2 bpf-next 01/17] bpftool: support dumping BTF VAR's "extern" linkage Andrii Nakryiko
                   ` (16 more replies)
  0 siblings, 17 replies; 67+ messages in thread
From: Andrii Nakryiko @ 2021-04-16 20:23 UTC (permalink / raw)
  To: bpf, netdev, ast, daniel; +Cc: andrii, kernel-team

Add BPF static linker support for extern resolution of global variables,
functions, and BTF-defined maps.

This patch set consists of 4 parts:
  - few patches are extending bpftool to simplify working with BTF dump;
  - libbpf object loading logic is extended to support __hidden functions and
    overriden (unused) __weak functions; also BTF-defined map parsing logic is
    refactored to be re-used by linker;
  - the crux of the patch set is BPF static linker logic extension to perform
    extern resolution for three categories: global variables, BPF
    sub-programs, BTF-defined maps;
  - a set of selftests that validate that all the combinations of
    extern/weak/__hidden are working as expected.

See respective patches for more details.

One aspect hasn't been addressed yet and is going to be resolved in the next
patch set, but is worth mentioning. With BPF static linking of multiple .o
files, dealing with static everything becomes more problematic for BPF
skeleton and in general for any by name look up APIs. This is due to static
entities are allowed to have non-unique name. Historically this was never
a problem due to BPF programs were always confined to a single C file. That
changes now and needs to be addressed. The thinking so far is for BPF static
linker to prepend filename to each static variable and static map (which is
currently not supported by libbpf, btw), so that they can be unambiguously
resolved by (mostly) unique name. Mostly, because even filenames can be
duplicated, but that should be rare and easy to address by wiser choice of
filenames by users. Fortunately, static BPF subprograms don't suffer from this
issues, as they are not independent entities and are neither exposed in BPF
skeleton, nor is lookup-able by any of libbpf APIs (and there is little reason
to do that anyways).

This and few other things will be the topic of the next set of patches.

Some tests rely on Clang fix ([0]), so need latest Clang built from main.

  [0] https://reviews.llvm.org/D100362

v1->v2:
  - make map externs support full attribute list, adjust linked_maps selftest
    to demonstrate that typedef works now (though no shared header file was
    added to simplicity sake) (Alexei);
  - remove commented out parts from selftests and fix few minor code style
    issues;
  - special __weak map definition semantics not yet implemented and will be
    addressed in a follow up.

Andrii Nakryiko (17):
  bpftool: support dumping BTF VAR's "extern" linkage
  bpftool: dump more info about DATASEC members
  libbpf: suppress compiler warning when using SEC() macro with externs
  libbpf: mark BPF subprogs with hidden visibility as static for BPF
    verifier
  libbpf: allow gaps in BPF program sections to support overriden weak
    functions
  libbpf: refactor BTF map definition parsing
  libbpf: factor out symtab and relos sanity checks
  libbpf: make few internal helpers available outside of libbpf.c
  libbpf: extend sanity checking ELF symbols with externs validation
  libbpf: tighten BTF type ID rewriting with error checking
  libbpf: add linker extern resolution support for functions and global
    variables
  libbpf: support extern resolution for BTF-defined maps in .maps
    section
  selftests/bpf: use -O0 instead of -Og in selftests builds
  selftests/bpf: omit skeleton generation for multi-linked BPF object
    files
  selftests/bpf: add function linking selftest
  selftests/bpf: add global variables linking selftest
  sleftests/bpf: add map linking selftest

 tools/bpf/bpftool/btf.c                       |   30 +-
 tools/lib/bpf/bpf_helpers.h                   |   19 +-
 tools/lib/bpf/btf.c                           |    5 -
 tools/lib/bpf/libbpf.c                        |  370 +++--
 tools/lib/bpf/libbpf_internal.h               |   45 +
 tools/lib/bpf/linker.c                        | 1261 ++++++++++++++---
 tools/testing/selftests/bpf/Makefile          |   18 +-
 .../selftests/bpf/prog_tests/linked_funcs.c   |   42 +
 .../selftests/bpf/prog_tests/linked_maps.c    |   30 +
 .../selftests/bpf/prog_tests/linked_vars.c    |   43 +
 .../selftests/bpf/progs/linked_funcs1.c       |   73 +
 .../selftests/bpf/progs/linked_funcs2.c       |   73 +
 .../selftests/bpf/progs/linked_maps1.c        |   82 ++
 .../selftests/bpf/progs/linked_maps2.c        |   76 +
 .../selftests/bpf/progs/linked_vars1.c        |   54 +
 .../selftests/bpf/progs/linked_vars2.c        |   55 +
 16 files changed, 1922 insertions(+), 354 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_funcs.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_maps.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/linked_vars.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_funcs1.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_funcs2.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_maps1.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_maps2.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_vars1.c
 create mode 100644 tools/testing/selftests/bpf/progs/linked_vars2.c

-- 
2.30.2


^ permalink raw reply	[flat|nested] 67+ messages in thread

end of thread, other threads:[~2021-04-23 18:00 UTC | newest]

Thread overview: 67+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-16 20:23 [PATCH v2 bpf-next 00/17] BPF static linker: support externs Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 01/17] bpftool: support dumping BTF VAR's "extern" linkage Andrii Nakryiko
2021-04-22  3:02   ` Yonghong Song
2021-04-16 20:23 ` [PATCH v2 bpf-next 02/17] bpftool: dump more info about DATASEC members Andrii Nakryiko
2021-04-22  3:09   ` Yonghong Song
2021-04-16 20:23 ` [PATCH v2 bpf-next 03/17] libbpf: suppress compiler warning when using SEC() macro with externs Andrii Nakryiko
2021-04-22  3:47   ` Yonghong Song
2021-04-22  3:59     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 04/17] libbpf: mark BPF subprogs with hidden visibility as static for BPF verifier Andrii Nakryiko
2021-04-22  5:43   ` Yonghong Song
2021-04-22 18:09     ` Andrii Nakryiko
2021-04-22 23:00       ` Yonghong Song
2021-04-22 23:28         ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 05/17] libbpf: allow gaps in BPF program sections to support overriden weak functions Andrii Nakryiko
2021-04-22  6:25   ` Yonghong Song
2021-04-22 18:10     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 06/17] libbpf: refactor BTF map definition parsing Andrii Nakryiko
2021-04-22 15:33   ` Yonghong Song
2021-04-22 18:40     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 07/17] libbpf: factor out symtab and relos sanity checks Andrii Nakryiko
2021-04-22 16:06   ` Yonghong Song
2021-04-22 18:16     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 08/17] libbpf: make few internal helpers available outside of libbpf.c Andrii Nakryiko
2021-04-22 16:19   ` Yonghong Song
2021-04-22 18:18     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 09/17] libbpf: extend sanity checking ELF symbols with externs validation Andrii Nakryiko
2021-04-22 16:35   ` Yonghong Song
2021-04-22 18:20     ` Andrii Nakryiko
2021-04-22 23:13       ` Yonghong Song
2021-04-16 20:23 ` [PATCH v2 bpf-next 10/17] libbpf: tighten BTF type ID rewriting with error checking Andrii Nakryiko
2021-04-22 16:50   ` Yonghong Song
2021-04-22 18:24     ` Andrii Nakryiko
2021-04-23  2:54       ` Alexei Starovoitov
2021-04-23  4:25         ` Andrii Nakryiko
2021-04-23  5:11           ` Alexei Starovoitov
2021-04-23 16:09             ` Andrii Nakryiko
2021-04-23 16:18               ` Alexei Starovoitov
2021-04-23 16:30                 ` Andrii Nakryiko
2021-04-23 16:34                   ` Alexei Starovoitov
2021-04-23 17:02                     ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 11/17] libbpf: add linker extern resolution support for functions and global variables Andrii Nakryiko
2021-04-22 21:27   ` Yonghong Song
2021-04-22 22:12     ` Andrii Nakryiko
2021-04-22 23:57       ` Yonghong Song
2021-04-23  2:36         ` Yonghong Song
2021-04-23  4:28           ` Andrii Nakryiko
2021-04-23  4:27         ` Andrii Nakryiko
2021-04-16 20:23 ` [PATCH v2 bpf-next 12/17] libbpf: support extern resolution for BTF-defined maps in .maps section Andrii Nakryiko
2021-04-22 22:56   ` Yonghong Song
2021-04-22 23:32     ` Andrii Nakryiko
2021-04-16 20:24 ` [PATCH v2 bpf-next 13/17] selftests/bpf: use -O0 instead of -Og in selftests builds Andrii Nakryiko
2021-04-23  0:05   ` Yonghong Song
2021-04-16 20:24 ` [PATCH v2 bpf-next 14/17] selftests/bpf: omit skeleton generation for multi-linked BPF object files Andrii Nakryiko
2021-04-23  0:13   ` Yonghong Song
2021-04-16 20:24 ` [PATCH v2 bpf-next 15/17] selftests/bpf: add function linking selftest Andrii Nakryiko
2021-04-23  0:50   ` Yonghong Song
2021-04-23  2:34     ` Alexei Starovoitov
2021-04-23  4:29       ` Andrii Nakryiko
2021-04-23 17:18     ` Andrii Nakryiko
2021-04-23 17:35       ` Yonghong Song
2021-04-23 17:55         ` Andrii Nakryiko
2021-04-23 17:58           ` Yonghong Song
2021-04-23 17:59             ` Andrii Nakryiko
2021-04-16 20:24 ` [PATCH v2 bpf-next 16/17] selftests/bpf: add global variables " Andrii Nakryiko
2021-04-23  1:01   ` Yonghong Song
2021-04-16 20:24 ` [PATCH v2 bpf-next 17/17] sleftests/bpf: add map " Andrii Nakryiko
2021-04-23  1:20   ` Yonghong Song

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.