bpf.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Andrii Nakryiko <andrii@kernel.org>
To: <bpf@vger.kernel.org>, <netdev@vger.kernel.org>, <ast@fb.com>,
	<daniel@iogearbox.net>
Cc: <andrii@kernel.org>, <kernel-team@fb.com>
Subject: [PATCH v4 bpf-next 00/12] BPF static linking
Date: Thu, 18 Mar 2021 12:40:24 -0700	[thread overview]
Message-ID: <20210318194036.3521577-1-andrii@kernel.org> (raw)

This patch set adds new libbpf APIs and their bpftool integration that allows
to perform static linking of BPF object files. Currently no extern resolution
across object files is performed. This is going to be the focus of the follow
up patches. But, given amount of code and logic necessary to perform just
basic functionality of linking together mostly independent BPF object files,
it was decided to land basic BPF linker code and logic first and extend it
afterwards.

The motivation for BPF static linking is to provide the functionality that is
naturally assumed for user-space development process: ability to structure
application's code without artificial restrictions of having all the code and
data (variables and maps) inside a single source code file.

This enables better engineering practices of splitting code into
well-encapsulated parts. It provides ability to hide internal state from other
parts of the code base through static variables and maps. It is also a first
steps towards having generic reusable BPF libraries.

Please see individual patches (mostly #6 and #7) for more details. Patch #10
passes all test_progs' individual BPF .o files through BPF static linker,
which is supposed to be a no-op operation, so is essentially validating that
BPF static linker doesn't produce corrupted ELF object files. Patch #11 adds
Makefile infra to be able to specify multi-file BPF object files and adds the
first multi-file test to validate correctness.

v3->v4:
  - fix Makefile copy/paste error of diff'ing invalid object files (Alexei);
  - fix uninitialized obj_name variable that could lead to bogus object names
    being used during skeleton generation (kernel-patches CI);
v2->v3:
  - added F(F(F(X))) = F(F(X)) test for all linked BPF object files (Alexei);
  - used reallocarray() more consistently in few places (Alexei);
  - improved bash completions for `gen object` (Quentin);
  - dropped .bpfo extension, but had to add optional `name OBJECT_FILE`
    parameter (path #8) to `gen skeleton` command to specify desired object
    name during skeleton generation;
  - fixed bug of merging DATASECS of special "license" and "version" sections.
    Linker currently strictly validates that all versions and licenses matches
    exactly and keeps only ELF symbols and BTF DATASEC from the very first
    object file with license/version. For all other object files, we ignore
    ELF symbols, but weren't ignoring DATASECs, which caused further problems
    of not being able to find a corresponding ELF symbol, if variable name
    differs between two files (which we test deliberately in multi-file
    linking selftest). The fix is to ignore BTF DATASECS;
v1->v2:
  - extracted `struct strset` to manage unique set of strings both for BTF and
    ELF SYMTAB (patch #4, refactors btf and btf_dedup logic as well) (Alexei);
  - fixed bugs in bpftool gen command; renamed it to `gen object`, added BASH
    completions and extended/updated man page (Quentin).


Andrii Nakryiko (12):
  libbpf: expose btf_type_by_id() internally
  libbpf: generalize BTF and BTF.ext type ID and strings iteration
  libbpf: rename internal memory-management helpers
  libbpf: extract internal set-of-strings datastructure APIs
  libbpf: add generic BTF type shallow copy API
  libbpf: add BPF static linker APIs
  libbpf: add BPF static linker BTF and BTF.ext support
  bpftool: add ability to specify custom skeleton object name
  bpftool: add `gen object` command to perform BPF static linking
  selftests/bpf: re-generate vmlinux.h and BPF skeletons if bpftool
    changed
  selftests/bpf: pass all BPF .o's through BPF static linker
  selftests/bpf: add multi-file statically linked BPF object file test

 .../bpf/bpftool/Documentation/bpftool-gen.rst |   78 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   17 +-
 tools/bpf/bpftool/gen.c                       |   72 +-
 tools/lib/bpf/Build                           |    2 +-
 tools/lib/bpf/btf.c                           |  714 +++---
 tools/lib/bpf/btf.h                           |    2 +
 tools/lib/bpf/btf_dump.c                      |    8 +-
 tools/lib/bpf/libbpf.c                        |   15 +-
 tools/lib/bpf/libbpf.h                        |   13 +
 tools/lib/bpf/libbpf.map                      |    5 +
 tools/lib/bpf/libbpf_internal.h               |   38 +-
 tools/lib/bpf/linker.c                        | 1941 +++++++++++++++++
 tools/lib/bpf/strset.c                        |  176 ++
 tools/lib/bpf/strset.h                        |   21 +
 tools/testing/selftests/bpf/Makefile          |   27 +-
 .../selftests/bpf/prog_tests/static_linked.c  |   40 +
 .../selftests/bpf/progs/test_static_linked1.c |   30 +
 .../selftests/bpf/progs/test_static_linked2.c |   31 +
 18 files changed, 2806 insertions(+), 424 deletions(-)
 create mode 100644 tools/lib/bpf/linker.c
 create mode 100644 tools/lib/bpf/strset.c
 create mode 100644 tools/lib/bpf/strset.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/static_linked.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_static_linked1.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_static_linked2.c

-- 
2.30.2


             reply	other threads:[~2021-03-18 19:41 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-18 19:40 Andrii Nakryiko [this message]
2021-03-18 19:40 ` [PATCH v4 bpf-next 01/12] libbpf: expose btf_type_by_id() internally Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 02/12] libbpf: generalize BTF and BTF.ext type ID and strings iteration Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 03/12] libbpf: rename internal memory-management helpers Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 04/12] libbpf: extract internal set-of-strings datastructure APIs Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 05/12] libbpf: add generic BTF type shallow copy API Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 06/12] libbpf: add BPF static linker APIs Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 07/12] libbpf: add BPF static linker BTF and BTF.ext support Andrii Nakryiko
2021-03-19 16:23   ` Jiri Olsa
2021-03-19 18:39     ` Andrii Nakryiko
2021-03-19 18:58       ` Jiri Olsa
2021-03-28 12:03         ` Jiri Olsa
2021-03-28 18:29           ` Andrii Nakryiko
2021-03-29 11:16             ` Jiri Olsa
2021-03-18 19:40 ` [PATCH v4 bpf-next 08/12] bpftool: add ability to specify custom skeleton object name Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 09/12] bpftool: add `gen object` command to perform BPF static linking Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 10/12] selftests/bpf: re-generate vmlinux.h and BPF skeletons if bpftool changed Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 11/12] selftests/bpf: pass all BPF .o's through BPF static linker Andrii Nakryiko
2021-03-18 19:40 ` [PATCH v4 bpf-next 12/12] selftests/bpf: add multi-file statically linked BPF object file test Andrii Nakryiko
2021-03-18 23:29 ` [PATCH v4 bpf-next 00/12] BPF static linking Alexei Starovoitov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210318194036.3521577-1-andrii@kernel.org \
    --to=andrii@kernel.org \
    --cc=ast@fb.com \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=kernel-team@fb.com \
    --cc=netdev@vger.kernel.org \
    --subject='Re: [PATCH v4 bpf-next 00/12] BPF static linking' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).