BPF Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH v2 bpf-next 00/11] libbpf: split BTF support
@ 2020-11-05  4:33 Andrii Nakryiko
  2020-11-05  4:33 ` [PATCH v2 bpf-next 01/11] libbpf: factor out common operations in BTF writing APIs Andrii Nakryiko
                   ` (12 more replies)
  0 siblings, 13 replies; 17+ messages in thread
From: Andrii Nakryiko @ 2020-11-05  4:33 UTC (permalink / raw)
  To: bpf, netdev, ast, daniel; +Cc: andrii, kernel-team

This patch set adds support for generating and deduplicating split BTF. This
is an enhancement to the BTF, which allows to designate one BTF as the "base
BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
kernel module BTF), which are building upon and extending base BTF with extra
types and strings.

Once loaded, split BTF appears as a single unified BTF superset of base BTF,
with continuous and transparent numbering scheme. This allows all the existing
users of BTF to work correctly and stay agnostic to the base/split BTFs
composition.  The only difference is in how to instantiate split BTF: it
requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
or btf__parse_xxx_split() "constructors" explicitly.

This split approach is necessary if we are to have a reasonably-sized kernel
module BTFs. By deduping each kernel module's BTF individually, resulting
module BTFs contain copies of a lot of kernel types that are already present
in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
kernel configuration with 700 modules built, non-split BTF approach results in
115MBs of BTFs across all modules. With split BTF deduplication approach,
total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
around 4MBs). This seems reasonable and practical. As to why we'd need kernel
module BTFs, that should be pretty obvious to anyone using BPF at this point,
as it allows all the BTF-powered features to be used with kernel modules:
tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.

This patch set is a pre-requisite to adding split BTF support to pahole, which
is a prerequisite to integrating split BTF into the Linux kernel build setup
to generate BTF for kernel modules. The latter will come as a follow-up patch
series once this series makes it to the libbpf and pahole makes use of it.

Patch #4 introduces necessary basic support for split BTF into libbpf APIs.
Patch #8 implements minimal changes to BTF dedup algorithm to allow
deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to
specify the path to base BTF for cases when one wants to dump or inspect split
BTF. All the rest are refactorings, clean ups, bug fixes and selftests.

v1->v2:
  - addressed Song's feedback.


Andrii Nakryiko (11):
  libbpf: factor out common operations in BTF writing APIs
  selftest/bpf: relax btf_dedup test checks
  libbpf: unify and speed up BTF string deduplication
  libbpf: implement basic split BTF support
  selftests/bpf: add split BTF basic test
  selftests/bpf: add checking of raw type dump in BTF writer APIs
    selftests
  libbpf: fix BTF data layout checks and allow empty BTF
  libbpf: support BTF dedup of split BTFs
  libbpf: accomodate DWARF/compiler bug with duplicated identical arrays
  selftests/bpf: add split BTF dedup selftests
  tools/bpftool: add bpftool support for split BTF

 tools/bpf/bpftool/btf.c                       |   9 +-
 tools/bpf/bpftool/main.c                      |  15 +-
 tools/bpf/bpftool/main.h                      |   1 +
 tools/lib/bpf/btf.c                           | 807 ++++++++++--------
 tools/lib/bpf/btf.h                           |   8 +
 tools/lib/bpf/libbpf.map                      |   9 +
 tools/testing/selftests/bpf/Makefile          |   2 +-
 tools/testing/selftests/bpf/btf_helpers.c     | 259 ++++++
 tools/testing/selftests/bpf/btf_helpers.h     |  19 +
 tools/testing/selftests/bpf/prog_tests/btf.c  |  40 +-
 .../bpf/prog_tests/btf_dedup_split.c          | 325 +++++++
 .../selftests/bpf/prog_tests/btf_split.c      |  99 +++
 .../selftests/bpf/prog_tests/btf_write.c      |  43 +
 tools/testing/selftests/bpf/test_progs.h      |  11 +
 14 files changed, 1292 insertions(+), 355 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/btf_helpers.c
 create mode 100644 tools/testing/selftests/bpf/btf_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_split.c

-- 
2.24.1


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, back to index

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-05  4:33 [PATCH v2 bpf-next 00/11] libbpf: split BTF support Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 01/11] libbpf: factor out common operations in BTF writing APIs Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 02/11] selftest/bpf: relax btf_dedup test checks Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 03/11] libbpf: unify and speed up BTF string deduplication Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 04/11] libbpf: implement basic split BTF support Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 05/11] selftests/bpf: add split BTF basic test Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 06/11] selftests/bpf: add checking of raw type dump in BTF writer APIs selftests Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 07/11] libbpf: fix BTF data layout checks and allow empty BTF Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 08/11] libbpf: support BTF dedup of split BTFs Andrii Nakryiko
2020-11-05  4:33 ` [PATCH v2 bpf-next 09/11] libbpf: accomodate DWARF/compiler bug with duplicated identical arrays Andrii Nakryiko
2020-11-05  4:34 ` [PATCH v2 bpf-next 10/11] selftests/bpf: add split BTF dedup selftests Andrii Nakryiko
2020-11-05  4:34 ` [PATCH v2 bpf-next 11/11] tools/bpftool: add bpftool support for split BTF Andrii Nakryiko
2020-11-05  9:52 ` [PATCH v2 bpf-next 00/11] libbpf: split BTF support Jesper Dangaard Brouer
2020-11-05 19:16   ` Andrii Nakryiko
2020-11-05 19:38     ` Saeed Mahameed
2020-11-05 20:02       ` Andrii Nakryiko
2020-11-06  2:50 ` patchwork-bot+netdevbpf

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git