From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Andrii Nakryiko <andrii@kernel.org>
Cc: brouer@redhat.com, <bpf@vger.kernel.org>,
<netdev@vger.kernel.org>, <ast@fb.com>, <daniel@iogearbox.net>,
<kernel-team@fb.com>, Saeed Mahameed <saeed@kernel.org>,
David Ahern <dsahern@kernel.org>,
Anton Protopopov <aspsk2@gmail.com>
Subject: Re: [PATCH v2 bpf-next 00/11] libbpf: split BTF support
Date: Thu, 5 Nov 2020 10:52:54 +0100 [thread overview]
Message-ID: <20201105105254.27c84b78@carbon> (raw)
In-Reply-To: <20201105043402.2530976-1-andrii@kernel.org>
On Wed, 4 Nov 2020 20:33:50 -0800
Andrii Nakryiko <andrii@kernel.org> wrote:
> This patch set adds support for generating and deduplicating split BTF. This
> is an enhancement to the BTF, which allows to designate one BTF as the "base
> BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
> kernel module BTF), which are building upon and extending base BTF with extra
> types and strings.
>
> Once loaded, split BTF appears as a single unified BTF superset of base BTF,
> with continuous and transparent numbering scheme. This allows all the existing
> users of BTF to work correctly and stay agnostic to the base/split BTFs
> composition. The only difference is in how to instantiate split BTF: it
> requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
> or btf__parse_xxx_split() "constructors" explicitly.
>
> This split approach is necessary if we are to have a reasonably-sized kernel
> module BTFs. By deduping each kernel module's BTF individually, resulting
> module BTFs contain copies of a lot of kernel types that are already present
> in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
> kernel configuration with 700 modules built, non-split BTF approach results in
> 115MBs of BTFs across all modules. With split BTF deduplication approach,
> total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
> around 4MBs). This seems reasonable and practical. As to why we'd need kernel
> module BTFs, that should be pretty obvious to anyone using BPF at this point,
> as it allows all the BTF-powered features to be used with kernel modules:
> tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
I love to see this work going forward.
My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules. I
want drivers to define a BTF struct that describe a meta-data area that
can be consumed/used by XDP, also available during xdp_frame to SKB
transition, which happens in net-core. So, I hope BTF-IDs are also
"available" from core kernel code?
> This patch set is a pre-requisite to adding split BTF support to pahole, which
> is a prerequisite to integrating split BTF into the Linux kernel build setup
> to generate BTF for kernel modules. The latter will come as a follow-up patch
> series once this series makes it to the libbpf and pahole makes use of it.
>
> Patch #4 introduces necessary basic support for split BTF into libbpf APIs.
> Patch #8 implements minimal changes to BTF dedup algorithm to allow
> deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to
> specify the path to base BTF for cases when one wants to dump or inspect split
> BTF. All the rest are refactorings, clean ups, bug fixes and selftests.
>
> v1->v2:
> - addressed Song's feedback.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2020-11-05 9:53 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-05 4:33 [PATCH v2 bpf-next 00/11] libbpf: split BTF support Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 01/11] libbpf: factor out common operations in BTF writing APIs Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 02/11] selftest/bpf: relax btf_dedup test checks Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 03/11] libbpf: unify and speed up BTF string deduplication Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 04/11] libbpf: implement basic split BTF support Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 05/11] selftests/bpf: add split BTF basic test Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 06/11] selftests/bpf: add checking of raw type dump in BTF writer APIs selftests Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 07/11] libbpf: fix BTF data layout checks and allow empty BTF Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 08/11] libbpf: support BTF dedup of split BTFs Andrii Nakryiko
2020-11-05 4:33 ` [PATCH v2 bpf-next 09/11] libbpf: accomodate DWARF/compiler bug with duplicated identical arrays Andrii Nakryiko
2020-11-05 4:34 ` [PATCH v2 bpf-next 10/11] selftests/bpf: add split BTF dedup selftests Andrii Nakryiko
2020-11-05 4:34 ` [PATCH v2 bpf-next 11/11] tools/bpftool: add bpftool support for split BTF Andrii Nakryiko
2020-11-05 9:52 ` Jesper Dangaard Brouer [this message]
2020-11-05 19:16 ` [PATCH v2 bpf-next 00/11] libbpf: split BTF support Andrii Nakryiko
2020-11-05 19:38 ` Saeed Mahameed
2020-11-05 20:02 ` Andrii Nakryiko
2020-11-06 2:50 ` patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201105105254.27c84b78@carbon \
--to=brouer@redhat.com \
--cc=andrii@kernel.org \
--cc=aspsk2@gmail.com \
--cc=ast@fb.com \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dsahern@kernel.org \
--cc=kernel-team@fb.com \
--cc=netdev@vger.kernel.org \
--cc=saeed@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).