Re: [PATCH v2 bpf-next 00/11] libbpf: split BTF support

From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
To: Saeed Mahameed <saeedm@nvidia.com>
Cc: "brouer@redhat.com" <brouer@redhat.com>,
	"aspsk2@gmail.com" <aspsk2@gmail.com>, "ast@fb.com" <ast@fb.com>,
	"dsahern@kernel.org" <dsahern@kernel.org>,
	"daniel@iogearbox.net" <daniel@iogearbox.net>,
	"kernel-team@fb.com" <kernel-team@fb.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"bpf@vger.kernel.org" <bpf@vger.kernel.org>,
	"andrii@kernel.org" <andrii@kernel.org>
Subject: Re: [PATCH v2 bpf-next 00/11] libbpf: split BTF support
Date: Thu, 5 Nov 2020 12:02:00 -0800	[thread overview]
Message-ID: <CAEf4Bza0HE-gJRt8kJWEu+i96aRDuroNVfPD6Od6+juoyv4CiQ@mail.gmail.com> (raw)
In-Reply-To: <105c48d56a550af6e0008b4b5867eb51764d41c9.camel@nvidia.com>

On Thu, Nov 5, 2020 at 11:38 AM Saeed Mahameed <saeedm@nvidia.com> wrote:
>
> On Thu, 2020-11-05 at 11:16 -0800, Andrii Nakryiko wrote:
> > > > This split approach is necessary if we are to have a reasonably-
> > > > sized kernel
> > > > module BTFs. By deduping each kernel module's BTF individually,
> > > > resulting
> > > > module BTFs contain copies of a lot of kernel types that are
> > > > already present
> > > > in vmlinux BTF. Even those single copies result in a big BTF size
> > > > bloat. On my
> > > > kernel configuration with 700 modules built, non-split BTF
> > > > approach results in
> > > > 115MBs of BTFs across all modules. With split BTF deduplication
> > > > approach,
> > > > total size is down to 5.2MBs total, which is on part with vmlinux
> > > > BTF (at
> > > > around 4MBs). This seems reasonable and practical. As to why we'd
> > > > need kernel
> > > > module BTFs, that should be pretty obvious to anyone using BPF at
> > > > this point,
> > > > as it allows all the BTF-powered features to be used with kernel
> > > > modules:
> > > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
> > > I love to see this work going forward.
> >
> >
> > Thanks.
> >
> >
> >
> > > My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules.
> > > I
> > > want drivers to define a BTF struct that describe a meta-data area
> > > that
> > > can be consumed/used by XDP, also available during xdp_frame to SKB
> > > transition, which happens in net-core. So, I hope BTF-IDs are also
> > > "available" from core kernel code?
> >
> >
> > I'll probably need a more specific example to understand what exactly
> >
> > you are asking and how you see everything working together, sorry.
> >
> >
>
> BTF-IDs can be made available for kernel/drivers, I've wrote a small
> patch for this a while ago.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6c1cb83629226889d6fadd3ba694e827fca3e247
>
> So the basic use case is that :
> 1- driver kernel/registers a BTF format (one or more).

This is now not needed, it happens automatically for module BTF.

> 2- Userland queries driver's registered BTF to be able understand the
> kernel/driver buffers format.

Here the module might need to know its BTF's ID, in addition to BTF
type ID. Or maybe it doesn't. User-space tools can just access BTF
from /sys/kernel/btf/module_name and use provided BTF type ID to dump
whatever is necessary.

>
> driver example of using this infrastructure:
> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=9c24657d6cb3a7852c2e948dc9782f3f39b60104

This, thankfully, won't be needed, you'll just have a normal C struct
and it will be just present in module's BTF.

>
> User Queries driver's XDP metadata BTF format:
> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6a117e2d9196f58de7cf067741e84ec242af27f6

For this we need support for BTF_ID macro for modules. As I said, it's
pretty easy to add, but feel free to contribute this once the basic
infra lands.

>
> Dump it as C header style
> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=
> 8bd99626879bff28379707ac3a2c3bb94fd5b410

This is available as libbpf-provided API now (see btf_dump APIs). And
bpftool has support to dump all BTF types as C definitions as well.
You might want to do something a bit more targeted, but that's
details.

>
> And then use it in your XDP program to parse packets meta data passed
> from this specific driver. ( i mean no real parsing is required, you
> just point to the meta data buffer with the metadata btf formatted C
> strucuter).
>
>
> >
> > If you are asking about support for using BTF_ID_LIST() macro in a
> >
> > kernel module, then right now we don't call resolve_btfids on
> > modules,
> >
> > so it's not supported there yet. It's trivial to add, but we'll
> >
> > probably need to teach resolve_btfids to understand split BTF. We can
> >
> > do that separately after the basic "infra" lands, though.
>
>