Re: Question: missing vmlinux BTF variable declarations

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Tue, May 3, 2022 at 3:32 PM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On Tue, 3 May 2022, Stephen Brennan wrote:
>
> > >> Ideally we structure BTFs as a multi level tree.  Where BTF with
> > >> global vars and other non essential BTF info can be added to vmlinux
> > >> BTF at run-time. BTF of kernel mods can add on top and mods can have
> > >> split BTF too.
> >
> > I see what you mean. It does sound a bit frustrating to have an
> > additional BTF module to augment every external module, which would be
> > the third level of that tree.
> >
> > We'll need to allocate more module structs and pages within the kernel
> > for that data, I wonder whether it would be cheaper for the
> > "non-essential" module BTF to just reside in the same BTF section of
> > that module.
> >
> > I suppose I can run my modified pahole on some sample modules and see
> > the BTF size difference, rather than just speculating, I'll do that in a
> > follow-up here.
> >
> > > Yeah, reuses existing mechanizm, doesn't increase the kernel BTF
> > > footprint by default, allows for debuggers, profilers, tracers, etc to
> > > ask for extra info in the form of just loading btf_global_variables.ko.
> >
> > I agree, this is a quite elegant solution. Though it'll require a fair
> > bit of work to achieve, I do think it's important to keep the footprint
> > down. One thing I'd like to see in this world is a way to instruct the
> > kernel that "I always want the non-essential BTF loaded", maybe via
> > cmdline or sysctl. This way, the module loader would know to search for
> > "$MODNAME-btf" for each module which doesn't end with "-btf".
> >
>
> Hmm, could we just have a tristate CONFIG_DEBUG_INFO_BTF_EXTRA?
> If set to y, the extra vars are builtin to vmlinux BTF and
> modules, and if set to m, they reside in /sys/kernel/btf/vmlinux-btf-extra
> courtesy of the vmlinux-btf-extra.ko module (or whatever naming
> scheme makes sense). Looks like pahole already has an option to
> store encoded BTF elsewhere:
>
> --btf_encode_detached=FILENAME
>
> ...so maybe all we need is something like --btf_gen_var_only for
> the case where we build the btf-extra module
>
> pahole -J --btf_base vmlinux  --btf_gen_var_only
> --btf_encode_detached=vmlinux_btf_extra
>
> ?
>

So BTF dedup would take care of keeping only extra DATASEC in such
module while reusing all the types from vmlinux BTF, as long as the
module itself has all the vmlinux BTF types plus those variables. It's
just a question of having ability to enable/disable global variables
generation. Which honestly is not a bad idea in general to have
overall more or less granular control over which subsets of BTF pahole
should emit.

> That's still only 2-way split BTF (base vmlinux BTF
> plus vmlinux variables); we'd only need the
> three-way split for the case where modules use the
> -extra approach too, and I'd wonder about the viability
> of having an -extra BTF module for each module, especially
> if space-saving is the goal.
>

Yeah, it feels like having that for modules is taking this to another
level of complexity, while adding this for vmlinux only seems pretty
doable with minimal changes (we don't need any extra BTF functionality
as we will just leave current start-based topology with vmlinux BTF in
the center; it's only Kbuild/Makefile modifications). I also wonder if
it will allow saving much for modules, they are probably not having
that many global variables anyways and it's acceptable to have them in
module's BTF. We can also separately control VMLINUX_BTF_EXTRAS as
y/n/m and MODULE_BTF_EXTRAS as y/n (that is, none or built into the
module itself), for starters at least.

> Alan