Re: Question: missing vmlinux BTF variable declarations

From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Stephen Brennan <stephen.s.brennan@oracle.com>,
	Andrii Nakryiko <andrii.nakryiko@gmail.com>,
	Yonghong Song <yhs@fb.com>, Shung-Hsi Yu <shung-hsi.yu@suse.com>,
	bpf <bpf@vger.kernel.org>, Omar Sandoval <osandov@osandov.com>,
	Arnaldo Carvalho de Melo <acme@redhat.com>
Subject: Re: Question: missing vmlinux BTF variable declarations
Date: Tue, 3 May 2022 11:39:15 -0300	[thread overview]
Message-ID: <YnE+k33iUtLH7Lks@kernel.org> (raw)
In-Reply-To: <CAADnVQ+YuxB8gZGjx+RP=04z4SgYEmPjEjDa_=Q6HmUecxK8QQ@mail.gmail.com>

Em Fri, Apr 29, 2022 at 10:10:01AM -0700, Alexei Starovoitov escreveu:
> On Wed, Apr 27, 2022 at 11:43 AM Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
> > [2]: https://github.com/brenns10/drgn/tree/kallsyms_plus_btf

> > Combining these three things, I've got a debugger which can open up a
> > vmcore _without DWARF debuginfo_ and allow you to print out typed
> > variable values. It just relies on BTF + kallsyms.

> > So the proof of concept is proven, and I'm quite excited about it!

> Exciting indeed. This is pretty cool.

Indeed!

> I'm afraid we cannot justify 2.5 Mb kernel memory increase for pure
> debugging. The existing vmlinux BTF is used by the kernel itself to
> validate bpf prog access.  bpf progs cannot access normal global vars.
> If/when they are we can reconsider.

> As an alternative path I think we could introduce hierarchical
> split BTF.

Which we already have in the form of BTF for modules that use vmlinux as
a base for common types.

> Currently vmlinux BTF and BTF of kernel modules is a tree
> of depth 2.

> We can keep such representation of BTFs and introduce a fake kernel
> module that contains kernel global vars.

pahole would generate a naked BTF just with variables and types not
present in the main vmlinux BTF and refer to it for all the other types.

> drgn can parse vmlinux BTF plus BTFs of all ko-s including fake one
> and obtain the same amount of debug info as if global vars
> were part of vmlinux BTF.

Right.

> Consuming 2.5Mb on demand via ko would be acceptable in some scenarios
> whereas unconditionally burning that much memory in vmlinux BTF (even
> optional via kconfig) is probably not.

And since it would be just an extra kernel module, the existing
packaging processes (in distros, embedded systems, etc) that care about
BTF would carry this without any modification to existing practices,
i.e. selecting CONFIG_DEBUG_INFO_BTF=y would bt default enable
CONFIG_DEBUG_INFO_GLOBAL_VARIABLES_BTF=y, which could be optionally
disabled by people not wanting to carry this extra info.

I.e. it would be always available but not always loaded.

> Ideally we structure BTFs as a multi level tree.  Where BTF with
> global vars and other non essential BTF info can be added to vmlinux
> BTF at run-time. BTF of kernel mods can add on top and mods can have
> split BTF too.

Yeah, reuses existing mechanizm, doesn't increase the kernel BTF
footprint by default, allows for debuggers, profilers, tracers, etc to
ask for extra info in the form of just loading btf_global_variables.ko.

- Arnaldo