[PATCH v2 dwarves, kernel, libbpf 0/9] Add support for generating BTF for all variables

[Stephen Brennan <stephen.s.brennan@oracle.com>]

Hi all,

It's been a few weeks since the last update to my prior thread [1] on this
topic. My apologies on that: Alan hunted down some BTF deduplicator bugs and he
found the root cause and got it fixed [2]. We went ahead and respun the dwarves
patches on top of the latest version, and Alan also created a kernel patch to
implement the tristate for BTF variables. I tested the full series and did a
size comparison which I'll share below.

[1] https://lore.kernel.org/bpf/20220826184911.168442-1-stephen.s.brennan@oracle.com/
[2] https://lore.kernel.org/bpf/1666622309-22289-1-git-send-email-alan.maguire@oracle.com/

To remind folks what the series is all about, BTF currently contains type
information for functions and percpu variables, but not for global variables.
Debuggers would find BTF quite useful as a source of type information, but they
would need that global variable data, as users tend to look more at data types
than function types. A major advantage of BTF is that it is compact, and
built-in to the kernel. Assuming that you can find the kallsyms symbol table
(which is possible via the vmcoreinfo note since 6.0), then you can locate BTF
data. With the kallsyms table and the type info, you have enough information to
enable reasonably user-friendly debugging. There are proof-of-concept patches
for drgn (a Python-based scriptable debugger) to leverage all this
functionality to debug a core dump without any external debug info.

So, this patch series is a re-roll & rebase of the prior one which was just for
dwarves/pahole. It includes three components:

(1) Alan's libbpf fix, just for reference
(2) The dwarves patches to add support for generating variable BTF
(3) The kernel patch adding the variable generation as a tristate

The only new portion is 3, I believe. But all of this should be a complete
product. I used this complete product to build a small-ish upstream kernel
configuration, and I measured the size of the .BTF sections. Here's the result:

Vars built-in:  0x7d3ad1  8,207,057 bytes
Vars in module: 0x62af5f  6,467,423 bytes
 -> module BTF: 0x1a8e66  1,740,390 bytes   (combined size 8,207,813)
Vars disabled:  0x62e90b  6,482,187 bytes

Sorry, I don't have a combined diffstat for the files since these patches are
both for dwarves and kernel repos. But here's the listing of patches:

dwarves: Stephen Brennan
  [1/9] dutil: return ELF section name when looked up by index
  [2/9] btf_encoder: Rename percpu structures to variables
  [3/9] btf_encoder: cache all ELF section info
  [4/9] btf_encoder: make the variable array dynamic
  [5/9] btf_encoder: record ELF section for collected variables
  [6/9] btf_encoder: collect all variables
  [7/9] btf_encoder: allow encoding all variables

libbpf: Alan Maguire
  [8/9] libbpf: btf dedup identical struct test needs check for nested structs/arrays
  (** note, this is already merged, but included for completeness **)

kernel: Alan Maguire
  [9/9] bpf: add support for CONFIG_DEBUG_INFO_BTF_VARS

Thanks for your consideration!
Stephen