Is BTF info sufficient enough for BPFTrace and other debug tools to run ?

Is BTF info sufficient enough for BPFTrace and other debug tools to run ?

[Johnny young]

Hello BPF

I understand that CONFIG_DEBUG_INFO_BTF=y will generate .BTF and
.BTF_xx  sections in the kernel image which are much smaller than
those DWARF sections.  But I also try to understand how BTF can impact
bpftrace and the existing debug tools:

1) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, can
bpftrace relies on BTF only without kernel_devel ?

2) Can the existing kernel debugging tools like crash(1) or
kgdb(1) take advantage of BTF ?

3) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, are the
symbolic info and types info in the debug-info section replaced with
BTF formatted info?

4) Given the current upstream development effort for BTF, can we run
bpftrace without LLVM now ? and can we run bpftrace without the help
of kernel header files (kernel-devel) ?

5) Has bpf CO-RE become reality now?

Thank you!
Johnny

Re: Is BTF info sufficient enough for BPFTrace and other debug tools to run ?

[Stephen Brennan]

Johnny young <johnny96.young@gmail.com> writes:
> Hello BPF
>
> I understand that CONFIG_DEBUG_INFO_BTF=y will generate .BTF and
> .BTF_xx  sections in the kernel image which are much smaller than
> those DWARF sections.  But I also try to understand how BTF can impact
> bpftrace and the existing debug tools:
>
> 1) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, can
> bpftrace relies on BTF only without kernel_devel ?
>
> 2) Can the existing kernel debugging tools like crash(1) or
> kgdb(1) take advantage of BTF ?

Hi Johnny,

I can't answer all your questions but I can chime in regarding
debuggers. BTF could provide information to both of these debuggers, but
currently neither of them know how to use it. There's a few reasons and
challenges.

1. Kernel and module BTF today only includes the type information
   necessary to describe all functions, and percpu variables. Most
   debuggers would also like to know the types of global variables. I've
   got a patch series [1] which allows pahole (which generates the
   kernel & module BTF) to output information for global variable
   declarations as well.

2. Assuming you had the BTF, you'd also need a symbol table.  The kernel
   has kallsyms, which is an internal symbol table, and most debuggers
   today don't know how to read that - instead, they rely on the symbol
   table from the ELF debuginfo file. I frequently work with a debugger
   library called drgn [2], and I've got a branch out for review [3]
   which enables drgn to read the kallsyms. In order to do that,
   debuggers need to be able to *find* the kallsyms table, so I added
   information to the vmcoreinfo note in the commits 5fd8fea935a1
   ("vmcoreinfo: include kallsyms symbols") and f09bddbd8661
   ("vmcoreinfo: add kallsyms_num_syms symbol").

3. Assuming you have symbol table access and the BTF is complete enough
   for you to use it for real debugging, then your debugger still needs
   to have logic to *find* the BTF and parse it (probably with libbpf).
   I've got a branch for drgn [4] which implements BTF parsing on top of
   the kallsyms parsing logic. With those things together, you get a
   debugger which relies on nothing except data it finds inside the
   kernel, and it works well enough.

So the short answer is - no, currently there is no debugger (that I know
of) which can leverage BTF. But it's in the works for drgn, and once we
get there with drgn, I'd hope to see that developers for other debuggers
might see the power of it and consider implementing it.

[1]: https://lore.kernel.org/bpf/20220826184911.168442-1-stephen.s.brennan@oracle.com/
[2]: https://drgn.readthedocs.io/
[3]: https://github.com/osandov/drgn/pull/177
[4]: https://github.com/brenns10/drgn/tree/kallsyms_plus_btf

>
> 3) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, are the
> symbolic info and types info in the debug-info section replaced with
> BTF formatted info?

No, BTF is generated _in addition to_ the existing type information. BTF
doesn't store "symbolic info" i.e. symbol table data, so it couldn't
replace that data. It only stores type information, so hypothetically it
could be used instead of DWARF .debug_types data, but in practice that
doesn't happen.

Stephen

>
> 4) Given the current upstream development effort for BTF, can we run
> bpftrace without LLVM now ? and can we run bpftrace without the help
> of kernel header files (kernel-devel) ?
>
> 5) Has bpf CO-RE become reality now?
>
> Thank you!
> Johnny

Re: Is BTF info sufficient enough for BPFTrace and other debug tools to run ?

[Johnny young]

Hey Stephen,

Thank you for the detailed explanation. I hope that after your work is
done for drgn as mentioned and the necessary work is done from the
debug tool side,  then the debug tools will rely on the kernel
internal symbol table and BTF only in the future.  That's wonderful.

Best,

On Thu, Sep 29, 2022 at 2:35 PM Stephen Brennan
<stephen.s.brennan@oracle.com> wrote:
>
> Johnny young <johnny96.young@gmail.com> writes:
> > Hello BPF
> >
> > I understand that CONFIG_DEBUG_INFO_BTF=y will generate .BTF and
> > .BTF_xx  sections in the kernel image which are much smaller than
> > those DWARF sections.  But I also try to understand how BTF can impact
> > bpftrace and the existing debug tools:
> >
> > 1) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, can
> > bpftrace relies on BTF only without kernel_devel ?
> >
> > 2) Can the existing kernel debugging tools like crash(1) or
> > kgdb(1) take advantage of BTF ?
>
> Hi Johnny,
>
> I can't answer all your questions but I can chime in regarding
> debuggers. BTF could provide information to both of these debuggers, but
> currently neither of them know how to use it. There's a few reasons and
> challenges.
>
> 1. Kernel and module BTF today only includes the type information
>    necessary to describe all functions, and percpu variables. Most
>    debuggers would also like to know the types of global variables. I've
>    got a patch series [1] which allows pahole (which generates the
>    kernel & module BTF) to output information for global variable
>    declarations as well.
>
> 2. Assuming you had the BTF, you'd also need a symbol table.  The kernel
>    has kallsyms, which is an internal symbol table, and most debuggers
>    today don't know how to read that - instead, they rely on the symbol
>    table from the ELF debuginfo file. I frequently work with a debugger
>    library called drgn [2], and I've got a branch out for review [3]
>    which enables drgn to read the kallsyms. In order to do that,
>    debuggers need to be able to *find* the kallsyms table, so I added
>    information to the vmcoreinfo note in the commits 5fd8fea935a1
>    ("vmcoreinfo: include kallsyms symbols") and f09bddbd8661
>    ("vmcoreinfo: add kallsyms_num_syms symbol").
>
> 3. Assuming you have symbol table access and the BTF is complete enough
>    for you to use it for real debugging, then your debugger still needs
>    to have logic to *find* the BTF and parse it (probably with libbpf).
>    I've got a branch for drgn [4] which implements BTF parsing on top of
>    the kallsyms parsing logic. With those things together, you get a
>    debugger which relies on nothing except data it finds inside the
>    kernel, and it works well enough.
>
> So the short answer is - no, currently there is no debugger (that I know
> of) which can leverage BTF. But it's in the works for drgn, and once we
> get there with drgn, I'd hope to see that developers for other debuggers
> might see the power of it and consider implementing it.
>
> [1]: https://lore.kernel.org/bpf/20220826184911.168442-1-stephen.s.brennan@oracle.com/
> [2]: https://drgn.readthedocs.io/
> [3]: https://github.com/osandov/drgn/pull/177
> [4]: https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
>
> >
> > 3) If the kernel is built with CONFIG_DEBUG_INFO_BTF=y, are the
> > symbolic info and types info in the debug-info section replaced with
> > BTF formatted info?
>
> No, BTF is generated _in addition to_ the existing type information. BTF
> doesn't store "symbolic info" i.e. symbol table data, so it couldn't
> replace that data. It only stores type information, so hypothetically it
> could be used instead of DWARF .debug_types data, but in practice that
> doesn't happen.
>
> Stephen
>
> >
> > 4) Given the current upstream development effort for BTF, can we run
> > bpftrace without LLVM now ? and can we run bpftrace without the help
> > of kernel header files (kernel-devel) ?
> >
> > 5) Has bpf CO-RE become reality now?
> >
> > Thank you!
> > Johnny