status of llvm BTF support

status of llvm BTF support

[Yonghong Song]

Hi,

As many of you already knew, latest bpf-next/net-next
kernel added supports for func_info and line_info.
   https://www.mail-archive.com/netdev@vger.kernel.org/msg255580.html
   https://www.mail-archive.com/netdev@vger.kernel.org/msg257914.html
   https://www.mail-archive.com/netdev@vger.kernel.org/msg272974.html

The func info and line info enables better ksym, correlation of
xlated/jited code with source code, and better source-annotated
verifier log. perf will be enhanced later to pick up
such debug info so better information can be displayed for hot
bpf programs.

Patch https://reviews.llvm.org/D53736
=====================================

Currently, the func info and line info are generated by LLVM
unmerged patch https://reviews.llvm.org/D53736. As stated in
the commit message of D53726, the main advantage of implementing
in LLVM is:
   . better integration/deployment as no extra tools are needed.
   . bpf JIT based compilation (like bcc, bpftrace, etc.) can get
     BTF without much extra effort.
   . BTF line_info needs selective source codes, which can be
     easily retrieved when inside the compiler.

Not in the above list is the future work related to
compile once and run everywhere (https://lwn.net/Articles/773198/),
which needs strong type. Such strong type is only available
in llvm IR. So implementation will have to be in llvm as well.

The llvm patch D53736 has been submitted on Oct 25, 2018.
The patch implemented BTF as a first class debug info, meaning
in a similar way to that Dwarf or CodeView debug info is generated.
The implementation is mainly placed under
   llvm/lib/CodeGen/AsmPrinter
directory. Since BTF currently is only used by BPF, cmake
file is changed to allow compilation of main BTFDebug.c only
if BPF is in the llvm build target list. Otherwise, a
stub implementation is used.

After initial review by Adrian Prantl where all comments
are addressed, the patch became dormand since Nov 16, except some
minor stylish review by Fangrui Song.

Patch https://reviews.llvm.org/{D55755,D55756,D55752}
=====================================================

During review of D53736, on Nov 16, I commented it was possible
to move main file BTFDebug.cpp to lib/Target/BPF
directory if moving some DebugInfo private headers
to llvm:include/llvm/CodeGen directory. But I never
got an answer whether this is a good idea or not.

Since review has dragged so long, to help facilitate discussion,
last weekend, I went ahead and implemented this scheme
with three patches in phabricator:
   https://reviews.llvm.org/D55755: refactor to move three DebugInfo
      private files into llvm:include/llvm/CodeGen directory.
   https://reviews.llvm.org/D55756: expose some private AsmPrinter
      members to protected so target extended AsmPrinter can use
      it to add their own DebugInfo handlers.
   https://reviews.llvm.org/D55752
      everything else is inside BPF directory.

This approach kept BTF completely under lib/Target/BPF
if patches D55755 and D55756 are implemented to enable
target specific debug info handlers.

Recap of two Approaches
=======================

To recap,
   https://reviews.llvm.org/D53736:
      everything in the llvm core, BTF is a "first-class" debug info.
      lib/Target/BPF unchanged.
   https://reviews.llvm.org/{D55755,D55756,D55752}:
      Some changes related DebugInfo to allow target specific
      debug info handlers.
      Major changes will be under /lib/Target/BPF.
At this point, it is not clear which approach will be accepted
by the community.

Going Forward
=============

We will continue to engage with LLVM community for integrating
BTF generation. Any help we can get from you should be helpful
and will be really appreciated.

Thanks,

Yonghong

Re: status of llvm BTF support

[Edward Cree]

On 17/12/18 22:45, Yonghong Song wrote:
> Hi,
>
> As many of you already knew, latest bpf-next/net-next
> kernel added supports for func_info and line_info.
Is there a proper spec yet, against which other implementations can
 be developed?  Or do folks still have to RTFS and reverse-engineer
 the semantics if they want to support this in tools other than
 LLVM and pahole?
Because IMHO writing a spec should be a higher priority than
 anything else in BTF development; and every patch series that
 changes the format should come with a patch to update the spec.
We're developing a format here, and acting like we don't need to
 do things properly because we 'own' both ends — which is not
 conducive to building an open ecosystem around it.

-Ed

Re: status of llvm BTF support

[Yonghong Song]

On 12/18/18 8:18 AM, Edward Cree wrote:
> On 17/12/18 22:45, Yonghong Song wrote:
>> Hi,
>>
>> As many of you already knew, latest bpf-next/net-next
>> kernel added supports for func_info and line_info.
> Is there a proper spec yet, against which other implementations can
>   be developed?  Or do folks still have to RTFS and reverse-engineer
>   the semantics if they want to support this in tools other than
>   LLVM and pahole?

Ed,

We are working on this and will send a patch as soon as it is ready.
I totally understand the the spec is very important now.
Totally agree with your assessment below.

Thanks,

Yonghong

> Because IMHO writing a spec should be a higher priority than
>   anything else in BTF development; and every patch series that
>   changes the format should come with a patch to update the spec.
> We're developing a format here, and acting like we don't need to
>   do things properly because we 'own' both ends — which is not
>   conducive to building an open ecosystem around it.
> 
> -Ed
> 

update on llvm BTF support

[Yonghong Song]

Hi,

This is an update of llvm BTF support.

The patches corresponding to 
https://reviews.llvm.org/{D55755,D55756,D55752} have been accepted and 
merged.
The llvm BTF support will be in llvm 8.0 whose release
time is around next March.

So with latest llvm/clang trunk, add -g to your compilation
flag, .BTF and .BTF.ext will be generated.
If you are curious about encoding, -S will produce an
assembly code including .BTF/.BTF.ext.

Thanks!

Yonghong

On 12/17/18 2:45 PM, Yonghong Song wrote:
> Hi,
> 
> As many of you already knew, latest bpf-next/net-next
> kernel added supports for func_info and line_info.
>     https://www.mail-archive.com/netdev@vger.kernel.org/msg255580.html
>     https://www.mail-archive.com/netdev@vger.kernel.org/msg257914.html
>     https://www.mail-archive.com/netdev@vger.kernel.org/msg272974.html
> 
> The func info and line info enables better ksym, correlation of
> xlated/jited code with source code, and better source-annotated
> verifier log. perf will be enhanced later to pick up
> such debug info so better information can be displayed for hot
> bpf programs.
> 
> Patch https://reviews.llvm.org/D53736
> =====================================
> 
> Currently, the func info and line info are generated by LLVM
> unmerged patch https://reviews.llvm.org/D53736. As stated in
> the commit message of D53726, the main advantage of implementing
> in LLVM is:
>     . better integration/deployment as no extra tools are needed.
>     . bpf JIT based compilation (like bcc, bpftrace, etc.) can get
>       BTF without much extra effort.
>     . BTF line_info needs selective source codes, which can be
>       easily retrieved when inside the compiler.
> 
> Not in the above list is the future work related to
> compile once and run everywhere (https://lwn.net/Articles/773198/),
> which needs strong type. Such strong type is only available
> in llvm IR. So implementation will have to be in llvm as well.
> 
> The llvm patch D53736 has been submitted on Oct 25, 2018.
> The patch implemented BTF as a first class debug info, meaning
> in a similar way to that Dwarf or CodeView debug info is generated.
> The implementation is mainly placed under
>     llvm/lib/CodeGen/AsmPrinter
> directory. Since BTF currently is only used by BPF, cmake
> file is changed to allow compilation of main BTFDebug.c only
> if BPF is in the llvm build target list. Otherwise, a
> stub implementation is used.
> 
> After initial review by Adrian Prantl where all comments
> are addressed, the patch became dormand since Nov 16, except some
> minor stylish review by Fangrui Song.
> 
> Patch https://reviews.llvm.org/{D55755,D55756,D55752}
> =====================================================
> 
> During review of D53736, on Nov 16, I commented it was possible
> to move main file BTFDebug.cpp to lib/Target/BPF
> directory if moving some DebugInfo private headers
> to llvm:include/llvm/CodeGen directory. But I never
> got an answer whether this is a good idea or not.
> 
> Since review has dragged so long, to help facilitate discussion,
> last weekend, I went ahead and implemented this scheme
> with three patches in phabricator:
>     https://reviews.llvm.org/D55755: refactor to move three DebugInfo
>        private files into llvm:include/llvm/CodeGen directory.
>     https://reviews.llvm.org/D55756: expose some private AsmPrinter
>        members to protected so target extended AsmPrinter can use
>        it to add their own DebugInfo handlers.
>     https://reviews.llvm.org/D55752
>        everything else is inside BPF directory.
> 
> This approach kept BTF completely under lib/Target/BPF
> if patches D55755 and D55756 are implemented to enable
> target specific debug info handlers.
> 
> Recap of two Approaches
> =======================
> 
> To recap,
>     https://reviews.llvm.org/D53736:
>        everything in the llvm core, BTF is a "first-class" debug info.
>        lib/Target/BPF unchanged.
>     https://reviews.llvm.org/{D55755,D55756,D55752}:
>        Some changes related DebugInfo to allow target specific
>        debug info handlers.
>        Major changes will be under /lib/Target/BPF.
> At this point, it is not clear which approach will be accepted
> by the community.
> 
> Going Forward
> =============
> 
> We will continue to engage with LLVM community for integrating
> BTF generation. Any help we can get from you should be helpful
> and will be really appreciated.
> 
> Thanks,
> 
> Yonghong
>