[PATCH bpf-next 00/13] bpf: add btf func info support

* [PATCH bpf-next 00/13] bpf: add btf func info support
@ 2018-10-12 18:54 Yonghong Song
  2018-10-12 18:54 ` [PATCH bpf-next 01/13] bpf: btf: Break up btf_type_is_void() Yonghong Song
                   ` (4 more replies)
  0 siblings, 5 replies; 12+ messages in thread
From: Yonghong Song @ 2018-10-12 18:54 UTC (permalink / raw)
  To: ast, kafai, daniel, netdev; +Cc: kernel-team

The BTF support was added to kernel by Commit 69b693f0aefa
("bpf: btf: Introduce BPF Type Format (BTF)"), which introduced
.BTF section into ELF file and is primarily
used for map pretty print.
pahole is used to convert dwarf to BTF for ELF files.

The next step would be add func type info and debug line info
into BTF. For debug line info, it is desirable to encode
source code directly in the BTF to ease deployment and
introspection.

The func type and debug line info are relative to byte code offset.
Also since byte codes may need to be relocated by the loader,
func info and line info are placed in a different section,
.BTF.ext, so the loader could manupilate it according to how
byte codes are placed, before loading into the kernel.

LLVM commit https://reviews.llvm.org/rL344366 (in llvm trunk)
now can generate type/func/line info.
For the below example, with a llvm compiler built with Debug mode,
the following is generated:

  -bash-4.2$ cat test.c
  int foo(int (*bar)(int)) { return bar(5); }
  -bash-4.2$ clang -target bpf -g -O2 -mllvm -debug-only=btf -c test.c
  Type Table:
  [1] FUNC name_off=1 info=0x0c000001 size/type=2
        param_type=3
  [2] INT name_off=11 info=0x01000000 size/type=4
        desc=0x01000020
  [3] PTR name_off=0 info=0x02000000 size/type=4
  [4] FUNC_PROTO name_off=0 info=0x0d000001 size/type=2
        param_type=2

  String Table:
  0 : 
  1 : foo
  5 : .text
  11 : int
  15 : test.c
  22 : int foo(int (*bar)(int)) { return bar(5); }

  FuncInfo Table:
  sec_name_off=5
        insn_offset=<Omitted> type_id=1

  LineInfo Table:
  sec_name_off=5
        insn_offset=<Omitted> file_name_off=15 line_off=22 line_num=1 column_num=0
        insn_offset=<Omitted> file_name_off=15 line_off=22 line_num=1 column_num=35
        insn_offset=<Omitted> file_name_off=15 line_off=22 line_num=1 column_num=28

In the above, type and string tables are in .BTF section, and
func and line info tables in .BTF.ext. The "<Omitted>" is the
insn offset which is not available during the dump time but
resolved during later compilation process.
Following the format specification at Patch #9 and examine the
raw data in .BTF.ext section, we have
  FuncInfo Table:
  sec_name_off=5
        insn_offset=0 type_id=1

  LineInfo Table:
  sec_name_off=5
        insn_offset=0  file_name_off=15 line_off=22 line_num=1 column_num=0
        insn_offset=8  file_name_off=15 line_off=22 line_num=1 column_num=35
        insn_offset=24 file_name_off=15 line_off=22 line_num=1 column_num=28
In the above insn_offset is the byte offset.

With this support, better ksym for bpf programs and functions can be
generated. Below is a demonstration from Patch #13.
  $ bpftool prog dump jited id 1
  int _dummy_tracepoint(struct dummy_tracepoint_args * ):
  bpf_prog_b07ccb89267cf242__dummy_tracepoint:
         0:   push   %rbp
         1:   mov    %rsp,%rbp
        ......
        3c:   add    $0x28,%rbp
        40:   leaveq
        41:   retq

  int test_long_fname_1(struct dummy_tracepoint_args * ):
  bpf_prog_2dcecc18072623fc_test_long_fname_1:
         0:   push   %rbp
         1:   mov    %rsp,%rbp
        ......
        3a:   add    $0x28,%rbp
        3e:   leaveq
        3f:   retq

  int test_long_fname_2(struct dummy_tracepoint_args * ):
  bpf_prog_89d64e4abf0f0126_test_long_fname_2:
         0:   push   %rbp
         1:   mov    %rsp,%rbp
        ......
        80:   add    $0x28,%rbp
        84:   leaveq
        85:   retq

For the patchset,
Patch #1  refactors the code to break up btf_type_is_void().
Patch #2  introduces new BTF types BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO.
Patch #3  syncs btf.h header to tools directory.
Patch #4  adds btf func/func_proto self tests in test_btf.
Patch #5  adds kernel interface to load func_info to kernel
          and pass func_info to userspace.
Patch #6  syncs bpf.h header to tools directory.
Patch #7  adds news btf/func_info related fields in libbpf
          program load function.
Patch #8  extends selftest test_btf to test load/retrieve func_type info.
Patch #9  adds .BTF.ext func info support.
Patch #10 changes Makefile to avoid using pahole if llvm is capable of
          generating BTF sections.
Patch #11 refactors to have btf_get_from_id() in libbpf for reuse.
Patch #12 enhance test_btf file testing to test func info.
Patch #13 adds bpftool support for func signature dump.

Yonghong Song (13):
  bpf: btf: Break up btf_type_is_void()
  bpf: btf: Add BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO
  tools/bpf: sync kernel btf.h header
  tools/bpf: add btf func/func_proto unit tests in selftest test_btf
  bpf: get better bpf_prog ksyms based on btf func type_id
  tools/bpf: sync kernel uapi bpf.h header to tools directory
  tools/bpf: add new fields for program load in lib/bpf
  tools/bpf: extends test_btf to test load/retrieve func_type info
  tools/bpf: add support to read .BTF.ext sections
  tools/bpf: do not use pahole if clang/llvm can generate BTF sections
  tools/bpf: refactor to implement btf_get_from_id() in lib/bpf
  tools/bpf: enhance test_btf file testing to test func info
  tools/bpf: bpftool: add support for jited func types

 include/linux/bpf.h                          |   2 +
 include/linux/bpf_verifier.h                 |   1 +
 include/linux/btf.h                          |   2 +
 include/uapi/linux/bpf.h                     |  11 +
 include/uapi/linux/btf.h                     |   9 +-
 kernel/bpf/btf.c                             | 322 +++++++++--
 kernel/bpf/core.c                            |   9 +
 kernel/bpf/syscall.c                         |  86 ++-
 kernel/bpf/verifier.c                        |  50 ++
 samples/bpf/Makefile                         |   8 +
 tools/bpf/bpftool/btf_dumper.c               |  96 ++++
 tools/bpf/bpftool/main.h                     |   2 +
 tools/bpf/bpftool/map.c                      |  68 +--
 tools/bpf/bpftool/prog.c                     |  54 ++
 tools/include/uapi/linux/bpf.h               |  11 +
 tools/include/uapi/linux/btf.h               |   9 +-
 tools/lib/bpf/bpf.c                          |   3 +
 tools/lib/bpf/bpf.h                          |   3 +
 tools/lib/bpf/btf.c                          | 305 ++++++++++
 tools/lib/bpf/btf.h                          |  32 ++
 tools/lib/bpf/libbpf.c                       |  53 +-
 tools/testing/selftests/bpf/Makefile         |   8 +
 tools/testing/selftests/bpf/test_btf.c       | 566 ++++++++++++++++++-
 tools/testing/selftests/bpf/test_btf_haskv.c |  16 +-
 tools/testing/selftests/bpf/test_btf_nokv.c  |  16 +-
 25 files changed, 1613 insertions(+), 129 deletions(-)

-- 
2.17.1

^ permalink raw reply	[flat|nested] 12+ messages in thread