[PATCH v2 bpf-next 0/9] bpf: support BTF kind layout info, CRCs

[Alan Maguire <alan.maguire@oracle.com>]

By separating parsing BTF from using all the information
it provides, we allow BTF to encode new features even if
they cannot be used.  This is helpful in particular for
cases where newer tools for BTF generation run on an
older kernel; BTF kinds may be present that the kernel
cannot yet use, but at least it can parse the BTF
provided.  Meanwhile userspace tools with newer libbpf
may be able to use the newer information.

The intent is to support encoding of kind layouts
optionally so that tools like pahole can add this
information.  So for each kind we record

- kind-related flags
- length of singular element following struct btf_type
- length of each of the btf_vlen() elements following

In addition we make space in the BTF header for
CRC32s computed over the BTF along with a CRC for
the base BTF; this allows split BTF to identify
a mismatch explicitly.

The ideas here were discussed at [1], with further
discussion at [2].

Future work can take more advantage of these features
such as

- using base CRC to identify base/module BTF mismatch
  explicitly
- using absence of a base BTF CRC as evidence that
  BTF is standalone

...and new BTF kind addition should present less
trouble, provided the kinds are optional.  BTF
parsing _will_ still fail if a non-optional
kind is encountered, as the assumption is that
such kinds are needed.  To take a few examples,
the tag kinds are optional, however enum64
is required, so BTF containing an enum64
(that did not know about enum64) would be
rejected.  This makes sense as if for example
a struct contained an enum64 we would not
be able to fully represent that struct unless
we knew about enum64s.

Patch 1 is the UAPI changes, patches 2-3 provide
libbpf support for handling and using kind layouts.
Patch 4 adds kernel support for handling and using
kind layouts.  Patch 5 adds libbpf support to add
kind layouts and CRCs based on options passed
to btf__new_empty_opts().   Patch 6 updates
pahole flags for kernel/module BTF generation
to generate kind layout/CRCs if that support
is available.  Patch 7 adds bpftool support
to dump BTF metadata (header + kind layout).
Patch 8 documents this.  Patch 9 is a set
of selftests covering encoding/decoding of
BTF kind layout information, and patch 10
is a dwarves patch to add kind layout encoding
and CRC support via btf__new_empty_opts().

Changes since RFC

- Terminology change from meta -> kind_layout
  (Alexei and Andrii)
- Simplify representation, removing meta header
  and just having kind layout section (Alexei)
- Fixed bpftool to have JSON support, support
  prefix match, documented changes (Quentin)
- Separated metadata opts into add_kind_layout
  and add_crc
- Added additional positive/negative tests
  to cover basic unknown kind, one with an
  info_sz object following it and one with
  N elem_sz elements following it.
- Updated pahole-flags to use help output
  rather than version to see if features
  are present

[1] https://lore.kernel.org/bpf/CAEf4BzYjWHRdNNw4B=eOXOs_ONrDwrgX4bn=Nuc1g8JPFC34MA@mail.gmail.com/
[2] https://lore.kernel.org/bpf/20230531201936.1992188-1-alan.maguire@oracle.com/

Alan Maguire (9):
  btf: add kind layout encoding, crcs to UAPI
  libbpf: support handling of kind layout section in BTF
  libbpf: use kind layout to compute an unknown kind size
  btf: support kernel parsing of BTF with kind layout
  libbpf: add kind layout encoding, crc support
  btf: generate BTF kind layout for vmlinux/module BTF
  bpftool: add BTF dump "format meta" to dump header/metadata
  bpftool: update doc to describe bpftool btf dump .. format meta
  selftests/bpf: test kind encoding/decoding

 include/uapi/linux/btf.h                      |  24 ++
 kernel/bpf/btf.c                              | 102 +++++--
 scripts/pahole-flags.sh                       |   7 +
 .../bpf/bpftool/Documentation/bpftool-btf.rst |  16 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   2 +-
 tools/bpf/bpftool/btf.c                       |  93 +++++-
 tools/include/uapi/linux/btf.h                |  24 ++
 tools/lib/bpf/btf.c                           | 272 +++++++++++++++---
 tools/lib/bpf/btf.h                           |  11 +
 tools/lib/bpf/libbpf.map                      |   1 +
 .../selftests/bpf/prog_tests/btf_kind.c       | 187 ++++++++++++
 11 files changed, 670 insertions(+), 69 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_kind.c

-- 
2.39.3