Where can I find the map's BTF type key/value specification?

Where can I find the map's BTF type key/value specification?

[Daniel T. Lee]

I'm looking for how BTF type definition '__type(key, int)' is being changed
to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
it can be considered the same)

    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, int);          => __uint(key_size, sizeof(int))
    __type(value, u32);    => __uint(value_size, sizeof(u32))
    __uint(max_entries, 2);

Whether the specific map type supports BTF or not can be inferred from
the file in kernel/bpf/*map.c and by checking each MAP type's
bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.

But how can I figure out that specific types of map support BTF types for
key/value? And how can I determine how this BTF key/value type is
converted?

I am aware that BTF information is created in the form of a compact
type by using pahole to deduplicate repeated types, strings information
from DWARF information. However, looking at the *btf or pahole file
in dwarves repository, it seemed that it was not responsible for the
conversion of the BTF key/value.

The remaining guess is that LLVM's BPF target compiler is responsible
for this, or it's probably somewhere in the kernel, but I'm not sure
where it is.

--
Best,
Daniel T. Lee

Re: Where can I find the map's BTF type key/value specification?

[Andrii Nakryiko]

On Sat, Oct 10, 2020 at 3:50 AM Daniel T. Lee <danieltimlee@gmail.com> wrote:
>
> I'm looking for how BTF type definition '__type(key, int)' is being changed
> to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
> it can be considered the same)

__type(key, int) captures both BTF ID of key and determines key_size
based on that type. You can specify both key and key_size, but that's
unnecessary (and resulting key size still has to match).

>
>     __uint(type, BPF_MAP_TYPE_ARRAY);
>     __type(key, int);          => __uint(key_size, sizeof(int))
>     __type(value, u32);    => __uint(value_size, sizeof(u32))
>     __uint(max_entries, 2);
>
> Whether the specific map type supports BTF or not can be inferred from
> the file in kernel/bpf/*map.c and by checking each MAP type's
> bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.
>
> But how can I figure out that specific types of map support BTF types for
> key/value? And how can I determine how this BTF key/value type is
> converted?

I think you answered your own question, you just search whether each
map implements .map_check_btf that allows key/value BTF type ID. E.g.,
see array_map_check_btf, which allows key/value type ID. And compare
to how perf_event_array_map_ops use map_check_no_btf for its
.map_check_btf callback.

So you can search for all struct bpf_map_ops declarations to see
operations for all map types, and then see what's there for
.map_check_btf. Ideally we should extend all maps to support BTF type
ID for key/value, but no one signed up to do that. If you are
interested, that should be a good way to contribute to kernel itself.

>
> I am aware that BTF information is created in the form of a compact
> type by using pahole to deduplicate repeated types, strings information
> from DWARF information. However, looking at the *btf or pahole file
> in dwarves repository, it seemed that it was not responsible for the
> conversion of the BTF key/value.
>
> The remaining guess is that LLVM's BPF target compiler is responsible
> for this, or it's probably somewhere in the kernel, but I'm not sure
> where it is.

BTF for the BPF program is emitted by Clang itself when you specify
`-target bpf -g`. pahole is used to convert kernel's (vmlinux) DWARF
into BTF and embed it into vmlinux image.

As for key/value BTF type id for maps, that's libbpf parsing map
definition and recording type IDs. So there are a few things playing
together. See abd29c931459 ("libbpf: allow specifying map definitions
using BTF") that introduced this feature.

>
> --
> Best,
> Daniel T. Lee

Re: Where can I find the map's BTF type key/value specification?

[Daniel T. Lee]

On Sun, Oct 11, 2020 at 9:06 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, Oct 10, 2020 at 3:50 AM Daniel T. Lee <danieltimlee@gmail.com> wrote:
> >
> > I'm looking for how BTF type definition '__type(key, int)' is being changed
> > to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
> > it can be considered the same)
>
> __type(key, int) captures both BTF ID of key and determines key_size
> based on that type. You can specify both key and key_size, but that's
> unnecessary (and resulting key size still has to match).
>
> >
> >     __uint(type, BPF_MAP_TYPE_ARRAY);
> >     __type(key, int);          => __uint(key_size, sizeof(int))
> >     __type(value, u32);    => __uint(value_size, sizeof(u32))
> >     __uint(max_entries, 2);
> >
> > Whether the specific map type supports BTF or not can be inferred from
> > the file in kernel/bpf/*map.c and by checking each MAP type's
> > bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.
> >
> > But how can I figure out that specific types of map support BTF types for
> > key/value? And how can I determine how this BTF key/value type is
> > converted?
>
> I think you answered your own question, you just search whether each
> map implements .map_check_btf that allows key/value BTF type ID. E.g.,
> see array_map_check_btf, which allows key/value type ID. And compare
> to how perf_event_array_map_ops use map_check_no_btf for its
> .map_check_btf callback.
>
> So you can search for all struct bpf_map_ops declarations to see
> operations for all map types, and then see what's there for
> .map_check_btf. Ideally we should extend all maps to support BTF type
> ID for key/value, but no one signed up to do that. If you are
> interested, that should be a good way to contribute to kernel itself.
>
> >
> > I am aware that BTF information is created in the form of a compact
> > type by using pahole to deduplicate repeated types, strings information
> > from DWARF information. However, looking at the *btf or pahole file
> > in dwarves repository, it seemed that it was not responsible for the
> > conversion of the BTF key/value.
> >
> > The remaining guess is that LLVM's BPF target compiler is responsible
> > for this, or it's probably somewhere in the kernel, but I'm not sure
> > where it is.
>
> BTF for the BPF program is emitted by Clang itself when you specify
> `-target bpf -g`. pahole is used to convert kernel's (vmlinux) DWARF
> into BTF and embed it into vmlinux image.
>
> As for key/value BTF type id for maps, that's libbpf parsing map
> definition and recording type IDs. So there are a few things playing
> together. See abd29c931459 ("libbpf: allow specifying map definitions
> using BTF") that introduced this feature.
>
> >
> > --
> > Best,
> > Daniel T. Lee


Thank you for taking the time and effort for the answer.
After following the implementation of key/value BTF type ID, A rough picture
began to be drawn.

What I didn't know well was clearly understood thanks to your explanation.

I'll look forward to contributing to this in the foreseeable future.

-- 
Best,
Daniel T. Lee