BPF Archive on lore.kernel.org
 help / color / Atom feed
* Where can I find the map's BTF type key/value specification?
@ 2020-10-10 10:50 Daniel T. Lee
  2020-10-11  0:06 ` Andrii Nakryiko
  0 siblings, 1 reply; 3+ messages in thread
From: Daniel T. Lee @ 2020-10-10 10:50 UTC (permalink / raw)
  To: bpf
  Cc: Andrii Nakryiko, Alexei Starovoitov, Daniel Borkmann,
	Yonghong Song, Andrii Nakryiko, Martin Lau

I'm looking for how BTF type definition '__type(key, int)' is being changed
to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
it can be considered the same)

    __uint(type, BPF_MAP_TYPE_ARRAY);
    __type(key, int);          => __uint(key_size, sizeof(int))
    __type(value, u32);    => __uint(value_size, sizeof(u32))
    __uint(max_entries, 2);

Whether the specific map type supports BTF or not can be inferred from
the file in kernel/bpf/*map.c and by checking each MAP type's
bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.

But how can I figure out that specific types of map support BTF types for
key/value? And how can I determine how this BTF key/value type is
converted?

I am aware that BTF information is created in the form of a compact
type by using pahole to deduplicate repeated types, strings information
from DWARF information. However, looking at the *btf or pahole file
in dwarves repository, it seemed that it was not responsible for the
conversion of the BTF key/value.

The remaining guess is that LLVM's BPF target compiler is responsible
for this, or it's probably somewhere in the kernel, but I'm not sure
where it is.

--
Best,
Daniel T. Lee

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Where can I find the map's BTF type key/value specification?
  2020-10-10 10:50 Where can I find the map's BTF type key/value specification? Daniel T. Lee
@ 2020-10-11  0:06 ` Andrii Nakryiko
  2020-10-13  5:13   ` Daniel T. Lee
  0 siblings, 1 reply; 3+ messages in thread
From: Andrii Nakryiko @ 2020-10-11  0:06 UTC (permalink / raw)
  To: Daniel T. Lee
  Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Yonghong Song,
	Andrii Nakryiko, Martin Lau

On Sat, Oct 10, 2020 at 3:50 AM Daniel T. Lee <danieltimlee@gmail.com> wrote:
>
> I'm looking for how BTF type definition '__type(key, int)' is being changed
> to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
> it can be considered the same)

__type(key, int) captures both BTF ID of key and determines key_size
based on that type. You can specify both key and key_size, but that's
unnecessary (and resulting key size still has to match).

>
>     __uint(type, BPF_MAP_TYPE_ARRAY);
>     __type(key, int);          => __uint(key_size, sizeof(int))
>     __type(value, u32);    => __uint(value_size, sizeof(u32))
>     __uint(max_entries, 2);
>
> Whether the specific map type supports BTF or not can be inferred from
> the file in kernel/bpf/*map.c and by checking each MAP type's
> bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.
>
> But how can I figure out that specific types of map support BTF types for
> key/value? And how can I determine how this BTF key/value type is
> converted?

I think you answered your own question, you just search whether each
map implements .map_check_btf that allows key/value BTF type ID. E.g.,
see array_map_check_btf, which allows key/value type ID. And compare
to how perf_event_array_map_ops use map_check_no_btf for its
.map_check_btf callback.

So you can search for all struct bpf_map_ops declarations to see
operations for all map types, and then see what's there for
.map_check_btf. Ideally we should extend all maps to support BTF type
ID for key/value, but no one signed up to do that. If you are
interested, that should be a good way to contribute to kernel itself.

>
> I am aware that BTF information is created in the form of a compact
> type by using pahole to deduplicate repeated types, strings information
> from DWARF information. However, looking at the *btf or pahole file
> in dwarves repository, it seemed that it was not responsible for the
> conversion of the BTF key/value.
>
> The remaining guess is that LLVM's BPF target compiler is responsible
> for this, or it's probably somewhere in the kernel, but I'm not sure
> where it is.

BTF for the BPF program is emitted by Clang itself when you specify
`-target bpf -g`. pahole is used to convert kernel's (vmlinux) DWARF
into BTF and embed it into vmlinux image.

As for key/value BTF type id for maps, that's libbpf parsing map
definition and recording type IDs. So there are a few things playing
together. See abd29c931459 ("libbpf: allow specifying map definitions
using BTF") that introduced this feature.

>
> --
> Best,
> Daniel T. Lee

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Where can I find the map's BTF type key/value specification?
  2020-10-11  0:06 ` Andrii Nakryiko
@ 2020-10-13  5:13   ` Daniel T. Lee
  0 siblings, 0 replies; 3+ messages in thread
From: Daniel T. Lee @ 2020-10-13  5:13 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Yonghong Song,
	Andrii Nakryiko, Martin Lau

On Sun, Oct 11, 2020 at 9:06 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, Oct 10, 2020 at 3:50 AM Daniel T. Lee <danieltimlee@gmail.com> wrote:
> >
> > I'm looking for how BTF type definition '__type(key, int)' is being changed
> > to '__uint(key_size, sizeof(int))'. (Not exactly "changed" but wonder how
> > it can be considered the same)
>
> __type(key, int) captures both BTF ID of key and determines key_size
> based on that type. You can specify both key and key_size, but that's
> unnecessary (and resulting key size still has to match).
>
> >
> >     __uint(type, BPF_MAP_TYPE_ARRAY);
> >     __type(key, int);          => __uint(key_size, sizeof(int))
> >     __type(value, u32);    => __uint(value_size, sizeof(u32))
> >     __uint(max_entries, 2);
> >
> > Whether the specific map type supports BTF or not can be inferred from
> > the file in kernel/bpf/*map.c and by checking each MAP type's
> > bpf_map_ops .map_check_btf pointer is initialized as map_check_no_btf.
> >
> > But how can I figure out that specific types of map support BTF types for
> > key/value? And how can I determine how this BTF key/value type is
> > converted?
>
> I think you answered your own question, you just search whether each
> map implements .map_check_btf that allows key/value BTF type ID. E.g.,
> see array_map_check_btf, which allows key/value type ID. And compare
> to how perf_event_array_map_ops use map_check_no_btf for its
> .map_check_btf callback.
>
> So you can search for all struct bpf_map_ops declarations to see
> operations for all map types, and then see what's there for
> .map_check_btf. Ideally we should extend all maps to support BTF type
> ID for key/value, but no one signed up to do that. If you are
> interested, that should be a good way to contribute to kernel itself.
>
> >
> > I am aware that BTF information is created in the form of a compact
> > type by using pahole to deduplicate repeated types, strings information
> > from DWARF information. However, looking at the *btf or pahole file
> > in dwarves repository, it seemed that it was not responsible for the
> > conversion of the BTF key/value.
> >
> > The remaining guess is that LLVM's BPF target compiler is responsible
> > for this, or it's probably somewhere in the kernel, but I'm not sure
> > where it is.
>
> BTF for the BPF program is emitted by Clang itself when you specify
> `-target bpf -g`. pahole is used to convert kernel's (vmlinux) DWARF
> into BTF and embed it into vmlinux image.
>
> As for key/value BTF type id for maps, that's libbpf parsing map
> definition and recording type IDs. So there are a few things playing
> together. See abd29c931459 ("libbpf: allow specifying map definitions
> using BTF") that introduced this feature.
>
> >
> > --
> > Best,
> > Daniel T. Lee


Thank you for taking the time and effort for the answer.
After following the implementation of key/value BTF type ID, A rough picture
began to be drawn.

What I didn't know well was clearly understood thanks to your explanation.

I'll look forward to contributing to this in the foreseeable future.

-- 
Best,
Daniel T. Lee

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-10 10:50 Where can I find the map's BTF type key/value specification? Daniel T. Lee
2020-10-11  0:06 ` Andrii Nakryiko
2020-10-13  5:13   ` Daniel T. Lee

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git