libbpf: failed to find BTF ID for ksym

libbpf: failed to find BTF ID for ksym

[Felix Maurer]

Hello,

I am working on a tracing tool for which I need know the address of some 
kernel data structures. The tool should be CO-RE and I am using libbpf 
(through the libbpf-rs Rust bindings, but this is not the issue I 
assume). However, I am having trouble to use ksyms with libbpf.

To get the address of the data structure (in my case 
skbuff_fclone_cache), I use an extern ksym declaration in my BPF code 
like this:

extern struct kmem_cache *skbuff_fclone_cache __ksym;

This compiles just fine, opening the compiled bytecode with libbpf also 
works. From the debug log[1], it can be seen that the extern is 
identified. However, loading the object fails with the following error:

libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: extern (ksym) 'skbuff_fclone_cache': failed to find BTF ID in 
kernel BTF(s).
libbpf: failed to load object 'skbuffTime_bpf'
libbpf: failed to load BPF skeleton 'skbuffTime_bpf': -22

I found, that libbpf has a check if it needs to load the BTF information 
or /proc/kallsyms. In my case, it loads the BTF information and cannot 
find the ksym in there. Searching in the BTF from the running kernel 
(bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep 
"skbuff_fclone_cache") indeed gives no results. However, it can be found 
in kallsyms (cat /proc/kallsyms | grep "skbuff_fclone_cache" gives one 
result). Therefore, a resolution of the ksym with kallsyms will probably 
work but is not even attempted.

Now, I am not really sure where the root cause of the issue can be 
found. Should the ksym be present in the BTF information (i.e., the 
issue comes from building the kernel) or is the BPF object file broken 
(i.e., an issue with clang) or is the identification of the ksym wrong 
(i.e., the issue is in the libbpf loading code). Or something completely 
different? Does anyone have a hint on what goes wrong here?

Some version information:
I am running Ubuntu 20.04, the kernel version is 5.8.0-45-generic (from 
HWE). The kernel has been build on the system to enable BTF; otherwise, 
the configuration is identical to the Ubuntu upstream. It has been 
compiled with gcc 9.3.0 and pahole v1.17. I am compiling the BPF code 
with clang 10.0.0 and tested with different libbpf versions from the 
GitHub mirror (0.2, 0.3, and 99bc17633).

Thank you already for taking a look at my issue!

Best regards,
Felix Maurer



[1]: Full libbpf opening debug log:
libbpf: loading object 'skbuffTime_bpf' from buffer
libbpf: elf: section(3) fexit/kmem_cache_alloc, size 176, link 0, flags 
6, type=1
libbpf: sec 'fexit/kmem_cache_alloc': found program 'kmem_cache_alloc' 
at insn offset 0 (0 bytes), code size 22 insns (176 bytes)
libbpf: elf: section(4) .relfexit/kmem_cache_alloc, size 32, link 26, 
flags 0, type=9
libbpf: elf: section(5) fexit/kmem_cache_alloc_node, size 176, link 0, 
flags 6, type=1
libbpf: sec 'fexit/kmem_cache_alloc_node': found program 
'kmem_cache_alloc_node' at insn offset 0 (0 bytes), code size 22 insns 
(176 bytes)
libbpf: elf: section(6) .relfexit/kmem_cache_alloc_node, size 32, link 
26, flags 0, type=9
libbpf: elf: section(7) license, size 4, link 0, flags 3, type=1
libbpf: license of skbuffTime_bpf is GPL
libbpf: elf: section(8) .maps, size 24, link 0, flags 3, type=1
libbpf: elf: section(17) .BTF, size 4419, link 0, flags 0, type=1
libbpf: elf: section(19) .BTF.ext, size 440, link 0, flags 0, type=1
libbpf: elf: section(26) .symtab, size 263040, link 1, flags 0, type=2
libbpf: looking for externs among 10960 symbols...
libbpf: collected 1 externs total
libbpf: extern (ksym) #0: symbol 10959, name skbuff_fclone_cache
libbpf: map 'events': at sec_idx 8, offset 0.
libbpf: map 'events': found type = 4.
libbpf: map 'events': found key_size = 4.
libbpf: map 'events': found value_size = 4.
libbpf: sec '.relfexit/kmem_cache_alloc': collecting relocation for 
section(3) 'fexit/kmem_cache_alloc'
libbpf: sec '.relfexit/kmem_cache_alloc': relo #0: insn #2 against 
'skbuff_fclone_cache'
libbpf: prog 'kmem_cache_alloc': found extern #0 'skbuff_fclone_cache' 
(sym 10959) for insn #2
libbpf: sec '.relfexit/kmem_cache_alloc': relo #1: insn #14 against 'events'
libbpf: prog 'kmem_cache_alloc': found map 0 (events, sec 8, off 0) for 
insn #14
libbpf: sec '.relfexit/kmem_cache_alloc_node': collecting relocation for 
section(5) 'fexit/kmem_cache_alloc_node'
libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #0: insn #2 against 
'skbuff_fclone_cache'
libbpf: prog 'kmem_cache_alloc_node': found extern #0 
'skbuff_fclone_cache' (sym 10959) for insn #2
libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #1: insn #14 against 
'events'
libbpf: prog 'kmem_cache_alloc_node': found map 0 (events, sec 8, off 0) 
for insn #14

Re: libbpf: failed to find BTF ID for ksym

[Yonghong Song]

On 4/2/21 3:44 PM, Felix Maurer wrote:
> Hello,
> 
> I am working on a tracing tool for which I need know the address of some 
> kernel data structures. The tool should be CO-RE and I am using libbpf 
> (through the libbpf-rs Rust bindings, but this is not the issue I 
> assume). However, I am having trouble to use ksyms with libbpf.
> 
> To get the address of the data structure (in my case 
> skbuff_fclone_cache), I use an extern ksym declaration in my BPF code 
> like this:
> 
> extern struct kmem_cache *skbuff_fclone_cache __ksym;
> 
> This compiles just fine, opening the compiled bytecode with libbpf also 
> works. From the debug log[1], it can be seen that the extern is 
> identified. However, loading the object fails with the following error:
> 
> libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
> libbpf: extern (ksym) 'skbuff_fclone_cache': failed to find BTF ID in 
> kernel BTF(s).
> libbpf: failed to load object 'skbuffTime_bpf'
> libbpf: failed to load BPF skeleton 'skbuffTime_bpf': -22

net/core/skbuff.c:static struct kmem_cache *skbuff_fclone_cache 
__ro_after_init;

skbuff_fclone_cache is a static variable. I think pahole does not
emit static variables into btf, only global variables.

> 
> I found, that libbpf has a check if it needs to load the BTF information 
> or /proc/kallsyms. In my case, it loads the BTF information and cannot 
> find the ksym in there. Searching in the BTF from the running kernel 
> (bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep 
> "skbuff_fclone_cache") indeed gives no results. However, it can be found 
> in kallsyms (cat /proc/kallsyms | grep "skbuff_fclone_cache" gives one 
> result). Therefore, a resolution of the ksym with kallsyms will probably 
> work but is not even attempted.
> 
> Now, I am not really sure where the root cause of the issue can be 
> found. Should the ksym be present in the BTF information (i.e., the 
> issue comes from building the kernel) or is the BPF object file broken 
> (i.e., an issue with clang) or is the identification of the ksym wrong 
> (i.e., the issue is in the libbpf loading code). Or something completely 
> different? Does anyone have a hint on what goes wrong here?
> 
> Some version information:
> I am running Ubuntu 20.04, the kernel version is 5.8.0-45-generic (from 
> HWE). The kernel has been build on the system to enable BTF; otherwise, 
> the configuration is identical to the Ubuntu upstream. It has been 
> compiled with gcc 9.3.0 and pahole v1.17. I am compiling the BPF code 
> with clang 10.0.0 and tested with different libbpf versions from the 
> GitHub mirror (0.2, 0.3, and 99bc17633).
> 
> Thank you already for taking a look at my issue!
> 
> Best regards,
> Felix Maurer
> 
> 
> 
> [1]: Full libbpf opening debug log:
> libbpf: loading object 'skbuffTime_bpf' from buffer
> libbpf: elf: section(3) fexit/kmem_cache_alloc, size 176, link 0, flags 
> 6, type=1
> libbpf: sec 'fexit/kmem_cache_alloc': found program 'kmem_cache_alloc' 
> at insn offset 0 (0 bytes), code size 22 insns (176 bytes)
> libbpf: elf: section(4) .relfexit/kmem_cache_alloc, size 32, link 26, 
> flags 0, type=9
> libbpf: elf: section(5) fexit/kmem_cache_alloc_node, size 176, link 0, 
> flags 6, type=1
> libbpf: sec 'fexit/kmem_cache_alloc_node': found program 
> 'kmem_cache_alloc_node' at insn offset 0 (0 bytes), code size 22 insns 
> (176 bytes)
> libbpf: elf: section(6) .relfexit/kmem_cache_alloc_node, size 32, link 
> 26, flags 0, type=9
> libbpf: elf: section(7) license, size 4, link 0, flags 3, type=1
> libbpf: license of skbuffTime_bpf is GPL
> libbpf: elf: section(8) .maps, size 24, link 0, flags 3, type=1
> libbpf: elf: section(17) .BTF, size 4419, link 0, flags 0, type=1
> libbpf: elf: section(19) .BTF.ext, size 440, link 0, flags 0, type=1
> libbpf: elf: section(26) .symtab, size 263040, link 1, flags 0, type=2
> libbpf: looking for externs among 10960 symbols...
> libbpf: collected 1 externs total
> libbpf: extern (ksym) #0: symbol 10959, name skbuff_fclone_cache
> libbpf: map 'events': at sec_idx 8, offset 0.
> libbpf: map 'events': found type = 4.
> libbpf: map 'events': found key_size = 4.
> libbpf: map 'events': found value_size = 4.
> libbpf: sec '.relfexit/kmem_cache_alloc': collecting relocation for 
> section(3) 'fexit/kmem_cache_alloc'
> libbpf: sec '.relfexit/kmem_cache_alloc': relo #0: insn #2 against 
> 'skbuff_fclone_cache'
> libbpf: prog 'kmem_cache_alloc': found extern #0 'skbuff_fclone_cache' 
> (sym 10959) for insn #2
> libbpf: sec '.relfexit/kmem_cache_alloc': relo #1: insn #14 against 
> 'events'
> libbpf: prog 'kmem_cache_alloc': found map 0 (events, sec 8, off 0) for 
> insn #14
> libbpf: sec '.relfexit/kmem_cache_alloc_node': collecting relocation for 
> section(5) 'fexit/kmem_cache_alloc_node'
> libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #0: insn #2 against 
> 'skbuff_fclone_cache'
> libbpf: prog 'kmem_cache_alloc_node': found extern #0 
> 'skbuff_fclone_cache' (sym 10959) for insn #2
> libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #1: insn #14 against 
> 'events'
> libbpf: prog 'kmem_cache_alloc_node': found map 0 (events, sec 8, off 0) 
> for insn #14

Re: libbpf: failed to find BTF ID for ksym

[Felix Maurer]

[-- Attachment #1.1: Type: text/plain, Size: 5628 bytes --]

On 03.04.21 00:59, Yonghong Song wrote:
> 
> 
> On 4/2/21 3:44 PM, Felix Maurer wrote:
>> Hello,
>>
>> I am working on a tracing tool for which I need know the address of 
>> some kernel data structures. The tool should be CO-RE and I am using 
>> libbpf (through the libbpf-rs Rust bindings, but this is not the issue 
>> I assume). However, I am having trouble to use ksyms with libbpf.
>>
>> To get the address of the data structure (in my case 
>> skbuff_fclone_cache), I use an extern ksym declaration in my BPF code 
>> like this:
>>
>> extern struct kmem_cache *skbuff_fclone_cache __ksym;
>>
>> This compiles just fine, opening the compiled bytecode with libbpf 
>> also works. From the debug log[1], it can be seen that the extern is 
>> identified. However, loading the object fails with the following error:
>>
>> libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
>> libbpf: extern (ksym) 'skbuff_fclone_cache': failed to find BTF ID in 
>> kernel BTF(s).
>> libbpf: failed to load object 'skbuffTime_bpf'
>> libbpf: failed to load BPF skeleton 'skbuffTime_bpf': -22
> 
> net/core/skbuff.c:static struct kmem_cache *skbuff_fclone_cache 
> __ro_after_init;
> 
> skbuff_fclone_cache is a static variable. I think pahole does not
> emit static variables into btf, only global variables.

Is that the expected behavior that static variables do not show up in 
BTF? If so, I guess the way to fix it would be that libbpf looks up 
static vars in kallsyms instead of BTF.

>>
>> I found, that libbpf has a check if it needs to load the BTF 
>> information or /proc/kallsyms. In my case, it loads the BTF 
>> information and cannot find the ksym in there. Searching in the BTF 
>> from the running kernel (bpftool btf dump file /sys/kernel/btf/vmlinux 
>> format raw | grep "skbuff_fclone_cache") indeed gives no results. 
>> However, it can be found in kallsyms (cat /proc/kallsyms | grep 
>> "skbuff_fclone_cache" gives one result). Therefore, a resolution of 
>> the ksym with kallsyms will probably work but is not even attempted.
>>
>> Now, I am not really sure where the root cause of the issue can be 
>> found. Should the ksym be present in the BTF information (i.e., the 
>> issue comes from building the kernel) or is the BPF object file broken 
>> (i.e., an issue with clang) or is the identification of the ksym wrong 
>> (i.e., the issue is in the libbpf loading code). Or something 
>> completely different? Does anyone have a hint on what goes wrong here?
>>
>> Some version information:
>> I am running Ubuntu 20.04, the kernel version is 5.8.0-45-generic 
>> (from HWE). The kernel has been build on the system to enable BTF; 
>> otherwise, the configuration is identical to the Ubuntu upstream. It 
>> has been compiled with gcc 9.3.0 and pahole v1.17. I am compiling the 
>> BPF code with clang 10.0.0 and tested with different libbpf versions 
>> from the GitHub mirror (0.2, 0.3, and 99bc17633).
>>
>> Thank you already for taking a look at my issue!
>>
>> Best regards,
>> Felix Maurer
>>
>>
>>
>> [1]: Full libbpf opening debug log:
>> libbpf: loading object 'skbuffTime_bpf' from buffer
>> libbpf: elf: section(3) fexit/kmem_cache_alloc, size 176, link 0, 
>> flags 6, type=1
>> libbpf: sec 'fexit/kmem_cache_alloc': found program 'kmem_cache_alloc' 
>> at insn offset 0 (0 bytes), code size 22 insns (176 bytes)
>> libbpf: elf: section(4) .relfexit/kmem_cache_alloc, size 32, link 26, 
>> flags 0, type=9
>> libbpf: elf: section(5) fexit/kmem_cache_alloc_node, size 176, link 0, 
>> flags 6, type=1
>> libbpf: sec 'fexit/kmem_cache_alloc_node': found program 
>> 'kmem_cache_alloc_node' at insn offset 0 (0 bytes), code size 22 insns 
>> (176 bytes)
>> libbpf: elf: section(6) .relfexit/kmem_cache_alloc_node, size 32, link 
>> 26, flags 0, type=9
>> libbpf: elf: section(7) license, size 4, link 0, flags 3, type=1
>> libbpf: license of skbuffTime_bpf is GPL
>> libbpf: elf: section(8) .maps, size 24, link 0, flags 3, type=1
>> libbpf: elf: section(17) .BTF, size 4419, link 0, flags 0, type=1
>> libbpf: elf: section(19) .BTF.ext, size 440, link 0, flags 0, type=1
>> libbpf: elf: section(26) .symtab, size 263040, link 1, flags 0, type=2
>> libbpf: looking for externs among 10960 symbols...
>> libbpf: collected 1 externs total
>> libbpf: extern (ksym) #0: symbol 10959, name skbuff_fclone_cache
>> libbpf: map 'events': at sec_idx 8, offset 0.
>> libbpf: map 'events': found type = 4.
>> libbpf: map 'events': found key_size = 4.
>> libbpf: map 'events': found value_size = 4.
>> libbpf: sec '.relfexit/kmem_cache_alloc': collecting relocation for 
>> section(3) 'fexit/kmem_cache_alloc'
>> libbpf: sec '.relfexit/kmem_cache_alloc': relo #0: insn #2 against 
>> 'skbuff_fclone_cache'
>> libbpf: prog 'kmem_cache_alloc': found extern #0 'skbuff_fclone_cache' 
>> (sym 10959) for insn #2
>> libbpf: sec '.relfexit/kmem_cache_alloc': relo #1: insn #14 against 
>> 'events'
>> libbpf: prog 'kmem_cache_alloc': found map 0 (events, sec 8, off 0) 
>> for insn #14
>> libbpf: sec '.relfexit/kmem_cache_alloc_node': collecting relocation 
>> for section(5) 'fexit/kmem_cache_alloc_node'
>> libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #0: insn #2 
>> against 'skbuff_fclone_cache'
>> libbpf: prog 'kmem_cache_alloc_node': found extern #0 
>> 'skbuff_fclone_cache' (sym 10959) for insn #2
>> libbpf: sec '.relfexit/kmem_cache_alloc_node': relo #1: insn #14 
>> against 'events'
>> libbpf: prog 'kmem_cache_alloc_node': found map 0 (events, sec 8, off 
>> 0) for insn #14


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: libbpf: failed to find BTF ID for ksym

[Andrii Nakryiko]

On Fri, Apr 2, 2021 at 3:47 PM Felix Maurer <felix@felix-maurer.de> wrote:
>
> Hello,
>
> I am working on a tracing tool for which I need know the address of some
> kernel data structures. The tool should be CO-RE and I am using libbpf
> (through the libbpf-rs Rust bindings, but this is not the issue I
> assume). However, I am having trouble to use ksyms with libbpf.
>
> To get the address of the data structure (in my case
> skbuff_fclone_cache), I use an extern ksym declaration in my BPF code
> like this:
>
> extern struct kmem_cache *skbuff_fclone_cache __ksym;
>

Due to skbuff_fclone_cache being a static variable, typed __ksym
(where you specify expected type and thus BPF verifier will expect BTF
ID and will allow reading it directly from your BPF program) won't
work. If you need to just have an address of the symbol, you can use
untyped __ksym, which will use /proc/kallsyms:

extern const void skbuff_fclone_cache __ksym;

If you need to further read any data (e.g., follow the pointer and
read struct kmem_cache's fields), you can use BPF_CORE_READ() macro.

[...]

>
> Now, I am not really sure where the root cause of the issue can be
> found. Should the ksym be present in the BTF information (i.e., the
> issue comes from building the kernel) or is the BPF object file broken
> (i.e., an issue with clang) or is the identification of the ksym wrong
> (i.e., the issue is in the libbpf loading code). Or something completely
> different? Does anyone have a hint on what goes wrong here?
>

As Yonghong already replied, pahole doesn't generate BTF type
information for static variables right now, so it's expected.

[...]

Re: libbpf: failed to find BTF ID for ksym

[Felix Maurer]

[-- Attachment #1.1: Type: text/plain, Size: 1960 bytes --]

On 03.04.21 17:50, Andrii Nakryiko wrote:
> On Fri, Apr 2, 2021 at 3:47 PM Felix Maurer <felix@felix-maurer.de> wrote:
>>
>> Hello,
>>
>> I am working on a tracing tool for which I need know the address of some
>> kernel data structures. The tool should be CO-RE and I am using libbpf
>> (through the libbpf-rs Rust bindings, but this is not the issue I
>> assume). However, I am having trouble to use ksyms with libbpf.
>>
>> To get the address of the data structure (in my case
>> skbuff_fclone_cache), I use an extern ksym declaration in my BPF code
>> like this:
>>
>> extern struct kmem_cache *skbuff_fclone_cache __ksym;
>>
> 
> Due to skbuff_fclone_cache being a static variable, typed __ksym
> (where you specify expected type and thus BPF verifier will expect BTF
> ID and will allow reading it directly from your BPF program) won't
> work. If you need to just have an address of the symbol, you can use
> untyped __ksym, which will use /proc/kallsyms:
> 
> extern const void skbuff_fclone_cache __ksym;
> 
> If you need to further read any data (e.g., follow the pointer and
> read struct kmem_cache's fields), you can use BPF_CORE_READ() macro.

Thank you very much for the explanation. The declaration like this works 
fine and solved my issues.

This also means that the whole chain works as expected, no bugs here :)

>>
>> Now, I am not really sure where the root cause of the issue can be
>> found. Should the ksym be present in the BTF information (i.e., the
>> issue comes from building the kernel) or is the BPF object file broken
>> (i.e., an issue with clang) or is the identification of the ksym wrong
>> (i.e., the issue is in the libbpf loading code). Or something completely
>> different? Does anyone have a hint on what goes wrong here?
>>
> 
> As Yonghong already replied, pahole doesn't generate BTF type
> information for static variables right now, so it's expected.
> 
> [...]
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]