Re: Trying the bpf trace a bpf xdp program

Re: Trying the bpf trace a bpf xdp program

[Alexei Starovoitov]

On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
> Trying out the BPF trace to trace a BPF program, but I’m already
> getting stuck loading the object with the fexit  :(

I can take a look after holidays.

> libbpf: load bpf program failed: Argument list too long
> libbpf: failed to load program 'fexit/xdp_prog_simple'
> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
> ERROR: Failed to load object file: Operation not permitted

please add -vvv and share full output.

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:

> On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>>
>> Trying out the BPF trace to trace a BPF program, but I’m already
>> getting stuck loading the object with the fexit  :(
>
> I can take a look after holidays.

Enjoy the Holidays!! I figured out my auto kernel install script failed 
whiteout me noticing, and I was running an old kernel :(

I will try tomorrow with the correct kernel…


>> libbpf: load bpf program failed: Argument list too long
>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>> ERROR: Failed to load object file: Operation not permitted
>
> please add -vvv and share full output.

Re: Trying the bpf trace a bpf xdp program

[Alexei Starovoitov]

On Thu, Nov 28, 2019 at 11:16 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:
>
> > On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com>
> > wrote:
> >>
> >> Trying out the BPF trace to trace a BPF program, but I’m already
> >> getting stuck loading the object with the fexit  :(
> >
> > I can take a look after holidays.
>
> Enjoy the Holidays!! I figured out my auto kernel install script failed
> whiteout me noticing, and I was running an old kernel :(
>
> I will try tomorrow with the correct kernel…

Please also check that you have the latest llvm and pahole.
pahole version should be >= 1.13.
clang ideally from master.
If all that is working then downgrade one by one and bisect whether the bug is.

>
> >> libbpf: load bpf program failed: Argument list too long
> >> libbpf: failed to load program 'fexit/xdp_prog_simple'
> >> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
> >> ERROR: Failed to load object file: Operation not permitted
> >
> > please add -vvv and share full output.
>

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 28 Nov 2019, at 20:47, Alexei Starovoitov wrote:

> On Thu, Nov 28, 2019 at 11:16 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>>
>>
>>
>> On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:
>>
>>> On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com>
>>> wrote:
>>>>
>>>> Trying out the BPF trace to trace a BPF program, but I’m already
>>>> getting stuck loading the object with the fexit  :(
>>>
>>> I can take a look after holidays.
>>
>> Enjoy the Holidays!! I figured out my auto kernel install script 
>> failed
>> whiteout me noticing, and I was running an old kernel :(
>>
>> I will try tomorrow with the correct kernel…
>
> Please also check that you have the latest llvm and pahole.
> pahole version should be >= 1.13.
> clang ideally from master.
> If all that is working then downgrade one by one and bisect whether 
> the bug is.
>

I tried it with the latest kernel loaded but still I got some errors on 
bpf_object__load_xattr():

$ sudo ./xdp_sample_fentry_fexit_user
libbpf: Cannot find bpf_func_info for main program sec 
fexit/xdp_prog_simple. Ignore all bpf_func_info.
libbpf: load bpf program failed: Operation not permitted
libbpf: failed to load program 'fexit/xdp_prog_simple'
libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
ERROR: Failed to load object file: Operation not permitted


With strace:

[vagrant@xdp-tutorial tracing05-xdp-fentry]$ sudo 
/home/vagrant/strace/strace -e bpf ./xdp_sample_fentry_fexit_user
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=36, next_id=0, open_flags=0}, 120) = 
3
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, 
insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0, 
log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
prog_name="", prog_ifindex=0, 
expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, 
insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0, 
log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
prog_name="test", prog_ifindex=0, 
expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, 
value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", 
map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0}, 120) = 
5
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5, 
insns=0x7ffcaedfd490, license="GPL", log_level=0, log_size=0, 
log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
prog_name="", prog_ifindex=0, 
expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 6
bpf(BPF_BTF_LOAD, 
{btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., 
btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) = 
5
bpf(BPF_BTF_LOAD, 
{btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"..., 
btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 120) = 
5
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, 
value_size=4, max_entries=1, map_flags=0x400 /* BPF_F_??? */, 
inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, 
btf_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
bpf(BPF_BTF_LOAD, 
{btf="\237\353\1\0\30\0\0\0\0\0\0\0`\1\0\0`\1\0\0\236\0\0\0\0\0\0\0\0\0\0\3"..., 
btf_log_buf=NULL, btf_size=534, btf_log_size=0, btf_log_level=0}, 120) = 
5
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
info=0x7ffcaedfd4a0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
info=0x2018c30}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=3}, 120) = 4
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=16, 
info=0x7ffcaedfd5a0}}, 120) = 0
libbpf: Cannot find bpf_func_info for main program sec 
fexit/xdp_prog_simple. Ignore all bpf_func_info.
bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=30, 
insns=0x20191f0, license="GPL", log_level=7, log_size=16777215, 
log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
prog_name="trace_on_exit", prog_ifindex=0, expected_attach_type=0x19 /* 
BPF_??? */, prog_btf_fd=5, func_info_rec_size=0, func_info=NULL, 
func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, 
...}, 120) = -1 EPERM (Operation not permitted)
libbpf: load bpf program failed: Operation not permitted
libbpf: failed to load program 'fexit/xdp_prog_simple'
libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
ERROR: Failed to load object file: Operation not permitted
+++ exited with 0 +++

And with full libbpf print debugging:

libbpf: loading ./xdp_sample_fentry_fexit_kern.o
libbpf: section(1) .strtab, size 250, link 0, flags 0, type=3
libbpf: skip section(1) .strtab
libbpf: section(2) .text, size 0, link 0, flags 6, type=1
libbpf: skip section(2) .text
libbpf: section(3) fexit/xdp_prog_simple, size 240, link 0, flags 6, 
type=1
libbpf: found program fexit/xdp_prog_simple
libbpf: section(4) license, size 4, link 0, flags 3, type=1
libbpf: license of ./xdp_sample_fentry_fexit_kern.o is GPL
libbpf: section(5) .rodata.str1.1, size 52, link 0, flags 32, type=1
libbpf: skip section(5) .rodata.str1.1
libbpf: section(6) .debug_str, size 344, link 0, flags 30, type=1
libbpf: skip section(6) .debug_str
libbpf: section(7) .debug_loc, size 70, link 0, flags 0, type=1
libbpf: skip section(7) .debug_loc
libbpf: section(8) .debug_abbrev, size 297, link 0, flags 0, type=1
libbpf: skip section(8) .debug_abbrev
libbpf: section(9) .debug_info, size 403, link 0, flags 0, type=1
libbpf: skip section(9) .debug_info
libbpf: section(10) .rel.debug_info, size 528, link 21, flags 0, type=9
libbpf: skip relo .rel.debug_info(10) for section(9)
libbpf: section(11) .debug_ranges, size 96, link 0, flags 0, type=1
libbpf: skip section(11) .debug_ranges
libbpf: section(12) .debug_macinfo, size 1, link 0, flags 0, type=1
libbpf: skip section(12) .debug_macinfo
libbpf: section(13) .BTF, size 534, link 0, flags 0, type=1
libbpf: section(14) .rel.BTF, size 16, link 21, flags 0, type=9
libbpf: skip relo .rel.BTF(14) for section(13)
libbpf: section(15) .BTF.ext, size 152, link 0, flags 0, type=1
libbpf: section(16) .rel.BTF.ext, size 96, link 21, flags 0, type=9
libbpf: skip relo .rel.BTF.ext(16) for section(15)
libbpf: section(17) .debug_frame, size 40, link 0, flags 0, type=1
libbpf: skip section(17) .debug_frame
libbpf: section(18) .rel.debug_frame, size 32, link 21, flags 0, type=9
libbpf: skip relo .rel.debug_frame(18) for section(17)
libbpf: section(19) .debug_line, size 206, link 0, flags 0, type=1
libbpf: skip section(19) .debug_line
libbpf: section(20) .rel.debug_line, size 16, link 21, flags 0, type=9
libbpf: skip relo .rel.debug_line(20) for section(19)
libbpf: section(21) .symtab, size 792, link 1, flags 0, type=2
libbpf: Cannot find bpf_func_info for main program sec 
fexit/xdp_prog_simple. Ignore all bpf_func_info.
libbpf: load bpf program failed: Operation not permitted
libbpf: failed to load program 'fexit/xdp_prog_simple'
libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
ERROR: Failed to load object file: Operation not permitted

I’m using the latest pahole, but standard llvm 9.0.0 from Fedora.

So I decided to at least run the self-test and see if this passes, and 
work my way from there.
But it’s failing to build with errors like “use of unknown builtin 
'__builtin_preserve_field_info’”, so I’m assuming my clang9 needs 
upgrading…

I quickly tried doing this while working on other stuff, but I failed to 
multitask, so will continue with this next week.

Any ideas are welcome…

Cheers,


Eelco

Re: Trying the bpf trace a bpf xdp program

[Yonghong Song]

On 11/29/19 8:30 AM, Eelco Chaudron wrote:
> 
> 
> On 28 Nov 2019, at 20:47, Alexei Starovoitov wrote:
> 
>> On Thu, Nov 28, 2019 at 11:16 AM Eelco Chaudron <echaudro@redhat.com> 
>> wrote:
>>>
>>>
>>>
>>> On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:
>>>
>>>> On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com>
>>>> wrote:
>>>>>
>>>>> Trying out the BPF trace to trace a BPF program, but I’m already
>>>>> getting stuck loading the object with the fexit  :(
>>>>
>>>> I can take a look after holidays.
>>>
>>> Enjoy the Holidays!! I figured out my auto kernel install script failed
>>> whiteout me noticing, and I was running an old kernel :(
>>>
>>> I will try tomorrow with the correct kernel…
>>
>> Please also check that you have the latest llvm and pahole.
>> pahole version should be >= 1.13.
>> clang ideally from master.
>> If all that is working then downgrade one by one and bisect whether 
>> the bug is.
>>
> 
> I tried it with the latest kernel loaded but still I got some errors on 
> bpf_object__load_xattr():
> 
> $ sudo ./xdp_sample_fentry_fexit_user
> libbpf: Cannot find bpf_func_info for main program sec 
> fexit/xdp_prog_simple. Ignore all bpf_func_info.
> libbpf: load bpf program failed: Operation not permitted
> libbpf: failed to load program 'fexit/xdp_prog_simple'
> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
> ERROR: Failed to load object file: Operation not permitted
> 
> 
> With strace:
> 
> [vagrant@xdp-tutorial tracing05-xdp-fentry]$ sudo 
> /home/vagrant/strace/strace -e bpf ./xdp_sample_fentry_fexit_user
> bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=36, next_id=0, open_flags=0}, 120) = 3
> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, 
> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0, 
> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
> prog_name="", prog_ifindex=0, 
> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
> func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, 
> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0, 
> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
> prog_name="test", prog_ifindex=0, 
> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
> func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, 
> value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", 
> map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0}, 120) = 5
> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5, 
> insns=0x7ffcaedfd490, license="GPL", log_level=0, log_size=0, 
> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
> prog_name="", prog_ifindex=0, 
> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, 
> func_info_rec_size=0, func_info=NULL, func_info_cnt=0, 
> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 6
> bpf(BPF_BTF_LOAD, 
> {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., 
> btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) = 5
> bpf(BPF_BTF_LOAD, 
> {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"..., 
> btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 120) = 5
> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, 
> value_size=4, max_entries=1, map_flags=0x400 /* BPF_F_??? */, 
> inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, 
> btf_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
> bpf(BPF_BTF_LOAD, 
> {btf="\237\353\1\0\30\0\0\0\0\0\0\0`\1\0\0`\1\0\0\236\0\0\0\0\0\0\0\0\0\0\3"..., 
> btf_log_buf=NULL, btf_size=534, btf_log_size=0, btf_log_level=0}, 120) = 5
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
> info=0x7ffcaedfd4a0}}, 120) = 0
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
> info=0x2018c30}}, 120) = 0
> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=3}, 120) = 4
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=16, 
> info=0x7ffcaedfd5a0}}, 120) = 0
> libbpf: Cannot find bpf_func_info for main program sec 
> fexit/xdp_prog_simple. Ignore all bpf_func_info.
> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=30, 
> insns=0x20191f0, license="GPL", log_level=7, log_size=16777215, 
> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, 
> prog_name="trace_on_exit", prog_ifindex=0, expected_attach_type=0x19 /* 
> BPF_??? */, prog_btf_fd=5, func_info_rec_size=0, func_info=NULL, 
> func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, 
> ...}, 120) = -1 EPERM (Operation not permitted)
> libbpf: load bpf program failed: Operation not permitted
> libbpf: failed to load program 'fexit/xdp_prog_simple'
> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
> ERROR: Failed to load object file: Operation not permitted
> +++ exited with 0 +++
> 
> And with full libbpf print debugging:
> 
> libbpf: loading ./xdp_sample_fentry_fexit_kern.o
> libbpf: section(1) .strtab, size 250, link 0, flags 0, type=3
> libbpf: skip section(1) .strtab
> libbpf: section(2) .text, size 0, link 0, flags 6, type=1
> libbpf: skip section(2) .text
> libbpf: section(3) fexit/xdp_prog_simple, size 240, link 0, flags 6, type=1
> libbpf: found program fexit/xdp_prog_simple
> libbpf: section(4) license, size 4, link 0, flags 3, type=1
> libbpf: license of ./xdp_sample_fentry_fexit_kern.o is GPL
> libbpf: section(5) .rodata.str1.1, size 52, link 0, flags 32, type=1
> libbpf: skip section(5) .rodata.str1.1
> libbpf: section(6) .debug_str, size 344, link 0, flags 30, type=1
> libbpf: skip section(6) .debug_str
> libbpf: section(7) .debug_loc, size 70, link 0, flags 0, type=1
> libbpf: skip section(7) .debug_loc
> libbpf: section(8) .debug_abbrev, size 297, link 0, flags 0, type=1
> libbpf: skip section(8) .debug_abbrev
> libbpf: section(9) .debug_info, size 403, link 0, flags 0, type=1
> libbpf: skip section(9) .debug_info
> libbpf: section(10) .rel.debug_info, size 528, link 21, flags 0, type=9
> libbpf: skip relo .rel.debug_info(10) for section(9)
> libbpf: section(11) .debug_ranges, size 96, link 0, flags 0, type=1
> libbpf: skip section(11) .debug_ranges
> libbpf: section(12) .debug_macinfo, size 1, link 0, flags 0, type=1
> libbpf: skip section(12) .debug_macinfo
> libbpf: section(13) .BTF, size 534, link 0, flags 0, type=1
> libbpf: section(14) .rel.BTF, size 16, link 21, flags 0, type=9
> libbpf: skip relo .rel.BTF(14) for section(13)
> libbpf: section(15) .BTF.ext, size 152, link 0, flags 0, type=1
> libbpf: section(16) .rel.BTF.ext, size 96, link 21, flags 0, type=9
> libbpf: skip relo .rel.BTF.ext(16) for section(15)
> libbpf: section(17) .debug_frame, size 40, link 0, flags 0, type=1
> libbpf: skip section(17) .debug_frame
> libbpf: section(18) .rel.debug_frame, size 32, link 21, flags 0, type=9
> libbpf: skip relo .rel.debug_frame(18) for section(17)
> libbpf: section(19) .debug_line, size 206, link 0, flags 0, type=1
> libbpf: skip section(19) .debug_line
> libbpf: section(20) .rel.debug_line, size 16, link 21, flags 0, type=9
> libbpf: skip relo .rel.debug_line(20) for section(19)
> libbpf: section(21) .symtab, size 792, link 1, flags 0, type=2
> libbpf: Cannot find bpf_func_info for main program sec 
> fexit/xdp_prog_simple. Ignore all bpf_func_info.
> libbpf: load bpf program failed: Operation not permitted
> libbpf: failed to load program 'fexit/xdp_prog_simple'
> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
> ERROR: Failed to load object file: Operation not permitted
> 
> I’m using the latest pahole, but standard llvm 9.0.0 from Fedora.
> 
> So I decided to at least run the self-test and see if this passes, and 
> work my way from there.
> But it’s failing to build with errors like “use of unknown builtin 
> '__builtin_preserve_field_info’”, so I’m assuming my clang9 needs 
> upgrading…

Yes, please try latest llvm trunk. You can find the information on the 
web. Below is the instruction from iovisor/bcc repo.
(https://github.com/iovisor/bcc/blob/master/INSTALL.md#older-instructions)

git clone https://github.com/llvm/llvm-project.git
mkdir -p llvm-project/llvm/build/install
cd llvm-project/llvm/build
cmake -G "Ninja" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
   -DLLVM_ENABLE_PROJECTS="clang" \
   -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
ninja && ninja install
export PATH=$PWD/install/bin:$PATH

> 
> I quickly tried doing this while working on other stuff, but I failed to 
> multitask, so will continue with this next week.
> 
> Any ideas are welcome…
> 
> Cheers,
> 
> 
> Eelco
> 

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 29 Nov 2019, at 17:52, Yonghong Song wrote:

> On 11/29/19 8:30 AM, Eelco Chaudron wrote:
>>
>>
>> On 28 Nov 2019, at 20:47, Alexei Starovoitov wrote:
>>
>>> On Thu, Nov 28, 2019 at 11:16 AM Eelco Chaudron 
>>> <echaudro@redhat.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:
>>>>
>>>>> On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron 
>>>>> <echaudro@redhat.com>
>>>>> wrote:
>>>>>>
>>>>>> Trying out the BPF trace to trace a BPF program, but I’m 
>>>>>> already
>>>>>> getting stuck loading the object with the fexit  :(
>>>>>
>>>>> I can take a look after holidays.
>>>>
>>>> Enjoy the Holidays!! I figured out my auto kernel install script 
>>>> failed
>>>> whiteout me noticing, and I was running an old kernel :(
>>>>
>>>> I will try tomorrow with the correct kernel…
>>>
>>> Please also check that you have the latest llvm and pahole.
>>> pahole version should be >= 1.13.
>>> clang ideally from master.
>>> If all that is working then downgrade one by one and bisect whether
>>> the bug is.
>>>
>>
>> I tried it with the latest kernel loaded but still I got some errors 
>> on
>> bpf_object__load_xattr():
>>
>> $ sudo ./xdp_sample_fentry_fexit_user
>> libbpf: Cannot find bpf_func_info for main program sec
>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>> libbpf: load bpf program failed: Operation not permitted
>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>> ERROR: Failed to load object file: Operation not permitted
>>
>>
>> With strace:
>>
>> [vagrant@xdp-tutorial tracing05-xdp-fentry]$ sudo
>> /home/vagrant/strace/strace -e bpf ./xdp_sample_fentry_fexit_user
>> bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=36, next_id=0, open_flags=0}, 
>> 120) = 3
>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, 
>> insn_cnt=2,
>> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0,
>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>> prog_name="", prog_ifindex=0,
>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, 
>> insn_cnt=2,
>> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0,
>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>> prog_name="test", prog_ifindex=0,
>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
>> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4,
>> value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, 
>> map_name="",
>> map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0}, 
>> 120) = 5
>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, 
>> insn_cnt=5,
>> insns=0x7ffcaedfd490, license="GPL", log_level=0, log_size=0,
>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>> prog_name="", prog_ifindex=0,
>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 6
>> bpf(BPF_BTF_LOAD,
>> {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"...,
>> btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) 
>> = 5
>> bpf(BPF_BTF_LOAD,
>> {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"...,
>> btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 120) 
>> = 5
>> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4,
>> value_size=4, max_entries=1, map_flags=0x400 /* BPF_F_??? */,
>> inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, 
>> btf_key_type_id=0,
>> btf_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
>> bpf(BPF_BTF_LOAD,
>> {btf="\237\353\1\0\30\0\0\0\0\0\0\0`\1\0\0`\1\0\0\236\0\0\0\0\0\0\0\0\0\0\3"...,
>> btf_log_buf=NULL, btf_size=534, btf_log_size=0, btf_log_level=0}, 
>> 120) = 5
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>> info=0x7ffcaedfd4a0}}, 120) = 0
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>> info=0x2018c30}}, 120) = 0
>> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=3}, 120) = 4
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=16,
>> info=0x7ffcaedfd5a0}}, 120) = 0
>> libbpf: Cannot find bpf_func_info for main program sec
>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, 
>> insn_cnt=30,
>> insns=0x20191f0, license="GPL", log_level=7, log_size=16777215,
>> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>> prog_name="trace_on_exit", prog_ifindex=0, expected_attach_type=0x19 
>> /*
>> BPF_??? */, prog_btf_fd=5, func_info_rec_size=0, func_info=NULL,
>> func_info_cnt=0, line_info_rec_size=0, line_info=NULL, 
>> line_info_cnt=0,
>> ...}, 120) = -1 EPERM (Operation not permitted)
>> libbpf: load bpf program failed: Operation not permitted
>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>> ERROR: Failed to load object file: Operation not permitted
>> +++ exited with 0 +++
>>
>> And with full libbpf print debugging:
>>
>> libbpf: loading ./xdp_sample_fentry_fexit_kern.o
>> libbpf: section(1) .strtab, size 250, link 0, flags 0, type=3
>> libbpf: skip section(1) .strtab
>> libbpf: section(2) .text, size 0, link 0, flags 6, type=1
>> libbpf: skip section(2) .text
>> libbpf: section(3) fexit/xdp_prog_simple, size 240, link 0, flags 6, 
>> type=1
>> libbpf: found program fexit/xdp_prog_simple
>> libbpf: section(4) license, size 4, link 0, flags 3, type=1
>> libbpf: license of ./xdp_sample_fentry_fexit_kern.o is GPL
>> libbpf: section(5) .rodata.str1.1, size 52, link 0, flags 32, type=1
>> libbpf: skip section(5) .rodata.str1.1
>> libbpf: section(6) .debug_str, size 344, link 0, flags 30, type=1
>> libbpf: skip section(6) .debug_str
>> libbpf: section(7) .debug_loc, size 70, link 0, flags 0, type=1
>> libbpf: skip section(7) .debug_loc
>> libbpf: section(8) .debug_abbrev, size 297, link 0, flags 0, type=1
>> libbpf: skip section(8) .debug_abbrev
>> libbpf: section(9) .debug_info, size 403, link 0, flags 0, type=1
>> libbpf: skip section(9) .debug_info
>> libbpf: section(10) .rel.debug_info, size 528, link 21, flags 0, 
>> type=9
>> libbpf: skip relo .rel.debug_info(10) for section(9)
>> libbpf: section(11) .debug_ranges, size 96, link 0, flags 0, type=1
>> libbpf: skip section(11) .debug_ranges
>> libbpf: section(12) .debug_macinfo, size 1, link 0, flags 0, type=1
>> libbpf: skip section(12) .debug_macinfo
>> libbpf: section(13) .BTF, size 534, link 0, flags 0, type=1
>> libbpf: section(14) .rel.BTF, size 16, link 21, flags 0, type=9
>> libbpf: skip relo .rel.BTF(14) for section(13)
>> libbpf: section(15) .BTF.ext, size 152, link 0, flags 0, type=1
>> libbpf: section(16) .rel.BTF.ext, size 96, link 21, flags 0, type=9
>> libbpf: skip relo .rel.BTF.ext(16) for section(15)
>> libbpf: section(17) .debug_frame, size 40, link 0, flags 0, type=1
>> libbpf: skip section(17) .debug_frame
>> libbpf: section(18) .rel.debug_frame, size 32, link 21, flags 0, 
>> type=9
>> libbpf: skip relo .rel.debug_frame(18) for section(17)
>> libbpf: section(19) .debug_line, size 206, link 0, flags 0, type=1
>> libbpf: skip section(19) .debug_line
>> libbpf: section(20) .rel.debug_line, size 16, link 21, flags 0, 
>> type=9
>> libbpf: skip relo .rel.debug_line(20) for section(19)
>> libbpf: section(21) .symtab, size 792, link 1, flags 0, type=2
>> libbpf: Cannot find bpf_func_info for main program sec
>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>> libbpf: load bpf program failed: Operation not permitted
>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>> ERROR: Failed to load object file: Operation not permitted
>>
>> I’m using the latest pahole, but standard llvm 9.0.0 from Fedora.

Same error with latest llvm

>> So I decided to at least run the self-test and see if this passes, 
>> and
>> work my way from there.
>> But it’s failing to build with errors like “use of unknown 
>> builtin
>> '__builtin_preserve_field_info’”, so I’m assuming my clang9 
>> needs
>> upgrading…
>
> Yes, please try latest llvm trunk. You can find the information on the
> web. Below is the instruction from iovisor/bcc repo.
> (https://github.com/iovisor/bcc/blob/master/INSTALL.md#older-instructions)
>
> git clone https://github.com/llvm/llvm-project.git
> mkdir -p llvm-project/llvm/build/install
> cd llvm-project/llvm/build
> cmake -G "Ninja" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
>    -DLLVM_ENABLE_PROJECTS="clang" \
>    -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
> ninja && ninja install
> export PATH=$PWD/install/bin:$PATH
>

I tried compiling with the latest master llvm, but the selftest failed 
for bpf2bpf (other fail to sudo ./test_progs, Summary: 44/138 PASSED, 1 
SKIPPED, 16 FAILED)

#5 core_reloc:FAIL
test_fentry_fexit:PASS:prog_load sched cls 32573 nsec
libbpf: failed to find valid kernel BTF
libbpf: vmlinux BTF is not found
libbpf: fentry/bpf_fentry_test1 is not found in vmlinux BTF
test_fentry_fexit:FAIL:prog_load fail err -2 errno 2
#6 fentry_fexit:FAIL
test_fentry_test:PASS:prog_load sched cls 0 nsec
libbpf: failed to find valid kernel BTF
libbpf: vmlinux BTF is not found
libbpf: fentry/bpf_fentry_test1 is not found in vmlinux BTF
test_fentry_test:FAIL:prog_load fail err -2 errno 2
#7 fentry_test:FAIL
test_fexit_bpf2bpf:PASS:prog_load sched cls 32764 nsec
test_fexit_bpf2bpf:PASS:obj_open 32764 nsec
libbpf: failed to find valid kernel BTF
libbpf: failed to get target BTF: -3
libbpf: failed to perform CO-RE relocations: -3
libbpf: failed to load object './fexit_bpf2bpf.o'
test_fexit_bpf2bpf:FAIL:obj_load err -3
#8 fexit_bpf2bpf:FAIL

Is there more I need to re-compile other than the test directory?
Or do I need additional configure flags?

$ grep -E -i "btf|bpf" .config
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_BPFILTER is not set
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_LIRC_MODE2=y
CONFIG_VIDEO_SONY_BTF_MPX=m
# CONFIG_DEBUG_INFO_BTF is not set
CONFIG_BPF_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
# CONFIG_TEST_BPF is not set

Cheers,

Eelco

Re: Trying the bpf trace a bpf xdp program

[Yonghong Song]

On 12/2/19 8:34 AM, Eelco Chaudron wrote:
> 
> 
> On 29 Nov 2019, at 17:52, Yonghong Song wrote:
> 
>> On 11/29/19 8:30 AM, Eelco Chaudron wrote:
>>>
>>>
>>> On 28 Nov 2019, at 20:47, Alexei Starovoitov wrote:
>>>
>>>> On Thu, Nov 28, 2019 at 11:16 AM Eelco Chaudron <echaudro@redhat.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 28 Nov 2019, at 19:18, Alexei Starovoitov wrote:
>>>>>
>>>>>> On Thu, Nov 28, 2019 at 9:20 AM Eelco Chaudron <echaudro@redhat.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Trying out the BPF trace to trace a BPF program, but I’m already
>>>>>>> getting stuck loading the object with the fexit  :(
>>>>>>
>>>>>> I can take a look after holidays.
>>>>>
>>>>> Enjoy the Holidays!! I figured out my auto kernel install script 
>>>>> failed
>>>>> whiteout me noticing, and I was running an old kernel :(
>>>>>
>>>>> I will try tomorrow with the correct kernel…
>>>>
>>>> Please also check that you have the latest llvm and pahole.
>>>> pahole version should be >= 1.13.
>>>> clang ideally from master.
>>>> If all that is working then downgrade one by one and bisect whether
>>>> the bug is.
>>>>
>>>
>>> I tried it with the latest kernel loaded but still I got some errors on
>>> bpf_object__load_xattr():
>>>
>>> $ sudo ./xdp_sample_fentry_fexit_user
>>> libbpf: Cannot find bpf_func_info for main program sec
>>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>>> libbpf: load bpf program failed: Operation not permitted
>>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>>> ERROR: Failed to load object file: Operation not permitted
>>>
>>>
>>> With strace:
>>>
>>> [vagrant@xdp-tutorial tracing05-xdp-fentry]$ sudo
>>> /home/vagrant/strace/strace -e bpf ./xdp_sample_fentry_fexit_user
>>> bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=36, next_id=0, open_flags=0}, 
>>> 120) = 3
>>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2,
>>> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0,
>>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="", prog_ifindex=0,
>>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
>>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2,
>>> insns=0x7ffcaedfd4f0, license="GPL", log_level=0, log_size=0,
>>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="test", prog_ifindex=0,
>>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 5
>>> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4,
>>> value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="",
>>> map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0}, 
>>> 120) = 5
>>> bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5,
>>> insns=0x7ffcaedfd490, license="GPL", log_level=0, log_size=0,
>>> log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="", prog_ifindex=0,
>>> expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0,
>>> func_info_rec_size=0, func_info=NULL, func_info_cnt=0,
>>> line_info_rec_size=0, line_info=NULL, line_info_cnt=0}, 120) = 6
>>> bpf(BPF_BTF_LOAD,
>>> {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., 
>>>
>>> btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) 
>>> = 5
>>> bpf(BPF_BTF_LOAD,
>>> {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"..., 
>>>
>>> btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 120) 
>>> = 5
>>> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4,
>>> value_size=4, max_entries=1, map_flags=0x400 /* BPF_F_??? */,
>>> inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0,
>>> btf_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
>>> bpf(BPF_BTF_LOAD,
>>> {btf="\237\353\1\0\30\0\0\0\0\0\0\0`\1\0\0`\1\0\0\236\0\0\0\0\0\0\0\0\0\0\3"..., 
>>>
>>> btf_log_buf=NULL, btf_size=534, btf_log_size=0, btf_log_level=0}, 
>>> 120) = 5
>>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>>> info=0x7ffcaedfd4a0}}, 120) = 0
>>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>>> info=0x2018c30}}, 120) = 0
>>> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=3}, 120) = 4
>>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=16,
>>> info=0x7ffcaedfd5a0}}, 120) = 0
>>> libbpf: Cannot find bpf_func_info for main program sec
>>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>>> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=30,
>>> insns=0x20191f0, license="GPL", log_level=7, log_size=16777215,
>>> log_buf="", kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0,
>>> prog_name="trace_on_exit", prog_ifindex=0, expected_attach_type=0x19 /*
>>> BPF_??? */, prog_btf_fd=5, func_info_rec_size=0, func_info=NULL,
>>> func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0,
>>> ...}, 120) = -1 EPERM (Operation not permitted)
>>> libbpf: load bpf program failed: Operation not permitted
>>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>>> ERROR: Failed to load object file: Operation not permitted
>>> +++ exited with 0 +++
>>>
>>> And with full libbpf print debugging:
>>>
>>> libbpf: loading ./xdp_sample_fentry_fexit_kern.o
>>> libbpf: section(1) .strtab, size 250, link 0, flags 0, type=3
>>> libbpf: skip section(1) .strtab
>>> libbpf: section(2) .text, size 0, link 0, flags 6, type=1
>>> libbpf: skip section(2) .text
>>> libbpf: section(3) fexit/xdp_prog_simple, size 240, link 0, flags 6, 
>>> type=1
>>> libbpf: found program fexit/xdp_prog_simple
>>> libbpf: section(4) license, size 4, link 0, flags 3, type=1
>>> libbpf: license of ./xdp_sample_fentry_fexit_kern.o is GPL
>>> libbpf: section(5) .rodata.str1.1, size 52, link 0, flags 32, type=1
>>> libbpf: skip section(5) .rodata.str1.1
>>> libbpf: section(6) .debug_str, size 344, link 0, flags 30, type=1
>>> libbpf: skip section(6) .debug_str
>>> libbpf: section(7) .debug_loc, size 70, link 0, flags 0, type=1
>>> libbpf: skip section(7) .debug_loc
>>> libbpf: section(8) .debug_abbrev, size 297, link 0, flags 0, type=1
>>> libbpf: skip section(8) .debug_abbrev
>>> libbpf: section(9) .debug_info, size 403, link 0, flags 0, type=1
>>> libbpf: skip section(9) .debug_info
>>> libbpf: section(10) .rel.debug_info, size 528, link 21, flags 0, type=9
>>> libbpf: skip relo .rel.debug_info(10) for section(9)
>>> libbpf: section(11) .debug_ranges, size 96, link 0, flags 0, type=1
>>> libbpf: skip section(11) .debug_ranges
>>> libbpf: section(12) .debug_macinfo, size 1, link 0, flags 0, type=1
>>> libbpf: skip section(12) .debug_macinfo
>>> libbpf: section(13) .BTF, size 534, link 0, flags 0, type=1
>>> libbpf: section(14) .rel.BTF, size 16, link 21, flags 0, type=9
>>> libbpf: skip relo .rel.BTF(14) for section(13)
>>> libbpf: section(15) .BTF.ext, size 152, link 0, flags 0, type=1
>>> libbpf: section(16) .rel.BTF.ext, size 96, link 21, flags 0, type=9
>>> libbpf: skip relo .rel.BTF.ext(16) for section(15)
>>> libbpf: section(17) .debug_frame, size 40, link 0, flags 0, type=1
>>> libbpf: skip section(17) .debug_frame
>>> libbpf: section(18) .rel.debug_frame, size 32, link 21, flags 0, type=9
>>> libbpf: skip relo .rel.debug_frame(18) for section(17)
>>> libbpf: section(19) .debug_line, size 206, link 0, flags 0, type=1
>>> libbpf: skip section(19) .debug_line
>>> libbpf: section(20) .rel.debug_line, size 16, link 21, flags 0, type=9
>>> libbpf: skip relo .rel.debug_line(20) for section(19)
>>> libbpf: section(21) .symtab, size 792, link 1, flags 0, type=2
>>> libbpf: Cannot find bpf_func_info for main program sec
>>> fexit/xdp_prog_simple. Ignore all bpf_func_info.
>>> libbpf: load bpf program failed: Operation not permitted
>>> libbpf: failed to load program 'fexit/xdp_prog_simple'
>>> libbpf: failed to load object './xdp_sample_fentry_fexit_kern.o'
>>> ERROR: Failed to load object file: Operation not permitted
>>>
>>> I’m using the latest pahole, but standard llvm 9.0.0 from Fedora.
> 
> Same error with latest llvm
> 
>>> So I decided to at least run the self-test and see if this passes, and
>>> work my way from there.
>>> But it’s failing to build with errors like “use of unknown builtin
>>> '__builtin_preserve_field_info’”, so I’m assuming my clang9 needs
>>> upgrading…
>>
>> Yes, please try latest llvm trunk. You can find the information on the
>> web. Below is the instruction from iovisor/bcc repo.
>> (https://github.com/iovisor/bcc/blob/master/INSTALL.md#older-instructions) 
>>
>>
>> git clone https://github.com/llvm/llvm-project.git
>> mkdir -p llvm-project/llvm/build/install
>> cd llvm-project/llvm/build
>> cmake -G "Ninja" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
>>    -DLLVM_ENABLE_PROJECTS="clang" \
>>    -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
>> ninja && ninja install
>> export PATH=$PWD/install/bin:$PATH
>>
> 
> I tried compiling with the latest master llvm, but the selftest failed 
> for bpf2bpf (other fail to sudo ./test_progs, Summary: 44/138 PASSED, 1 
> SKIPPED, 16 FAILED)
> 
> #5 core_reloc:FAIL
> test_fentry_fexit:PASS:prog_load sched cls 32573 nsec
> libbpf: failed to find valid kernel BTF
> libbpf: vmlinux BTF is not found
> libbpf: fentry/bpf_fentry_test1 is not found in vmlinux BTF
> test_fentry_fexit:FAIL:prog_load fail err -2 errno 2
> #6 fentry_fexit:FAIL
> test_fentry_test:PASS:prog_load sched cls 0 nsec
> libbpf: failed to find valid kernel BTF
> libbpf: vmlinux BTF is not found
> libbpf: fentry/bpf_fentry_test1 is not found in vmlinux BTF
> test_fentry_test:FAIL:prog_load fail err -2 errno 2
> #7 fentry_test:FAIL
> test_fexit_bpf2bpf:PASS:prog_load sched cls 32764 nsec
> test_fexit_bpf2bpf:PASS:obj_open 32764 nsec
> libbpf: failed to find valid kernel BTF
> libbpf: failed to get target BTF: -3
> libbpf: failed to perform CO-RE relocations: -3
> libbpf: failed to load object './fexit_bpf2bpf.o'
> test_fexit_bpf2bpf:FAIL:obj_load err -3
> #8 fexit_bpf2bpf:FAIL
> 
> Is there more I need to re-compile other than the test directory?
> Or do I need additional configure flags?
> 
> $ grep -E -i "btf|bpf" .config
> CONFIG_CGROUP_BPF=y
> CONFIG_BPF=y
> CONFIG_BPF_SYSCALL=y
> CONFIG_BPF_JIT_ALWAYS_ON=y
> CONFIG_IPV6_SEG6_BPF=y
> CONFIG_NETFILTER_XT_MATCH_BPF=m
> # CONFIG_BPFILTER is not set
> CONFIG_NET_CLS_BPF=m
> CONFIG_NET_ACT_BPF=m
> CONFIG_BPF_JIT=y
> CONFIG_BPF_STREAM_PARSER=y
> CONFIG_LWTUNNEL_BPF=y
> CONFIG_HAVE_EBPF_JIT=y
> CONFIG_BPF_LIRC_MODE2=y
> CONFIG_VIDEO_SONY_BTF_MPX=m
> # CONFIG_DEBUG_INFO_BTF is not set

You need to build the kernel with
   CONFIG_DEBUG_INFO_BTF=y
Make sure on the build machine you have recent pahole version >= 1.13.


> CONFIG_BPF_EVENTS=y
> # CONFIG_BPF_KPROBE_OVERRIDE is not set
> # CONFIG_TEST_BPF is not set
> 
> Cheers,
> 
> Eelco
> 
> 

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 2 Dec 2019, at 17:48, Yonghong Song wrote:

> On 12/2/19 8:34 AM, Eelco Chaudron wrote:
>> On 29 Nov 2019, at 17:52, Yonghong Song wrote:

<SNIP>
>
> You need to build the kernel with
>    CONFIG_DEBUG_INFO_BTF=y
> Make sure on the build machine you have recent pahole version >= 1.13.

With the latest LLVM and CONFIG_DEBUG_INFO_BTF=y the self-test for 
bpf2bpf is passing!


However I still have problems with my code, which is getting to the next 
step, but no my program is killed when trying to load the eBPG fexit 
code. If I replace my generated eBPF programs for the once generated by 
the self-test (test_pkt_access.o/fexit_bpf2bpf.o) it works fine.


I decided to build my objects just like the example programs (so have a 
hacked build.sh file) but I get the same results. I.e. being killed by 
the kernel:

bpf(BPF_BTF_LOAD, 
{btf="\237\353\1\0\30\0\0\0\0\0\0\0\330\0\0\0\330\0\0\0\244\0\0\0\0\0\0\0\0\0\0\2"..., 
btf_log_buf=NULL, btf_size=404, btf_log_size=0, btf_log_level=0}, 120) = 
6
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
info=0x7ffdfbdac3b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
info=0xafb600}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=90}, 120) = 5
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=16, 
info=0x7ffdfbdac4b0}}, 120) = 0
- Opened object file: 0xafb440
bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2, 
insns=0xafbaa0, license="GPL", log_level=7, log_size=16777215, 
log_buf="\237\353\1", kern_version=KERNEL_VERSION(0, 0, 0), 
prog_flags=0, prog_name="test_main", prog_ifindex=0, 
expected_attach_type=0x19 /* BPF_??? */, prog_btf_fd=6, 
func_info_rec_size=8, func_info=0xafb9f0, func_info_cnt=1, 
line_info_rec_size=16, line_info=0xafba10, line_info_cnt=1, ...}, 120
) = ?
+++ killed by SIGKILL +++
Killed


[79162.619208] BUG: kernel NULL pointer dereference, address: 
0000000000000000
[79162.619906] #PF: supervisor read access in kernel mode
[79162.620582] #PF: error_code(0x0000) - not-present page
[79162.621255] PGD 80000001e2409067 P4D 80000001e2409067 PUD 22eba9067 
PMD 0
[79162.621933] Oops: 0000 [#12] SMP PTI
[79162.622599] CPU: 5 PID: 3191 Comm: xdp_sample_fent Tainted: G      D  
          5.4.0+ #3
[79162.623274] Hardware name: Red Hat KVM, BIOS 
1.11.1-3.module+el8+2529+a9686a4d 04/01/2014
[79162.623962] RIP: 0010:bpf_check+0x1648/0x250b
[79162.624650] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17 
0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40 
68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
[79162.626088] RSP: 0018:ffffb5f6c07c3c88 EFLAGS: 00010293
[79162.626822] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
ffffb5f6c07c3c40
[79162.627560] RDX: ffffa0a1e6e01818 RSI: 00000000fffffffa RDI: 
0000000000000000
[79162.628304] RBP: ffffb5f6c07c3d70 R08: 000000000000000e R09: 
ffffa0a1f5c9dc90
[79162.629053] R10: ffffa0a1f5c9dc80 R11: ffffa0a1e6e0199a R12: 
ffffa0a1eac48000
[79162.629806] R13: 0000000000000000 R14: ffffb5f6c043e000 R15: 
ffffb5f6c033f000
[79162.630562] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000) 
knlGS:0000000000000000
[79162.631324] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[79162.632072] CR2: 0000000000000000 CR3: 00000001e242a005 CR4: 
0000000000360ee0
[79162.632813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[79162.633539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[79162.634255] Call Trace:
[79162.634974]  ? _cond_resched+0x15/0x30
[79162.635686]  ? kmem_cache_alloc_trace+0x162/0x220
[79162.636398]  ? selinux_bpf_prog_alloc+0x1f/0x60
[79162.637111]  bpf_prog_load+0x3de/0x690
[79162.637809]  __do_sys_bpf+0x105/0x1740
[79162.638488]  do_syscall_64+0x5b/0x180
[79162.639147]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[79162.639792] RIP: 0033:0x7f560c3fe1ad
[79162.640415] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 
48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ab 5c 0c 00 f7 d8 64 89 01 48
[79162.641703] RSP: 002b:00007ffdfbdac318 EFLAGS: 00000202 ORIG_RAX: 
0000000000000141
[79162.642363] RAX: ffffffffffffffda RBX: 0000000000afb440 RCX: 
00007f560c3fe1ad
[79162.643026] RDX: 0000000000000078 RSI: 00007ffdfbdac390 RDI: 
0000000000000005
[79162.643676] RBP: 00007ffdfbdac330 R08: 0000000000afba70 R09: 
00007ffdfbdac390
[79162.644310] R10: 0000000000afcf10 R11: 0000000000000202 R12: 
0000000000402690
[79162.644935] R13: 00007ffdfbdac790 R14: 0000000000000000 R15: 
0000000000000000
[79162.645559] Modules linked in: ip6t_REJECT nf_reject_ipv6 
ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat 
ebtable_broute ip6table_nat ip6table_mangle ip6table_raw 
ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw 
iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set 
nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cirrus drm_kms_helper 
virtio_net net_failover joydev drm failover i2c_piix4 virtio_balloon 
pcspkr ip_tables xfs libcrc32c crc32c_intel ata_generic floppy 
virtio_scsi serio_raw pata_acpi qemu_fw_cfg
[79162.649591] CR2: 0000000000000000
[79162.650272] ---[ end trace 5119c5364c1e9c83 ]---
[79162.650957] RIP: 0010:bpf_check+0x1648/0x250b
[79162.651646] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17 
0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40 
68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
[79162.653081] RSP: 0018:ffffb5f6c072bc88 EFLAGS: 00010293
[79162.653807] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
ffffb5f6c072bc40
[79162.654536] RDX: ffffa0a1e76b1418 RSI: 00000000fffffffa RDI: 
0000000000000000
[79162.655270] RBP: ffffb5f6c072bd70 R08: 000000000000000e R09: 
ffffa0a1e4d3fa90
[79162.655996] R10: ffffa0a1e4d3fa80 R11: ffffa0a1e76b159a R12: 
ffffa0a1eac7c000
[79162.656715] R13: 0000000000000000 R14: ffffb5f6c01e3000 R15: 
ffffb5f6c015f000
[79162.657429] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000) 
knlGS:0000000000000000
[79162.658137] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[79162.658826] CR2: 0000000000000000 CR3: 00000001e242a005 CR4: 
0000000000360ee0
[79162.659515] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[79162.660196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400


I’ve put my code on GitHub, maybe it’s just something stupid…

https://github.com/chaudron/bpf2bpf-tracing


Cheers,

Eelco


PS: If I run the latest pahole (v1.15) on the .o files, I get the 
following libbpf error: “libbpf: Cannot find bpf_func_info for main 
program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”

Re: Trying the bpf trace a bpf xdp program

[Yonghong Song]

On 12/4/19 5:19 AM, Eelco Chaudron wrote:
> 
> 
> On 2 Dec 2019, at 17:48, Yonghong Song wrote:
> 
>> On 12/2/19 8:34 AM, Eelco Chaudron wrote:
>>> On 29 Nov 2019, at 17:52, Yonghong Song wrote:
> 
> <SNIP>
>>
>> You need to build the kernel with
>>    CONFIG_DEBUG_INFO_BTF=y
>> Make sure on the build machine you have recent pahole version >= 1.13.
> 
> With the latest LLVM and CONFIG_DEBUG_INFO_BTF=y the self-test for 
> bpf2bpf is passing!

Great!

> 
> However I still have problems with my code, which is getting to the next 
> step, but no my program is killed when trying to load the eBPG fexit 
> code. If I replace my generated eBPF programs for the once generated by 
> the self-test (test_pkt_access.o/fexit_bpf2bpf.o) it works fine.
> 
> 
> I decided to build my objects just like the example programs (so have a 
> hacked build.sh file) but I get the same results. I.e. being killed by 
> the kernel:
> 
> bpf(BPF_BTF_LOAD, 
> {btf="\237\353\1\0\30\0\0\0\0\0\0\0\330\0\0\0\330\0\0\0\244\0\0\0\0\0\0\0\0\0\0\2"..., 
> btf_log_buf=NULL, btf_size=404, btf_log_size=0, btf_log_level=0}, 120) = 6
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
> info=0x7ffdfbdac3b0}}, 120) = 0
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208, 
> info=0xafb600}}, 120) = 0
> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=90}, 120) = 5
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=16, 
> info=0x7ffdfbdac4b0}}, 120) = 0
> - Opened object file: 0xafb440
> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2, 
> insns=0xafbaa0, license="GPL", log_level=7, log_size=16777215, 
> log_buf="\237\353\1", kern_version=KERNEL_VERSION(0, 0, 0), 
> prog_flags=0, prog_name="test_main", prog_ifindex=0, 
> expected_attach_type=0x19 /* BPF_??? */, prog_btf_fd=6, 
> func_info_rec_size=8, func_info=0xafb9f0, func_info_cnt=1, 
> line_info_rec_size=16, line_info=0xafba10, line_info_cnt=1, ...}, 120
> ) = ?
> +++ killed by SIGKILL +++
> Killed
> 
> 
> [79162.619208] BUG: kernel NULL pointer dereference, address: 

This should be a kernel bug. I will take a look at it today.

> 0000000000000000
> [79162.619906] #PF: supervisor read access in kernel mode
> [79162.620582] #PF: error_code(0x0000) - not-present page
> [79162.621255] PGD 80000001e2409067 P4D 80000001e2409067 PUD 22eba9067 
> PMD 0
> [79162.621933] Oops: 0000 [#12] SMP PTI
> [79162.622599] CPU: 5 PID: 3191 Comm: xdp_sample_fent Tainted: G      D 
>           5.4.0+ #3
> [79162.623274] Hardware name: Red Hat KVM, BIOS 
> 1.11.1-3.module+el8+2529+a9686a4d 04/01/2014
> [79162.623962] RIP: 0010:bpf_check+0x1648/0x250b
> [79162.624650] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17 
> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40 
> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
> [79162.626088] RSP: 0018:ffffb5f6c07c3c88 EFLAGS: 00010293
> [79162.626822] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
> ffffb5f6c07c3c40
> [79162.627560] RDX: ffffa0a1e6e01818 RSI: 00000000fffffffa RDI: 
> 0000000000000000
> [79162.628304] RBP: ffffb5f6c07c3d70 R08: 000000000000000e R09: 
> ffffa0a1f5c9dc90
> [79162.629053] R10: ffffa0a1f5c9dc80 R11: ffffa0a1e6e0199a R12: 
> ffffa0a1eac48000
> [79162.629806] R13: 0000000000000000 R14: ffffb5f6c043e000 R15: 
> ffffb5f6c033f000
> [79162.630562] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000) 
> knlGS:0000000000000000
> [79162.631324] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [79162.632072] CR2: 0000000000000000 CR3: 00000001e242a005 CR4: 
> 0000000000360ee0
> [79162.632813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [79162.633539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> [79162.634255] Call Trace:
> [79162.634974]  ? _cond_resched+0x15/0x30
> [79162.635686]  ? kmem_cache_alloc_trace+0x162/0x220
> [79162.636398]  ? selinux_bpf_prog_alloc+0x1f/0x60
> [79162.637111]  bpf_prog_load+0x3de/0x690
> [79162.637809]  __do_sys_bpf+0x105/0x1740
> [79162.638488]  do_syscall_64+0x5b/0x180
> [79162.639147]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [79162.639792] RIP: 0033:0x7f560c3fe1ad
> [79162.640415] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 
> 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ab 5c 0c 00 f7 d8 64 89 01 48
> [79162.641703] RSP: 002b:00007ffdfbdac318 EFLAGS: 00000202 ORIG_RAX: 
> 0000000000000141
> [79162.642363] RAX: ffffffffffffffda RBX: 0000000000afb440 RCX: 
> 00007f560c3fe1ad
> [79162.643026] RDX: 0000000000000078 RSI: 00007ffdfbdac390 RDI: 
> 0000000000000005
> [79162.643676] RBP: 00007ffdfbdac330 R08: 0000000000afba70 R09: 
> 00007ffdfbdac390
> [79162.644310] R10: 0000000000afcf10 R11: 0000000000000202 R12: 
> 0000000000402690
> [79162.644935] R13: 00007ffdfbdac790 R14: 0000000000000000 R15: 
> 0000000000000000
> [79162.645559] Modules linked in: ip6t_REJECT nf_reject_ipv6 
> ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat 
> ebtable_broute ip6table_nat ip6table_mangle ip6table_raw 
> ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw 
> iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set 
> nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables 
> iptable_filter intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass 
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cirrus drm_kms_helper 
> virtio_net net_failover joydev drm failover i2c_piix4 virtio_balloon 
> pcspkr ip_tables xfs libcrc32c crc32c_intel ata_generic floppy 
> virtio_scsi serio_raw pata_acpi qemu_fw_cfg
> [79162.649591] CR2: 0000000000000000
> [79162.650272] ---[ end trace 5119c5364c1e9c83 ]---
> [79162.650957] RIP: 0010:bpf_check+0x1648/0x250b
> [79162.651646] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17 
> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40 
> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
> [79162.653081] RSP: 0018:ffffb5f6c072bc88 EFLAGS: 00010293
> [79162.653807] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 
> ffffb5f6c072bc40
> [79162.654536] RDX: ffffa0a1e76b1418 RSI: 00000000fffffffa RDI: 
> 0000000000000000
> [79162.655270] RBP: ffffb5f6c072bd70 R08: 000000000000000e R09: 
> ffffa0a1e4d3fa90
> [79162.655996] R10: ffffa0a1e4d3fa80 R11: ffffa0a1e76b159a R12: 
> ffffa0a1eac7c000
> [79162.656715] R13: 0000000000000000 R14: ffffb5f6c01e3000 R15: 
> ffffb5f6c015f000
> [79162.657429] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000) 
> knlGS:0000000000000000
> [79162.658137] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [79162.658826] CR2: 0000000000000000 CR3: 00000001e242a005 CR4: 
> 0000000000360ee0
> [79162.659515] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [79162.660196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
> 0000000000000400
> 
> 
> I’ve put my code on GitHub, maybe it’s just something stupid…
> 
> https://github.com/chaudron/bpf2bpf-tracing
> 
> 
> Cheers,
> 
> Eelco
> 
> 
> PS: If I run the latest pahole (v1.15) on the .o files, I get the 
> following libbpf error: “libbpf: Cannot find bpf_func_info for main 
> program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”
> 

Re: Trying the bpf trace a bpf xdp program

[Andrii Nakryiko]

On Wed, Dec 4, 2019 at 5:20 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 2 Dec 2019, at 17:48, Yonghong Song wrote:
>
> > On 12/2/19 8:34 AM, Eelco Chaudron wrote:
> >> On 29 Nov 2019, at 17:52, Yonghong Song wrote:
>
> <SNIP>
> >
> > You need to build the kernel with
> >    CONFIG_DEBUG_INFO_BTF=y
> > Make sure on the build machine you have recent pahole version >= 1.13.
>
> With the latest LLVM and CONFIG_DEBUG_INFO_BTF=y the self-test for
> bpf2bpf is passing!
>
>
> However I still have problems with my code, which is getting to the next
> step, but no my program is killed when trying to load the eBPG fexit
> code. If I replace my generated eBPF programs for the once generated by
> the self-test (test_pkt_access.o/fexit_bpf2bpf.o) it works fine.
>
>
> I decided to build my objects just like the example programs (so have a
> hacked build.sh file) but I get the same results. I.e. being killed by
> the kernel:
>
> bpf(BPF_BTF_LOAD,
> {btf="\237\353\1\0\30\0\0\0\0\0\0\0\330\0\0\0\330\0\0\0\244\0\0\0\0\0\0\0\0\0\0\2"...,
> btf_log_buf=NULL, btf_size=404, btf_log_size=0, btf_log_level=0}, 120) =
> 6
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
> info=0x7ffdfbdac3b0}}, 120) = 0
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
> info=0xafb600}}, 120) = 0
> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=90}, 120) = 5
> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=16,
> info=0x7ffdfbdac4b0}}, 120) = 0
> - Opened object file: 0xafb440
> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2,
> insns=0xafbaa0, license="GPL", log_level=7, log_size=16777215,
> log_buf="\237\353\1", kern_version=KERNEL_VERSION(0, 0, 0),
> prog_flags=0, prog_name="test_main", prog_ifindex=0,
> expected_attach_type=0x19 /* BPF_??? */, prog_btf_fd=6,
> func_info_rec_size=8, func_info=0xafb9f0, func_info_cnt=1,
> line_info_rec_size=16, line_info=0xafba10, line_info_cnt=1, ...}, 120
> ) = ?
> +++ killed by SIGKILL +++
> Killed
>
>
> [79162.619208] BUG: kernel NULL pointer dereference, address:
> 0000000000000000
> [79162.619906] #PF: supervisor read access in kernel mode
> [79162.620582] #PF: error_code(0x0000) - not-present page
> [79162.621255] PGD 80000001e2409067 P4D 80000001e2409067 PUD 22eba9067
> PMD 0
> [79162.621933] Oops: 0000 [#12] SMP PTI
> [79162.622599] CPU: 5 PID: 3191 Comm: xdp_sample_fent Tainted: G      D
>           5.4.0+ #3
> [79162.623274] Hardware name: Red Hat KVM, BIOS
> 1.11.1-3.module+el8+2529+a9686a4d 04/01/2014
> [79162.623962] RIP: 0010:bpf_check+0x1648/0x250b
> [79162.624650] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17
> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40
> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
> [79162.626088] RSP: 0018:ffffb5f6c07c3c88 EFLAGS: 00010293
> [79162.626822] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> ffffb5f6c07c3c40
> [79162.627560] RDX: ffffa0a1e6e01818 RSI: 00000000fffffffa RDI:
> 0000000000000000
> [79162.628304] RBP: ffffb5f6c07c3d70 R08: 000000000000000e R09:
> ffffa0a1f5c9dc90
> [79162.629053] R10: ffffa0a1f5c9dc80 R11: ffffa0a1e6e0199a R12:
> ffffa0a1eac48000
> [79162.629806] R13: 0000000000000000 R14: ffffb5f6c043e000 R15:
> ffffb5f6c033f000
> [79162.630562] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000)
> knlGS:0000000000000000
> [79162.631324] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [79162.632072] CR2: 0000000000000000 CR3: 00000001e242a005 CR4:
> 0000000000360ee0
> [79162.632813] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [79162.633539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [79162.634255] Call Trace:
> [79162.634974]  ? _cond_resched+0x15/0x30
> [79162.635686]  ? kmem_cache_alloc_trace+0x162/0x220
> [79162.636398]  ? selinux_bpf_prog_alloc+0x1f/0x60
> [79162.637111]  bpf_prog_load+0x3de/0x690
> [79162.637809]  __do_sys_bpf+0x105/0x1740
> [79162.638488]  do_syscall_64+0x5b/0x180
> [79162.639147]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [79162.639792] RIP: 0033:0x7f560c3fe1ad
> [79162.640415] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa
> 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ab 5c 0c 00 f7 d8 64 89 01 48
> [79162.641703] RSP: 002b:00007ffdfbdac318 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000141
> [79162.642363] RAX: ffffffffffffffda RBX: 0000000000afb440 RCX:
> 00007f560c3fe1ad
> [79162.643026] RDX: 0000000000000078 RSI: 00007ffdfbdac390 RDI:
> 0000000000000005
> [79162.643676] RBP: 00007ffdfbdac330 R08: 0000000000afba70 R09:
> 00007ffdfbdac390
> [79162.644310] R10: 0000000000afcf10 R11: 0000000000000202 R12:
> 0000000000402690
> [79162.644935] R13: 00007ffdfbdac790 R14: 0000000000000000 R15:
> 0000000000000000
> [79162.645559] Modules linked in: ip6t_REJECT nf_reject_ipv6
> ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat
> ebtable_broute ip6table_nat ip6table_mangle ip6table_raw
> ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw
> iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set
> nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables
> iptable_filter intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass
> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cirrus drm_kms_helper
> virtio_net net_failover joydev drm failover i2c_piix4 virtio_balloon
> pcspkr ip_tables xfs libcrc32c crc32c_intel ata_generic floppy
> virtio_scsi serio_raw pata_acpi qemu_fw_cfg
> [79162.649591] CR2: 0000000000000000
> [79162.650272] ---[ end trace 5119c5364c1e9c83 ]---
> [79162.650957] RIP: 0010:bpf_check+0x1648/0x250b
> [79162.651646] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17
> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40
> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
> [79162.653081] RSP: 0018:ffffb5f6c072bc88 EFLAGS: 00010293
> [79162.653807] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> ffffb5f6c072bc40
> [79162.654536] RDX: ffffa0a1e76b1418 RSI: 00000000fffffffa RDI:
> 0000000000000000
> [79162.655270] RBP: ffffb5f6c072bd70 R08: 000000000000000e R09:
> ffffa0a1e4d3fa90
> [79162.655996] R10: ffffa0a1e4d3fa80 R11: ffffa0a1e76b159a R12:
> ffffa0a1eac7c000
> [79162.656715] R13: 0000000000000000 R14: ffffb5f6c01e3000 R15:
> ffffb5f6c015f000
> [79162.657429] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000)
> knlGS:0000000000000000
> [79162.658137] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [79162.658826] CR2: 0000000000000000 CR3: 00000001e242a005 CR4:
> 0000000000360ee0
> [79162.659515] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [79162.660196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
>
>
> I’ve put my code on GitHub, maybe it’s just something stupid…
>
> https://github.com/chaudron/bpf2bpf-tracing
>
>
> Cheers,
>
> Eelco
>
>
> PS: If I run the latest pahole (v1.15) on the .o files, I get the
> following libbpf error: “libbpf: Cannot find bpf_func_info for main
> program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”
>

pahole is not supposed to be run on BPF object file. It's needed only
to do DWARF to BTF conversion for kernel itself. So never mind this
one. The NULL dereference, though, seems like a bug, I agree with
Yonghong.

Re: Trying the bpf trace a bpf xdp program

[Yonghong Song]

On 12/4/19 6:58 AM, Yonghong Song wrote:
> 
> 
> On 12/4/19 5:19 AM, Eelco Chaudron wrote:
>>
>>
>> On 2 Dec 2019, at 17:48, Yonghong Song wrote:
>>
>>> On 12/2/19 8:34 AM, Eelco Chaudron wrote:
>>>> On 29 Nov 2019, at 17:52, Yonghong Song wrote:
>>
>> <SNIP>
>>>
>>> You need to build the kernel with
>>>     CONFIG_DEBUG_INFO_BTF=y
>>> Make sure on the build machine you have recent pahole version >= 1.13.
>>
>> With the latest LLVM and CONFIG_DEBUG_INFO_BTF=y the self-test for
>> bpf2bpf is passing!
> 
> Great!
> 
>>
>> However I still have problems with my code, which is getting to the next
>> step, but no my program is killed when trying to load the eBPG fexit
>> code. If I replace my generated eBPF programs for the once generated by
>> the self-test (test_pkt_access.o/fexit_bpf2bpf.o) it works fine.
>>
>>
>> I decided to build my objects just like the example programs (so have a
>> hacked build.sh file) but I get the same results. I.e. being killed by
>> the kernel:
>>
>> bpf(BPF_BTF_LOAD,
>> {btf="\237\353\1\0\30\0\0\0\0\0\0\0\330\0\0\0\330\0\0\0\244\0\0\0\0\0\0\0\0\0\0\2"...,
>> btf_log_buf=NULL, btf_size=404, btf_log_size=0, btf_log_level=0}, 120) = 6
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>> info=0x7ffdfbdac3b0}}, 120) = 0
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=208,
>> info=0xafb600}}, 120) = 0
>> bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=90}, 120) = 5
>> bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=16,
>> info=0x7ffdfbdac4b0}}, 120) = 0
>> - Opened object file: 0xafb440
>> bpf(BPF_PROG_LOAD, {prog_type=0x1a /* BPF_PROG_TYPE_??? */, insn_cnt=2,
>> insns=0xafbaa0, license="GPL", log_level=7, log_size=16777215,
>> log_buf="\237\353\1", kern_version=KERNEL_VERSION(0, 0, 0),
>> prog_flags=0, prog_name="test_main", prog_ifindex=0,
>> expected_attach_type=0x19 /* BPF_??? */, prog_btf_fd=6,
>> func_info_rec_size=8, func_info=0xafb9f0, func_info_cnt=1,
>> line_info_rec_size=16, line_info=0xafba10, line_info_cnt=1, ...}, 120
>> ) = ?
>> +++ killed by SIGKILL +++
>> Killed
>>
>>
>> [79162.619208] BUG: kernel NULL pointer dereference, address:
> 
> This should be a kernel bug. I will take a look at it today.
> 
>> 0000000000000000
>> [79162.619906] #PF: supervisor read access in kernel mode
>> [79162.620582] #PF: error_code(0x0000) - not-present page
>> [79162.621255] PGD 80000001e2409067 P4D 80000001e2409067 PUD 22eba9067
>> PMD 0
>> [79162.621933] Oops: 0000 [#12] SMP PTI
>> [79162.622599] CPU: 5 PID: 3191 Comm: xdp_sample_fent Tainted: G      D
>>            5.4.0+ #3
>> [79162.623274] Hardware name: Red Hat KVM, BIOS
>> 1.11.1-3.module+el8+2529+a9686a4d 04/01/2014
>> [79162.623962] RIP: 0010:bpf_check+0x1648/0x250b
>> [79162.624650] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17
>> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40
>> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
>> [79162.626088] RSP: 0018:ffffb5f6c07c3c88 EFLAGS: 00010293
>> [79162.626822] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
>> ffffb5f6c07c3c40
>> [79162.627560] RDX: ffffa0a1e6e01818 RSI: 00000000fffffffa RDI:
>> 0000000000000000
>> [79162.628304] RBP: ffffb5f6c07c3d70 R08: 000000000000000e R09:
>> ffffa0a1f5c9dc90
>> [79162.629053] R10: ffffa0a1f5c9dc80 R11: ffffa0a1e6e0199a R12:
>> ffffa0a1eac48000
>> [79162.629806] R13: 0000000000000000 R14: ffffb5f6c043e000 R15:
>> ffffb5f6c033f000
>> [79162.630562] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000)
>> knlGS:0000000000000000
>> [79162.631324] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [79162.632072] CR2: 0000000000000000 CR3: 00000001e242a005 CR4:
>> 0000000000360ee0
>> [79162.632813] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [79162.633539] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>> 0000000000000400
>> [79162.634255] Call Trace:
>> [79162.634974]  ? _cond_resched+0x15/0x30
>> [79162.635686]  ? kmem_cache_alloc_trace+0x162/0x220
>> [79162.636398]  ? selinux_bpf_prog_alloc+0x1f/0x60
>> [79162.637111]  bpf_prog_load+0x3de/0x690
>> [79162.637809]  __do_sys_bpf+0x105/0x1740
>> [79162.638488]  do_syscall_64+0x5b/0x180
>> [79162.639147]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>> [79162.639792] RIP: 0033:0x7f560c3fe1ad
>> [79162.640415] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa
>> 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f
>> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ab 5c 0c 00 f7 d8 64 89 01 48
>> [79162.641703] RSP: 002b:00007ffdfbdac318 EFLAGS: 00000202 ORIG_RAX:
>> 0000000000000141
>> [79162.642363] RAX: ffffffffffffffda RBX: 0000000000afb440 RCX:
>> 00007f560c3fe1ad
>> [79162.643026] RDX: 0000000000000078 RSI: 00007ffdfbdac390 RDI:
>> 0000000000000005
>> [79162.643676] RBP: 00007ffdfbdac330 R08: 0000000000afba70 R09:
>> 00007ffdfbdac390
>> [79162.644310] R10: 0000000000afcf10 R11: 0000000000000202 R12:
>> 0000000000402690
>> [79162.644935] R13: 00007ffdfbdac790 R14: 0000000000000000 R15:
>> 0000000000000000
>> [79162.645559] Modules linked in: ip6t_REJECT nf_reject_ipv6
>> ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat
>> ebtable_broute ip6table_nat ip6table_mangle ip6table_raw
>> ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw
>> iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set
>> nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables
>> iptable_filter intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass
>> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cirrus drm_kms_helper
>> virtio_net net_failover joydev drm failover i2c_piix4 virtio_balloon
>> pcspkr ip_tables xfs libcrc32c crc32c_intel ata_generic floppy
>> virtio_scsi serio_raw pata_acpi qemu_fw_cfg
>> [79162.649591] CR2: 0000000000000000
>> [79162.650272] ---[ end trace 5119c5364c1e9c83 ]---
>> [79162.650957] RIP: 0010:bpf_check+0x1648/0x250b
>> [79162.651646] Code: 41 89 c5 0f 88 d1 0a 00 00 41 f6 47 02 01 0f 84 17
>> 0b 00 00 41 83 7f 04 1a 0f 84 0c 0c 00 00 49 8b 47 20 48 63 db 48 8b 40
>> 68 <48> 8b 04 d8 48 8b 40 30 49 89 42 50 49 8b 46 20 4c 89 cf 4c 89 95
>> [79162.653081] RSP: 0018:ffffb5f6c072bc88 EFLAGS: 00010293
>> [79162.653807] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
>> ffffb5f6c072bc40
>> [79162.654536] RDX: ffffa0a1e76b1418 RSI: 00000000fffffffa RDI:
>> 0000000000000000
>> [79162.655270] RBP: ffffb5f6c072bd70 R08: 000000000000000e R09:
>> ffffa0a1e4d3fa90
>> [79162.655996] R10: ffffa0a1e4d3fa80 R11: ffffa0a1e76b159a R12:
>> ffffa0a1eac7c000
>> [79162.656715] R13: 0000000000000000 R14: ffffb5f6c01e3000 R15:
>> ffffb5f6c015f000
>> [79162.657429] FS:  00007f560c2e3740(0000) GS:ffffa0a1f7940000(0000)
>> knlGS:0000000000000000
>> [79162.658137] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [79162.658826] CR2: 0000000000000000 CR3: 00000001e242a005 CR4:
>> 0000000000360ee0
>> [79162.659515] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [79162.660196] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
>> 0000000000000400
>>
>>
>> I’ve put my code on GitHub, maybe it’s just something stupid…

Thanks for the test case. This indeed a kernel bug.
The following change fixed the issue:


-bash-4.4$ git diff
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a0482e1c4a77..034ef81f935b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9636,7 +9636,10 @@ static int check_attach_btf_id(struct 
bpf_verifier_env *env)
                                 ret = -EINVAL;
                                 goto out;
                         }
-                       addr = (long) 
tgt_prog->aux->func[subprog]->bpf_func;
+                       if (subprog == 0)
+                               addr = (long) tgt_prog->bpf_func;
+                       else
+                               addr = (long) 
tgt_prog->aux->func[subprog]->bpf_func;
                 } else {
                         addr = kallsyms_lookup_name(tname);
                         if (!addr) {
-bash-4.4$

The reason is for a bpf program without any additional subprogram 
(callees), tgt_prog->aux->func is not populated and is a NULL pointer,
so the access tgt_prog->aux->func[0]->bpf_func will segfault.

With the above change, your test works properly.

Will send a patch to upstream soon.

>>
>> https://github.com/chaudron/bpf2bpf-tracing
>>
>>
>> Cheers,
>>
>> Eelco
>>
>>
>> PS: If I run the latest pahole (v1.15) on the .o files, I get the
>> following libbpf error: “libbpf: Cannot find bpf_func_info for main
>> program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”
>>

Re: Trying the bpf trace a bpf xdp program

[John Fastabend]

Andrii Nakryiko wrote:
> On Wed, Dec 4, 2019 at 5:20 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> >

[...]

> >
> > PS: If I run the latest pahole (v1.15) on the .o files, I get the
> > following libbpf error: “libbpf: Cannot find bpf_func_info for main
> > program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”
> >
> 
> pahole is not supposed to be run on BPF object file. It's needed only
> to do DWARF to BTF conversion for kernel itself. So never mind this
> one. The NULL dereference, though, seems like a bug, I agree with
> Yonghong.

Really? I've been using pahole on BPF object files regularly mostly
to test structures match up with kernel pahole output but its always
worked on my side. Even pahole -j seems to work fine here.

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 4 Dec 2019, at 19:01, Yonghong Song wrote:

<SNIP>

>>> I’ve put my code on GitHub, maybe it’s just something stupid…
>
> Thanks for the test case. This indeed a kernel bug.
> The following change fixed the issue:
>
>
> -bash-4.4$ git diff
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index a0482e1c4a77..034ef81f935b 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -9636,7 +9636,10 @@ static int check_attach_btf_id(struct
> bpf_verifier_env *env)
>                                  ret = -EINVAL;
>                                  goto out;
>                          }
> -                       addr = (long)
> tgt_prog->aux->func[subprog]->bpf_func;
> +                       if (subprog == 0)
> +                               addr = (long) tgt_prog->bpf_func;
> +                       else
> +                               addr = (long)
> tgt_prog->aux->func[subprog]->bpf_func;
>                  } else {
>                          addr = kallsyms_lookup_name(tname);
>                          if (!addr) {
> -bash-4.4$
>
> The reason is for a bpf program without any additional subprogram
> (callees), tgt_prog->aux->func is not populated and is a NULL pointer,
> so the access tgt_prog->aux->func[0]->bpf_func will segfault.
>
> With the above change, your test works properly.

Thanks for the quick response, and as you mention the test passes with 
the patch above.

I will continue my experiments later this week, and let you know if I 
run into any other problems.

Cheers,

Eelco

Re: Trying the bpf trace a bpf xdp program

[Andrii Nakryiko]

On Wed, Dec 4, 2019 at 10:03 AM John Fastabend <john.fastabend@gmail.com> wrote:
>
> Andrii Nakryiko wrote:
> > On Wed, Dec 4, 2019 at 5:20 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> > >
>
> [...]
>
> > >
> > > PS: If I run the latest pahole (v1.15) on the .o files, I get the
> > > following libbpf error: “libbpf: Cannot find bpf_func_info for main
> > > program sec fexit/xdp_prog_simple. Ignore all bpf_func_info.”
> > >
> >
> > pahole is not supposed to be run on BPF object file. It's needed only
> > to do DWARF to BTF conversion for kernel itself. So never mind this
> > one. The NULL dereference, though, seems like a bug, I agree with
> > Yonghong.
>
> Really? I've been using pahole on BPF object files regularly mostly
> to test structures match up with kernel pahole output but its always
> worked on my side. Even pahole -j seems to work fine here.

Well, it still works for simpler stuff, but it certainly won't emit
global variables type info, CO-RE relocations, etc. BTF for BPF object
files should get produced by Clang,  which is perfectly capable of
doing this (provided you have recent enough version of it, of course).

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 4 Dec 2019, at 19:52, Eelco Chaudron wrote:

> On 4 Dec 2019, at 19:01, Yonghong Song wrote:
>
> <SNIP>
>
>>>> I’ve put my code on GitHub, maybe it’s just something stupid…
>>
>> Thanks for the test case. This indeed a kernel bug.
>> The following change fixed the issue:
>>
>>
>> -bash-4.4$ git diff
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index a0482e1c4a77..034ef81f935b 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> @@ -9636,7 +9636,10 @@ static int check_attach_btf_id(struct
>> bpf_verifier_env *env)
>>                                  ret = -EINVAL;
>>                                  goto out;
>>                          }
>> -                       addr = (long)
>> tgt_prog->aux->func[subprog]->bpf_func;
>> +                       if (subprog == 0)
>> +                               addr = (long) tgt_prog->bpf_func;
>> +                       else
>> +                               addr = (long)
>> tgt_prog->aux->func[subprog]->bpf_func;
>>                  } else {
>>                          addr = kallsyms_lookup_name(tname);
>>                          if (!addr) {
>> -bash-4.4$
>>
>> The reason is for a bpf program without any additional subprogram
>> (callees), tgt_prog->aux->func is not populated and is a NULL 
>> pointer,
>> so the access tgt_prog->aux->func[0]->bpf_func will segfault.
>>
>> With the above change, your test works properly.
>
> Thanks for the quick response, and as you mention the test passes with 
> the patch above.
>
> I will continue my experiments later this week, and let you know if I 
> run into any other problems.
>

With the following program I get some access errors:

#define bpf_debug(fmt, ...)                         \
{                                                   \
   char __fmt[] = fmt;                               \
   bpf_trace_printk(__fmt, sizeof(__fmt),            \
                    ##__VA_ARGS__);                  \
}

BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
             struct xdp_md *, xdp, int, ret)
{
   __u32 rx_queue;

   __builtin_preserve_access_index(({
         rx_queue = xdp->rx_queue_index;
       }));

   bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);

   return 0;
}

I assume the XDP context has not been vetted?

libbpf: -- BEGIN DUMP LOG ---
libbpf:
func#0 @0
BPF program ctx type is not a struct
Type info disagrees with actual arguments due to compiler optimizations
0: R1=ctx(id=0,off=0,imm=0) R10=fp0
; BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
0: (b7) r2 = 16
1: R1=ctx(id=0,off=0,imm=0) R2_w=inv16 R10=fp0
; BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
1: (79) r3 = *(u64 *)(r1 +0)
2: R1=ctx(id=0,off=0,imm=0) R2_w=inv16 
R3_w=ptr_xdp_buff(id=0,off=0,imm=0) R10=fp0
2: (0f) r3 += r2
last_idx 2 first_idx 0
regs=4 stack=0 before 1: (79) r3 = *(u64 *)(r1 +0)
regs=4 stack=0 before 0: (b7) r2 = 16
3: R1=ctx(id=0,off=0,imm=0) R2_w=invP16 
R3_w=ptr_xdp_buff(id=0,off=16,imm=0) R10=fp0
; rx_queue = xdp->rx_queue_index;
3: (61) r3 = *(u32 *)(r3 +0)
cannot access ptr member data_meta with moff 16 in struct xdp_buff with 
off 16 size 4
verification time 102 usec
stack depth 0
processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 
peak_states 0 mark_read 0
libbpf: -- END LOG --


Trying to use the helpers, passes verification, however, it’s dumping 
invalid content:

BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
             struct xdp_md *, xdp, int, ret)
{
   __u32 rx_queue;

   bpf_probe_read_kernel(&rx_queue, sizeof(rx_queue),
                         __builtin_preserve_access_index(&xdp->rx_queue_index));

   bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);
   return 0;
}

Debug output:

    ping6-2752  [004] ..s1 60763.917790: 0: SIMPLE: [ifindex = 4, queue 
=  0]
    ping6-2752  [004] ..s1 60763.917800: 0: fexit: queue = 2969379072, 
ret = 2
    ping6-2752  [004] ..s1 60764.941817: 0: SIMPLE: [ifindex = 4, queue 
=  0]
    ping6-2752  [004] ..s1 60764.941828: 0: fexit: queue = 2969379072, 
ret = 2
    ping6-2752  [004] ..s1 60765.965835: 0: SIMPLE: [ifindex = 4, queue 
=  0]


Tried the same with fentry for this function, but the same results as 
fexit.

Any hints?

Thanks,


Eelco

Re: Trying the bpf trace a bpf xdp program

[Y Song]

On Thu, Dec 5, 2019 at 4:41 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 4 Dec 2019, at 19:52, Eelco Chaudron wrote:
>
> > On 4 Dec 2019, at 19:01, Yonghong Song wrote:
> >
> > <SNIP>
> >
> >>>> I’ve put my code on GitHub, maybe it’s just something stupid…
> >>
> >> Thanks for the test case. This indeed a kernel bug.
> >> The following change fixed the issue:
> >>
> >>
> >> -bash-4.4$ git diff
> >> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> >> index a0482e1c4a77..034ef81f935b 100644
> >> --- a/kernel/bpf/verifier.c
> >> +++ b/kernel/bpf/verifier.c
> >> @@ -9636,7 +9636,10 @@ static int check_attach_btf_id(struct
> >> bpf_verifier_env *env)
> >>                                  ret = -EINVAL;
> >>                                  goto out;
> >>                          }
> >> -                       addr = (long)
> >> tgt_prog->aux->func[subprog]->bpf_func;
> >> +                       if (subprog == 0)
> >> +                               addr = (long) tgt_prog->bpf_func;
> >> +                       else
> >> +                               addr = (long)
> >> tgt_prog->aux->func[subprog]->bpf_func;
> >>                  } else {
> >>                          addr = kallsyms_lookup_name(tname);
> >>                          if (!addr) {
> >> -bash-4.4$
> >>
> >> The reason is for a bpf program without any additional subprogram
> >> (callees), tgt_prog->aux->func is not populated and is a NULL
> >> pointer,
> >> so the access tgt_prog->aux->func[0]->bpf_func will segfault.
> >>
> >> With the above change, your test works properly.
> >
> > Thanks for the quick response, and as you mention the test passes with
> > the patch above.
> >
> > I will continue my experiments later this week, and let you know if I
> > run into any other problems.
> >
>
> With the following program I get some access errors:
>
> #define bpf_debug(fmt, ...)                         \
> {                                                   \
>    char __fmt[] = fmt;                               \
>    bpf_trace_printk(__fmt, sizeof(__fmt),            \
>                     ##__VA_ARGS__);                  \
> }
>
> BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
>              struct xdp_md *, xdp, int, ret)
> {
>    __u32 rx_queue;
>
>    __builtin_preserve_access_index(({
>          rx_queue = xdp->rx_queue_index;
>        }));
>
>    bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);
>
>    return 0;
> }
>
> I assume the XDP context has not been vetted?
>
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> func#0 @0
> BPF program ctx type is not a struct
> Type info disagrees with actual arguments due to compiler optimizations
> 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> ; BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
> 0: (b7) r2 = 16
> 1: R1=ctx(id=0,off=0,imm=0) R2_w=inv16 R10=fp0
> ; BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
> 1: (79) r3 = *(u64 *)(r1 +0)
> 2: R1=ctx(id=0,off=0,imm=0) R2_w=inv16
> R3_w=ptr_xdp_buff(id=0,off=0,imm=0) R10=fp0
> 2: (0f) r3 += r2
> last_idx 2 first_idx 0
> regs=4 stack=0 before 1: (79) r3 = *(u64 *)(r1 +0)
> regs=4 stack=0 before 0: (b7) r2 = 16
> 3: R1=ctx(id=0,off=0,imm=0) R2_w=invP16
> R3_w=ptr_xdp_buff(id=0,off=16,imm=0) R10=fp0
> ; rx_queue = xdp->rx_queue_index;
> 3: (61) r3 = *(u32 *)(r3 +0)
> cannot access ptr member data_meta with moff 16 in struct xdp_buff with
> off 16 size 4
> verification time 102 usec
> stack depth 0
> processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
> libbpf: -- END LOG --
>

It is a little tricky. The below change can make verifier happy. I did
not test it so not sure whether produces correct result or not.

==========
struct xdp_rxq_info {
        __u32 queue_index;
} __attribute__((preserve_access_index));

struct xdp_buff {
        struct xdp_rxq_info *rxq;
} __attribute__((preserve_access_index));

BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
            struct xdp_buff *, ctx, int, ret)
{
   __u32 rx_queue;

   rx_queue = ctx->rxq->queue_index;
   bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);

   return 0;
}
==========

In the above, I am using newly added clang attribute "__preserve_access_index"
(in llvm-project trunk since Nov. 13) to make code
a little bit cleaner. The old way as in selftests fexit_bpf2bpf.c
should work too.

Basically, the argument for fexit function should be types actually
passing to the jited function.
For user visible 'xdp_md`. the jited function will receive `xdp_buff`.
The access for each field
sometimes is not one-to-one mapping. You need to check kernel code to
find the correct
way. We probably should make this part better to improve user experience.

>
> Trying to use the helpers, passes verification, however, it’s dumping
> invalid content:
>
> BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
>              struct xdp_md *, xdp, int, ret)
> {
>    __u32 rx_queue;
>
>    bpf_probe_read_kernel(&rx_queue, sizeof(rx_queue),
>                          __builtin_preserve_access_index(&xdp->rx_queue_index));
>
>    bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);
>    return 0;
> }
>
> Debug output:
>
>     ping6-2752  [004] ..s1 60763.917790: 0: SIMPLE: [ifindex = 4, queue
> =  0]
>     ping6-2752  [004] ..s1 60763.917800: 0: fexit: queue = 2969379072,
> ret = 2
>     ping6-2752  [004] ..s1 60764.941817: 0: SIMPLE: [ifindex = 4, queue
> =  0]
>     ping6-2752  [004] ..s1 60764.941828: 0: fexit: queue = 2969379072,
> ret = 2
>     ping6-2752  [004] ..s1 60765.965835: 0: SIMPLE: [ifindex = 4, queue
> =  0]
>
>
> Tried the same with fentry for this function, but the same results as
> fexit.
>
> Any hints?
>
> Thanks,
>
>
> Eelco
>

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 5 Dec 2019, at 18:35, Y Song wrote:

> On Thu, Dec 5, 2019 at 4:41 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:

>
> It is a little tricky. The below change can make verifier happy. I did
> not test it so not sure whether produces correct result or not.
>
> ==========
> struct xdp_rxq_info {
>         __u32 queue_index;
> } __attribute__((preserve_access_index));
>
> struct xdp_buff {
>         struct xdp_rxq_info *rxq;
> } __attribute__((preserve_access_index));
>
> BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
>             struct xdp_buff *, ctx, int, ret)
> {
>    __u32 rx_queue;
>
>    rx_queue = ctx->rxq->queue_index;
>    bpf_debug("fexit: queue = %u, ret = %d\n", rx_queue, ret);
>
>    return 0;
> }
> ==========
>
> In the above, I am using newly added clang attribute 
> "__preserve_access_index"
> (in llvm-project trunk since Nov. 13) to make code
> a little bit cleaner. The old way as in selftests fexit_bpf2bpf.c
> should work too.
>
> Basically, the argument for fexit function should be types actually
> passing to the jited function.
> For user visible 'xdp_md`. the jited function will receive `xdp_buff`.
> The access for each field
> sometimes is not one-to-one mapping. You need to check kernel code to
> find the correct
> way. We probably should make this part better to improve user 
> experience.
>

Thanks the hint that it should be the jitted arguments solved it… And 
you quick example worked, just in case some one else is playing with it, 
here is my working example:

// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include "bpf_helpers.h"
#include "bpf_trace_helpers.h"

#define bpf_debug(fmt, ...)                \
{                                          \
     char __fmt[] = fmt;                    \
     bpf_trace_printk(__fmt, sizeof(__fmt), \
                      ##__VA_ARGS__);       \
}

struct net_device {
     /* Structure does not need to contain all entries,
      * as "preserve_access_index" will use BTF to fix this... */
     int                    ifindex;
} __attribute__((preserve_access_index));

struct xdp_rxq_info {
     /* Structure does not need to contain all entries,
      * as "preserve_access_index" will use BTF to fix this... */
     struct net_device *dev;
     __u32 queue_index;
} __attribute__((preserve_access_index));

struct xdp_buff {
     void *data;
     void *data_end;
     void *data_meta;
     void *data_hard_start;
     unsigned long handle;
     struct xdp_rxq_info *rxq;
} __attribute__((preserve_access_index));


BPF_TRACE_1("fentry/xdp_prog_simple", trace_on_entry,
             struct xdp_buff *, xdp)
{
     bpf_debug("fentry: [ifindex = %u, queue =  %u]\n",
               xdp->rxq->dev->ifindex, xdp->rxq->queue_index);
     return 0;
}


BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
             struct xdp_buff*, xdp, int, ret)
{
     bpf_debug("fexit: [ifindex = %u, queue =  %u, ret = %d]\n",
               xdp->rxq->dev->ifindex, xdp->rxq->queue_index, ret);

     return 0;
}

char _license[] SEC("license") = "GPL";

Re: Trying the bpf trace a bpf xdp program

[Alexei Starovoitov]

On Fri, Dec 6, 2019 at 5:05 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
> Thanks the hint that it should be the jitted arguments solved it… And
> you quick example worked, just in case some one else is playing with it,
> here is my working example:
>
> // SPDX-License-Identifier: GPL-2.0
> #include <linux/bpf.h>
> #include "bpf_helpers.h"
> #include "bpf_trace_helpers.h"
>
> #define bpf_debug(fmt, ...)                \
> {                                          \
>      char __fmt[] = fmt;                    \
>      bpf_trace_printk(__fmt, sizeof(__fmt), \
>                       ##__VA_ARGS__);       \
> }
>
> struct net_device {
>      /* Structure does not need to contain all entries,
>       * as "preserve_access_index" will use BTF to fix this... */
>      int                    ifindex;
> } __attribute__((preserve_access_index));
>
> struct xdp_rxq_info {
>      /* Structure does not need to contain all entries,
>       * as "preserve_access_index" will use BTF to fix this... */
>      struct net_device *dev;
>      __u32 queue_index;
> } __attribute__((preserve_access_index));
>
> struct xdp_buff {
>      void *data;
>      void *data_end;
>      void *data_meta;
>      void *data_hard_start;
>      unsigned long handle;
>      struct xdp_rxq_info *rxq;
> } __attribute__((preserve_access_index));
>
>
> BPF_TRACE_1("fentry/xdp_prog_simple", trace_on_entry,
>              struct xdp_buff *, xdp)
> {
>      bpf_debug("fentry: [ifindex = %u, queue =  %u]\n",
>                xdp->rxq->dev->ifindex, xdp->rxq->queue_index);
>      return 0;
> }
>
>
> BPF_TRACE_2("fexit/xdp_prog_simple", trace_on_exit,
>              struct xdp_buff*, xdp, int, ret)
> {
>      bpf_debug("fexit: [ifindex = %u, queue =  %u, ret = %d]\n",
>                xdp->rxq->dev->ifindex, xdp->rxq->queue_index, ret);

This is great. Could you submit it as selftests/bpf ?
It will help others trying to do the same.
May be instead of bpf_debug() use global variables so the test will be
self checking ?

Long term we should teach verifier to understand 'struct xdp_md*'
in addition to 'struct xdp_buff *'. That will help ease of use.

Re: Trying the bpf trace a bpf xdp program

[Eelco Chaudron]

On 7 Dec 2019, at 17:51, Alexei Starovoitov wrote:

> On Fri, Dec 6, 2019 at 5:05 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>>
>> Thanks the hint that it should be the jitted arguments solved it… And
>> you quick example worked, just in case some one else is playing with it,
>> here is my working example:
>>
<SNIP>

> This is great. Could you submit it as selftests/bpf ?

Good idea, sent a patch out...

> It will help others trying to do the same.
> May be instead of bpf_debug() use global variables so the test will be
> self checking ?
>
> Long term we should teach verifier to understand 'struct xdp_md*'
> in addition to 'struct xdp_buff *'. That will help ease of use.