* Inconsistent BTF entries for `struct pt_regs *regs` parameter
@ 2022-10-06 7:34 Akihiro HARAI
2022-10-06 9:54 ` Jiri Olsa
0 siblings, 1 reply; 4+ messages in thread
From: Akihiro HARAI @ 2022-10-06 7:34 UTC (permalink / raw)
To: bpf
Depending on distribution/kernel/syscall combination, BTF entry for
`struct pt_regs *regs` parameter differs.
For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
entry for `__x64_sys_recvmsg` function:
```
$ uname -a
Linux ip-10-1-1-66.ap-northeast-1.compute.internal
5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
x86_64 x86_64 GNU/Linux
$ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
...
[15439] FWD 'pt_regs' fwd_kind=struct
[15440] CONST '(anon)' type_id=15439
[15441] PTR '(anon)' type_id=15440
[15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
'__unused' type_id=15441
...
[15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
...
```
while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
same function:
```
$ uname -a
Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
[1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
...
[226] STRUCT 'pt_regs' size=168 vlen=21
'r15' type_id=1 bits_offset=0
'r14' type_id=1 bits_offset=64
'r13' type_id=1 bits_offset=128
'r12' type_id=1 bits_offset=192
'bp' type_id=1 bits_offset=256
'bx' type_id=1 bits_offset=320
'r11' type_id=1 bits_offset=384
'r10' type_id=1 bits_offset=448
'r9' type_id=1 bits_offset=512
'r8' type_id=1 bits_offset=576
'ax' type_id=1 bits_offset=640
'cx' type_id=1 bits_offset=704
'dx' type_id=1 bits_offset=768
'si' type_id=1 bits_offset=832
'di' type_id=1 bits_offset=896
'orig_ax' type_id=1 bits_offset=960
'ip' type_id=1 bits_offset=1024
'cs' type_id=1 bits_offset=1088
'flags' type_id=1 bits_offset=1152
'sp' type_id=1 bits_offset=1216
'ss' type_id=1 bits_offset=1280
...
[5183] CONST '(anon)' type_id=226
...
[5189] PTR '(anon)' type_id=5183
...
[5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
'__unused' type_id=5189
...
[17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
...
```
Yet another distribution/kernel/syscall combination has multiple `FUNC
'__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
```
$ uname -a
Linux ip-10-5-0-115.ap-northeast-1.compute.internal
5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
x86_64 x86_64 x86_64 GNU/Linux
```
$ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
__x64_sys_mprotect
...
[175] STRUCT 'pt_regs' size=168 vlen=21
'r15' type_id=2 bits_offset=0
'r14' type_id=2 bits_offset=64
'r13' type_id=2 bits_offset=128
'r12' type_id=2 bits_offset=192
'bp' type_id=2 bits_offset=256
...
[4215] CONST '(anon)' type_id=175
...
[4220] PTR '(anon)' type_id=4215
...
[6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
'regs' type_id=4220
...
[11461] FWD 'pt_regs' fwd_kind=struct
[11462] CONST '(anon)' type_id=11461
[11463] PTR '(anon)' type_id=11462
[11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
'__unused' type_id=11463
...
[11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
...
[23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
...
```
Trying to read `regs` parameter with FWD entry results in "invalid
bpf_context access" error:
```
SEC("fentry/__x64_sys_recvfrom")
int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
struct event t;
bpf_get_current_comm(t.comm, TASK_COMM_LEN);
u64 id = bpf_get_current_pid_tgid();
t.pid = id >> 32;
// This causes an error on some environments.
t.fd = PT_REGS_PARM1_CORE(regs);
bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
return 0;
```
```
$ sudo ./output
2022/07/01 03:33:01 loading objects: field FentrySyscall: program
fentry_syscall: load program: permission denied:
arg#0 type is not a struct
Unrecognized arg#0 type PTR
; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
0: (79) r6 = *(u64 *)(r1 +0)
func '__x64_sys_recvfrom' arg0 type FWD is not a struct
invalid bpf_context access off=0 size=8
processed 1 insns (limit 1000000) max_states_per_insn 0
total_states 0 peak_states 0 mark_read 0
```
Is this a bug related to toolchain?
I've asked this question to AWS support (owner of Amazon Linux 2) but
they couldn't answer. I've also asked on StackOverflow but no
responses as of now. A cilium/ebpf developer told me to ask here.
* https://stackoverflow.com/questions/72824924/invalid-bpf-context-access-when-trying-to-read-regs-parameter
* https://github.com/cilium/ebpf/issues/723#issuecomment-1190050109
* https://github.com/harai/invalidbpfcontext
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Inconsistent BTF entries for `struct pt_regs *regs` parameter
2022-10-06 7:34 Inconsistent BTF entries for `struct pt_regs *regs` parameter Akihiro HARAI
@ 2022-10-06 9:54 ` Jiri Olsa
2022-10-15 11:33 ` Jiri Olsa
0 siblings, 1 reply; 4+ messages in thread
From: Jiri Olsa @ 2022-10-06 9:54 UTC (permalink / raw)
To: Akihiro HARAI; +Cc: bpf
On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> Depending on distribution/kernel/syscall combination, BTF entry for
> `struct pt_regs *regs` parameter differs.
>
> For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> entry for `__x64_sys_recvmsg` function:
>
> ```
> $ uname -a
> Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> x86_64 x86_64 GNU/Linux
>
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> ...
> [15439] FWD 'pt_regs' fwd_kind=struct
> [15440] CONST '(anon)' type_id=15439
> [15441] PTR '(anon)' type_id=15440
> [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
> '__unused' type_id=15441
> ...
> [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> ...
> ```
>
> while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> same function:
>
> ```
> $ uname -a
> Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> ...
> [226] STRUCT 'pt_regs' size=168 vlen=21
> 'r15' type_id=1 bits_offset=0
> 'r14' type_id=1 bits_offset=64
> 'r13' type_id=1 bits_offset=128
> 'r12' type_id=1 bits_offset=192
> 'bp' type_id=1 bits_offset=256
> 'bx' type_id=1 bits_offset=320
> 'r11' type_id=1 bits_offset=384
> 'r10' type_id=1 bits_offset=448
> 'r9' type_id=1 bits_offset=512
> 'r8' type_id=1 bits_offset=576
> 'ax' type_id=1 bits_offset=640
> 'cx' type_id=1 bits_offset=704
> 'dx' type_id=1 bits_offset=768
> 'si' type_id=1 bits_offset=832
> 'di' type_id=1 bits_offset=896
> 'orig_ax' type_id=1 bits_offset=960
> 'ip' type_id=1 bits_offset=1024
> 'cs' type_id=1 bits_offset=1088
> 'flags' type_id=1 bits_offset=1152
> 'sp' type_id=1 bits_offset=1216
> 'ss' type_id=1 bits_offset=1280
> ...
> [5183] CONST '(anon)' type_id=226
> ...
> [5189] PTR '(anon)' type_id=5183
> ...
> [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
> '__unused' type_id=5189
> ...
> [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> ...
> ```
>
> Yet another distribution/kernel/syscall combination has multiple `FUNC
> '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
>
> ```
> $ uname -a
> Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> x86_64 x86_64 x86_64 GNU/Linux
>
> ```
> $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> __x64_sys_mprotect
> ...
> [175] STRUCT 'pt_regs' size=168 vlen=21
> 'r15' type_id=2 bits_offset=0
> 'r14' type_id=2 bits_offset=64
> 'r13' type_id=2 bits_offset=128
> 'r12' type_id=2 bits_offset=192
> 'bp' type_id=2 bits_offset=256
> ...
> [4215] CONST '(anon)' type_id=175
> ...
> [4220] PTR '(anon)' type_id=4215
> ...
> [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> 'regs' type_id=4220
> ...
> [11461] FWD 'pt_regs' fwd_kind=struct
> [11462] CONST '(anon)' type_id=11461
> [11463] PTR '(anon)' type_id=11462
> [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> '__unused' type_id=11463
> ...
> [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> ...
> [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> ...
> ```
>
> Trying to read `regs` parameter with FWD entry results in "invalid
> bpf_context access" error:
>
> ```
> SEC("fentry/__x64_sys_recvfrom")
> int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> struct event t;
>
> bpf_get_current_comm(t.comm, TASK_COMM_LEN);
>
> u64 id = bpf_get_current_pid_tgid();
> t.pid = id >> 32;
>
> // This causes an error on some environments.
> t.fd = PT_REGS_PARM1_CORE(regs);
>
> bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
>
> return 0;
> ```
>
> ```
> $ sudo ./output
> 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> fentry_syscall: load program: permission denied:
> arg#0 type is not a struct
> Unrecognized arg#0 type PTR
> ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> 0: (79) r6 = *(u64 *)(r1 +0)
> func '__x64_sys_recvfrom' arg0 type FWD is not a struct
> invalid bpf_context access off=0 size=8
> processed 1 insns (limit 1000000) max_states_per_insn 0
> total_states 0 peak_states 0 mark_read 0
> ```
>
> Is this a bug related to toolchain?
nice, I think it's specific to each object that defines syscall
if such object has 'struct pt_regs' header with definition included
it will have full struct pt_regs, if not it will be just fwd ref
not sure this would break anything else, but change below
fixes it for me
jirka
---
diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
index 59358d1bf880..fd2669b1cb2d 100644
--- a/arch/x86/include/asm/syscall_wrapper.h
+++ b/arch/x86/include/asm/syscall_wrapper.h
@@ -6,7 +6,7 @@
#ifndef _ASM_X86_SYSCALL_WRAPPER_H
#define _ASM_X86_SYSCALL_WRAPPER_H
-struct pt_regs;
+#include <asm/ptrace.h>
extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: Inconsistent BTF entries for `struct pt_regs *regs` parameter
2022-10-06 9:54 ` Jiri Olsa
@ 2022-10-15 11:33 ` Jiri Olsa
2022-10-16 0:49 ` Akihiro HARAI
0 siblings, 1 reply; 4+ messages in thread
From: Jiri Olsa @ 2022-10-15 11:33 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Akihiro HARAI, bpf
On Thu, Oct 06, 2022 at 11:54:13AM +0200, Jiri Olsa wrote:
> On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> > Depending on distribution/kernel/syscall combination, BTF entry for
> > `struct pt_regs *regs` parameter differs.
> >
> > For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> > entry for `__x64_sys_recvmsg` function:
> >
> > ```
> > $ uname -a
> > Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> > 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > ...
> > [15439] FWD 'pt_regs' fwd_kind=struct
> > [15440] CONST '(anon)' type_id=15439
> > [15441] PTR '(anon)' type_id=15440
> > [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
> > '__unused' type_id=15441
> > ...
> > [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> > ...
> > ```
> >
> > while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> > same function:
> >
> > ```
> > $ uname -a
> > Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> > 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > ...
> > [226] STRUCT 'pt_regs' size=168 vlen=21
> > 'r15' type_id=1 bits_offset=0
> > 'r14' type_id=1 bits_offset=64
> > 'r13' type_id=1 bits_offset=128
> > 'r12' type_id=1 bits_offset=192
> > 'bp' type_id=1 bits_offset=256
> > 'bx' type_id=1 bits_offset=320
> > 'r11' type_id=1 bits_offset=384
> > 'r10' type_id=1 bits_offset=448
> > 'r9' type_id=1 bits_offset=512
> > 'r8' type_id=1 bits_offset=576
> > 'ax' type_id=1 bits_offset=640
> > 'cx' type_id=1 bits_offset=704
> > 'dx' type_id=1 bits_offset=768
> > 'si' type_id=1 bits_offset=832
> > 'di' type_id=1 bits_offset=896
> > 'orig_ax' type_id=1 bits_offset=960
> > 'ip' type_id=1 bits_offset=1024
> > 'cs' type_id=1 bits_offset=1088
> > 'flags' type_id=1 bits_offset=1152
> > 'sp' type_id=1 bits_offset=1216
> > 'ss' type_id=1 bits_offset=1280
> > ...
> > [5183] CONST '(anon)' type_id=226
> > ...
> > [5189] PTR '(anon)' type_id=5183
> > ...
> > [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
> > '__unused' type_id=5189
> > ...
> > [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> > ...
> > ```
> >
> > Yet another distribution/kernel/syscall combination has multiple `FUNC
> > '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
> >
> > ```
> > $ uname -a
> > Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> > 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > ```
> > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> > __x64_sys_mprotect
> > ...
> > [175] STRUCT 'pt_regs' size=168 vlen=21
> > 'r15' type_id=2 bits_offset=0
> > 'r14' type_id=2 bits_offset=64
> > 'r13' type_id=2 bits_offset=128
> > 'r12' type_id=2 bits_offset=192
> > 'bp' type_id=2 bits_offset=256
> > ...
> > [4215] CONST '(anon)' type_id=175
> > ...
> > [4220] PTR '(anon)' type_id=4215
> > ...
> > [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > 'regs' type_id=4220
> > ...
> > [11461] FWD 'pt_regs' fwd_kind=struct
> > [11462] CONST '(anon)' type_id=11461
> > [11463] PTR '(anon)' type_id=11462
> > [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > '__unused' type_id=11463
> > ...
> > [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> > ...
> > [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> > ...
> > ```
> >
> > Trying to read `regs` parameter with FWD entry results in "invalid
> > bpf_context access" error:
> >
> > ```
> > SEC("fentry/__x64_sys_recvfrom")
> > int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > struct event t;
> >
> > bpf_get_current_comm(t.comm, TASK_COMM_LEN);
> >
> > u64 id = bpf_get_current_pid_tgid();
> > t.pid = id >> 32;
> >
> > // This causes an error on some environments.
> > t.fd = PT_REGS_PARM1_CORE(regs);
> >
> > bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
> >
> > return 0;
> > ```
> >
> > ```
> > $ sudo ./output
> > 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> > fentry_syscall: load program: permission denied:
> > arg#0 type is not a struct
> > Unrecognized arg#0 type PTR
> > ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > 0: (79) r6 = *(u64 *)(r1 +0)
> > func '__x64_sys_recvfrom' arg0 type FWD is not a struct
> > invalid bpf_context access off=0 size=8
> > processed 1 insns (limit 1000000) max_states_per_insn 0
> > total_states 0 peak_states 0 mark_read 0
> > ```
> >
> > Is this a bug related to toolchain?
>
> nice, I think it's specific to each object that defines syscall
>
> if such object has 'struct pt_regs' header with definition included
> it will have full struct pt_regs, if not it will be just fwd ref
>
> not sure this would break anything else, but change below
> fixes it for me
hi,
did this help? I'll send it as formal patch anyway,
since it fixes the issue for me
jirka
>
> jirka
>
>
> ---
> diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
> index 59358d1bf880..fd2669b1cb2d 100644
> --- a/arch/x86/include/asm/syscall_wrapper.h
> +++ b/arch/x86/include/asm/syscall_wrapper.h
> @@ -6,7 +6,7 @@
> #ifndef _ASM_X86_SYSCALL_WRAPPER_H
> #define _ASM_X86_SYSCALL_WRAPPER_H
>
> -struct pt_regs;
> +#include <asm/ptrace.h>
>
> extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
> extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Inconsistent BTF entries for `struct pt_regs *regs` parameter
2022-10-15 11:33 ` Jiri Olsa
@ 2022-10-16 0:49 ` Akihiro HARAI
0 siblings, 0 replies; 4+ messages in thread
From: Akihiro HARAI @ 2022-10-16 0:49 UTC (permalink / raw)
To: Jiri Olsa; +Cc: bpf
Thanks for your quick reply.
I've done patching the kernel right now and confirmed the BTF entries
have been fixed. (This is the first time I patched Linux. Wow!)
The entry "FWD 'pt_regs'" has been removed altogether.
I want this kind of fix to be applied to the Linux kernel itself. How
can I support it?
Kernel without patch:
```
$ uname -r
5.15.43-20.123.0withoutpatch.amzn2.x86_64
$ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
...
[15439] FWD 'pt_regs' fwd_kind=struct
[15440] CONST '(anon)' type_id=15439
[15441] PTR '(anon)' type_id=15440
[15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
'__unused' type_id=15441
...
[15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
...
```
Patched kernel:
```
$ uname -r
5.15.43-20.123.1patched.amzn2.x86_64
$ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
...
[172] STRUCT 'pt_regs' size=168 vlen=21
'r15' type_id=1 bits_offset=0
'r14' type_id=1 bits_offset=64
'r13' type_id=1 bits_offset=128
'r12' type_id=1 bits_offset=192
'bp' type_id=1 bits_offset=256
'bx' type_id=1 bits_offset=320
'r11' type_id=1 bits_offset=384
'r10' type_id=1 bits_offset=448
'r9' type_id=1 bits_offset=512
'r8' type_id=1 bits_offset=576
'ax' type_id=1 bits_offset=640
'cx' type_id=1 bits_offset=704
'dx' type_id=1 bits_offset=768
'si' type_id=1 bits_offset=832
'di' type_id=1 bits_offset=896
'orig_ax' type_id=1 bits_offset=960
'ip' type_id=1 bits_offset=1024
'cs' type_id=1 bits_offset=1088
'flags' type_id=1 bits_offset=1152
'sp' type_id=1 bits_offset=1216
'ss' type_id=1 bits_offset=1280
...
[4265] CONST '(anon)' type_id=172
...
[4270] PTR '(anon)' type_id=4265
...
[4282] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
'__unused' type_id=4270
...
[15690] FUNC '__x64_sys_recvmsg' type_id=4282 linkage=static
...
```
On Sat, Oct 15, 2022 at 8:33 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Oct 06, 2022 at 11:54:13AM +0200, Jiri Olsa wrote:
> > On Thu, Oct 06, 2022 at 04:34:46PM +0900, Akihiro HARAI wrote:
> > > Depending on distribution/kernel/syscall combination, BTF entry for
> > > `struct pt_regs *regs` parameter differs.
> > >
> > > For example, Amazon Linux 2 with kernel-5.15 package enabled has a FWD
> > > entry for `__x64_sys_recvmsg` function:
> > >
> > > ```
> > > $ uname -a
> > > Linux ip-10-1-1-66.ap-northeast-1.compute.internal
> > > 5.15.43-20.123.amzn2.x86_64 #1 SMP Fri May 27 00:28:44 UTC 2022 x86_64
> > > x86_64 x86_64 GNU/Linux
> > >
> > > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > > ...
> > > [15439] FWD 'pt_regs' fwd_kind=struct
> > > [15440] CONST '(anon)' type_id=15439
> > > [15441] PTR '(anon)' type_id=15440
> > > [15442] FUNC_PROTO '(anon)' ret_type_id=34 vlen=1
> > > '__unused' type_id=15441
> > > ...
> > > [15694] FUNC '__x64_sys_recvmsg' type_id=15442 linkage=static
> > > ...
> > > ```
> > >
> > > while Ubuntu 20.04 LTS with newer kernel has a STRUCT entry for the
> > > same function:
> > >
> > > ```
> > > $ uname -a
> > > Linux xxx-XPS-13-9300 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun
> > > 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw
> > > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > > ...
> > > [226] STRUCT 'pt_regs' size=168 vlen=21
> > > 'r15' type_id=1 bits_offset=0
> > > 'r14' type_id=1 bits_offset=64
> > > 'r13' type_id=1 bits_offset=128
> > > 'r12' type_id=1 bits_offset=192
> > > 'bp' type_id=1 bits_offset=256
> > > 'bx' type_id=1 bits_offset=320
> > > 'r11' type_id=1 bits_offset=384
> > > 'r10' type_id=1 bits_offset=448
> > > 'r9' type_id=1 bits_offset=512
> > > 'r8' type_id=1 bits_offset=576
> > > 'ax' type_id=1 bits_offset=640
> > > 'cx' type_id=1 bits_offset=704
> > > 'dx' type_id=1 bits_offset=768
> > > 'si' type_id=1 bits_offset=832
> > > 'di' type_id=1 bits_offset=896
> > > 'orig_ax' type_id=1 bits_offset=960
> > > 'ip' type_id=1 bits_offset=1024
> > > 'cs' type_id=1 bits_offset=1088
> > > 'flags' type_id=1 bits_offset=1152
> > > 'sp' type_id=1 bits_offset=1216
> > > 'ss' type_id=1 bits_offset=1280
> > > ...
> > > [5183] CONST '(anon)' type_id=226
> > > ...
> > > [5189] PTR '(anon)' type_id=5183
> > > ...
> > > [5321] FUNC_PROTO '(anon)' ret_type_id=42 vlen=1
> > > '__unused' type_id=5189
> > > ...
> > > [17648] FUNC '__x64_sys_recvmsg' type_id=5321 linkage=static
> > > ...
> > > ```
> > >
> > > Yet another distribution/kernel/syscall combination has multiple `FUNC
> > > '__x64_sys_[SYSCALL]'` entries, one for FWD and the other for STRUCT:
> > >
> > > ```
> > > $ uname -a
> > > Linux ip-10-5-0-115.ap-northeast-1.compute.internal
> > > 5.10.112-108.499.amzn2.x86_64 #1 SMP Wed Apr 27 23:39:40 UTC 2022
> > > x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > ```
> > > $ bpftool btf dump file /sys/kernel/btf/vmlinux format raw | grep
> > > __x64_sys_mprotect
> > > ...
> > > [175] STRUCT 'pt_regs' size=168 vlen=21
> > > 'r15' type_id=2 bits_offset=0
> > > 'r14' type_id=2 bits_offset=64
> > > 'r13' type_id=2 bits_offset=128
> > > 'r12' type_id=2 bits_offset=192
> > > 'bp' type_id=2 bits_offset=256
> > > ...
> > > [4215] CONST '(anon)' type_id=175
> > > ...
> > > [4220] PTR '(anon)' type_id=4215
> > > ...
> > > [6062] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > > 'regs' type_id=4220
> > > ...
> > > [11461] FWD 'pt_regs' fwd_kind=struct
> > > [11462] CONST '(anon)' type_id=11461
> > > [11463] PTR '(anon)' type_id=11462
> > > [11464] FUNC_PROTO '(anon)' ret_type_id=36 vlen=1
> > > '__unused' type_id=11463
> > > ...
> > > [11698] FUNC '__x64_sys_mprotect' type_id=11464 linkage=static
> > > ...
> > > [23528] FUNC '__x64_sys_mprotect' type_id=6062 linkage=static
> > > ...
> > > ```
> > >
> > > Trying to read `regs` parameter with FWD entry results in "invalid
> > > bpf_context access" error:
> > >
> > > ```
> > > SEC("fentry/__x64_sys_recvfrom")
> > > int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > > struct event t;
> > >
> > > bpf_get_current_comm(t.comm, TASK_COMM_LEN);
> > >
> > > u64 id = bpf_get_current_pid_tgid();
> > > t.pid = id >> 32;
> > >
> > > // This causes an error on some environments.
> > > t.fd = PT_REGS_PARM1_CORE(regs);
> > >
> > > bpf_printk("comm: %s, pid: %d, fd: %d", t.comm, t.pid, t.fd);
> > >
> > > return 0;
> > > ```
> > >
> > > ```
> > > $ sudo ./output
> > > 2022/07/01 03:33:01 loading objects: field FentrySyscall: program
> > > fentry_syscall: load program: permission denied:
> > > arg#0 type is not a struct
> > > Unrecognized arg#0 type PTR
> > > ; int BPF_PROG(fentry_syscall, struct pt_regs *regs) {
> > > 0: (79) r6 = *(u64 *)(r1 +0)
> > > func '__x64_sys_recvfrom' arg0 type FWD is not a struct
> > > invalid bpf_context access off=0 size=8
> > > processed 1 insns (limit 1000000) max_states_per_insn 0
> > > total_states 0 peak_states 0 mark_read 0
> > > ```
> > >
> > > Is this a bug related to toolchain?
> >
> > nice, I think it's specific to each object that defines syscall
> >
> > if such object has 'struct pt_regs' header with definition included
> > it will have full struct pt_regs, if not it will be just fwd ref
> >
> > not sure this would break anything else, but change below
> > fixes it for me
>
> hi,
> did this help? I'll send it as formal patch anyway,
> since it fixes the issue for me
>
> jirka
>
> >
> > jirka
> >
> >
> > ---
> > diff --git a/arch/x86/include/asm/syscall_wrapper.h b/arch/x86/include/asm/syscall_wrapper.h
> > index 59358d1bf880..fd2669b1cb2d 100644
> > --- a/arch/x86/include/asm/syscall_wrapper.h
> > +++ b/arch/x86/include/asm/syscall_wrapper.h
> > @@ -6,7 +6,7 @@
> > #ifndef _ASM_X86_SYSCALL_WRAPPER_H
> > #define _ASM_X86_SYSCALL_WRAPPER_H
> >
> > -struct pt_regs;
> > +#include <asm/ptrace.h>
> >
> > extern long __x64_sys_ni_syscall(const struct pt_regs *regs);
> > extern long __ia32_sys_ni_syscall(const struct pt_regs *regs);
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-10-16 0:49 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-06 7:34 Inconsistent BTF entries for `struct pt_regs *regs` parameter Akihiro HARAI
2022-10-06 9:54 ` Jiri Olsa
2022-10-15 11:33 ` Jiri Olsa
2022-10-16 0:49 ` Akihiro HARAI
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.