* How to disassemble a BPF program?
@ 2022-04-12 11:04 Tetsuo Handa
2022-04-20 11:38 ` Tetsuo Handa
0 siblings, 1 reply; 7+ messages in thread
From: Tetsuo Handa @ 2022-04-12 11:04 UTC (permalink / raw)
To: bpf; +Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Hello.
I'm not a BPF user, but I want to know what a BPF program stored in
"static const char program[2053] =" at
https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
is doing so that I can parse syzkaller-generated BPF programs like C programs.
Do you have a utility for this purpose?
Regards.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-12 11:04 How to disassemble a BPF program? Tetsuo Handa
@ 2022-04-20 11:38 ` Tetsuo Handa
2022-04-20 16:48 ` Andrii Nakryiko
0 siblings, 1 reply; 7+ messages in thread
From: Tetsuo Handa @ 2022-04-20 11:38 UTC (permalink / raw)
To: bpf; +Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
Ping?
Since how to fix this "current top five crasher" bug depends on how a kernel
socket is created via BPF program, this bug wants help from BPF developers.
On 2022/04/12 20:04, Tetsuo Handa wrote:
> Hello.
>
> I'm not a BPF user, but I want to know what a BPF program stored in
> "static const char program[2053] =" at
> https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
> is doing so that I can parse syzkaller-generated BPF programs like C programs.
>
> Do you have a utility for this purpose?
>
> Regards.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-20 11:38 ` Tetsuo Handa
@ 2022-04-20 16:48 ` Andrii Nakryiko
2022-04-21 7:17 ` Tetsuo Handa
2022-04-25 13:48 ` Jakub Sitnicki
0 siblings, 2 replies; 7+ messages in thread
From: Andrii Nakryiko @ 2022-04-20 16:48 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
On Wed, Apr 20, 2022 at 4:38 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> Ping?
>
> Since how to fix this "current top five crasher" bug depends on how a kernel
> socket is created via BPF program, this bug wants help from BPF developers.
If the BPF program is loaded/verified successfully, the easiest way to
go about this would be to prevent repro from proceeding right after
successful validation (e.g, do scanf()) and then use bpftool to find
that program's ID and dump disassembly while that program is in the
kernel.
$ sudo bpftool prog show
...
654439: cgroup_skb tag 6deef7357e7b4530 gpl
loaded_at 2022-04-20T06:14:08-0700 uid 0
xlated 64B jited 54B memlock 4096B
pids systemd(1)
$ sudo bpftool prog dump xlat id 654439
0: (bf) r6 = r1
1: (69) r7 = *(u16 *)(r6 +176)
2: (b4) w8 = 0
3: (44) w8 |= 2
4: (b7) r0 = 1
5: (55) if r8 != 0x2 goto pc+1
6: (b7) r0 = 0
7: (95) exit
Hope that helps. I don't know any tool that allows to disassemble raw
bytes into BPF assembly. Normally I use llvm-objdump to disassemble
well-formed BPF ELF files. Not sure if you can wrange llvm-objdump to
disassemble raw bytes without ELF file itself.
>
> On 2022/04/12 20:04, Tetsuo Handa wrote:
> > Hello.
> >
> > I'm not a BPF user, but I want to know what a BPF program stored in
> > "static const char program[2053] =" at
> > https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
> > is doing so that I can parse syzkaller-generated BPF programs like C programs.
> >
> > Do you have a utility for this purpose?
> >
> > Regards.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-20 16:48 ` Andrii Nakryiko
@ 2022-04-21 7:17 ` Tetsuo Handa
2022-04-21 11:15 ` Tetsuo Handa
2022-04-25 13:48 ` Jakub Sitnicki
1 sibling, 1 reply; 7+ messages in thread
From: Tetsuo Handa @ 2022-04-21 7:17 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
On 2022/04/21 1:48, Andrii Nakryiko wrote:
> If the BPF program is loaded/verified successfully, the easiest way to
> go about this would be to prevent repro from proceeding right after
> successful validation (e.g, do scanf()) and then use bpftool to find
> that program's ID and dump disassembly while that program is in the
> kernel.
Thank you for a command line example.
As of commit b253435746d9a4a7 ("Merge tag 'xtensa-20220416' of https://github.com/jcmvbkbc/linux-xtensa")
I got the following output with "tools/bpf/bpftool/bpftool prog dump xlat id $NUM".
----------------------------------------
0: (bf) r6 = r1
1: (b7) r7 = -1048575
2: (bf) r2 = r7
3: (bf) r1 = r6
4: (85) call bpf_skb_load_helper_8_no_cache#12742912
5: (75) if r0 s>= 0x0 goto pc+2
6: (ac) w0 ^= w0
7: (95) exit
8: (b7) r2 = 12582912
9: (bf) r1 = r6
10: (85) call bpf_skb_load_helper_8_no_cache#12742912
11: (75) if r0 s>= 0x0 goto pc+2
12: (ac) w0 ^= w0
13: (95) exit
14: (95) exit
----------------------------------------
I feel that amount of output above is too short for "char program[2053]".
How can TCP/IPv6 socket be created from this quite limited operations?
Since insn_cnt = 5 when bpf(BPF_PROG_LOAD) is called, am I failing to dump
some of programs in "char program[2053]" ?
const union bpf_attr attr = {
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
.insn_cnt = 5,
.insns = (unsigned long long) program,
.license = (unsigned long long) license,
};
const int bpf_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, 72);
----------------------------------------
[ 63.156414][ T2733] a.out (2733) used greatest stack depth: 11736 bytes left
[ 224.313093][ C0] general protection fault, probably for non-canonical address 0x6b6af3ebe92b6cab: 0000 [#1] PREEMPT SMP
[ 224.324686][ C0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.0-rc3-00016-gb253435746d9 #761
[ 224.337077][ C0] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 224.349213][ C0] RIP: 0010:tcp_retransmit_timer+0x33c/0xcc0
[ 224.357069][ C0] Code: 12 02 01 e8 a0 7c 5b 00 e9 22 fd ff ff 48 c7 c7 76 52 36 83 41 bd 0c 00 00 00 e8 bf 54 66 00 48 8b 45 30 48 8b 80 c0 02 00 00 <65> 48 ff 80 40 01 00 00 0f b6 4d 12 48 8b 5d 30 41 d3 fd 41 83 e5
[ 224.367728][ C0] RSP: 0018:ffffc90000003db8 EFLAGS: 00010282
[ 224.371159][ C0] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: 0000000000000001
[ 224.376266][ C0] RDX: 0000000000000000 RSI: ffffffff83365276 RDI: ffffffff8324665e
[ 224.380835][ C0] RBP: ffff88810aec9bc0 R08: ffff88800bf7c040 R09: 0000000000000001
[ 224.385458][ C0] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88810aec9cf0
[ 224.389093][ C0] R13: 000000000000000c R14: ffff88810b5d8040 R15: ffff88810aec9e48
[ 224.392236][ C0] FS: 0000000000000000(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[ 224.395757][ C0] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 224.397935][ C0] CR2: 00007ff7ddf6101c CR3: 0000000003e3a000 CR4: 00000000000506f0
[ 224.401532][ C0] Call Trace:
[ 224.402898][ C0] <IRQ>
[ 224.403963][ C0] ? mark_held_locks+0x49/0x70
[ 224.405609][ C0] ? ktime_get+0x1cb/0x260
[ 224.407147][ C0] ? lockdep_hardirqs_on+0x79/0x100
[ 224.409040][ C0] ? tcp_write_timer_handler+0x280/0x280
[ 224.410906][ C0] tcp_write_timer_handler+0x1c2/0x280
[ 224.412975][ C0] tcp_write_timer+0xa5/0x110
[ 224.414661][ C0] ? tcp_write_timer_handler+0x280/0x280
[ 224.416786][ C0] call_timer_fn+0xa6/0x300
[ 224.418609][ C0] __run_timers.part.0+0x209/0x320
[ 224.420428][ C0] run_timer_softirq+0x2c/0x60
[ 224.422104][ C0] __do_softirq+0x174/0x53f
[ 224.423762][ C0] __irq_exit_rcu+0xcb/0x120
[ 224.425402][ C0] irq_exit_rcu+0x5/0x20
[ 224.427016][ C0] sysvec_apic_timer_interrupt+0x8e/0xc0
[ 224.429080][ C0] </IRQ>
[ 224.430250][ C0] <TASK>
[ 224.431407][ C0] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 224.436669][ C0] RIP: 0010:default_idle+0xb/0x10
[ 224.438742][ C0] Code: 8b 04 25 40 af 01 00 f0 80 60 02 df c3 0f ae f0 0f ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d e3 b6 56 00 fb f4 <c3> cc cc cc cc 53 48 89 fb e8 67 fb fe ff 48 8b 15 a0 91 4e 02 89
[ 224.445032][ C0] RSP: 0018:ffffffff83e03ea8 EFLAGS: 00000206
[ 224.446996][ C0] RAX: 00000000000234bb RBX: ffffffff83e61a00 RCX: 0000000000000001
[ 224.450133][ C0] RDX: 0000000000000000 RSI: ffffffff832e9bc9 RDI: ffffffff8324665e
[ 224.453066][ C0] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000001
[ 224.455758][ C0] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 224.458531][ C0] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 224.461234][ C0] default_idle_call+0x54/0x90
[ 224.462885][ C0] do_idle+0x1f3/0x240
[ 224.464307][ C0] cpu_startup_entry+0x14/0x20
[ 224.465970][ C0] start_kernel+0x69c/0x6c1
[ 224.467689][ C0] secondary_startup_64_no_verify+0xc3/0xcb
[ 224.470086][ C0] </TASK>
[ 224.471219][ C0] Modules linked in:
[ 224.472849][ C0] ---[ end trace 0000000000000000 ]---
[ 224.474935][ C0] RIP: 0010:tcp_retransmit_timer+0x33c/0xcc0
[ 224.477175][ C0] Code: 12 02 01 e8 a0 7c 5b 00 e9 22 fd ff ff 48 c7 c7 76 52 36 83 41 bd 0c 00 00 00 e8 bf 54 66 00 48 8b 45 30 48 8b 80 c0 02 00 00 <65> 48 ff 80 40 01 00 00 0f b6 4d 12 48 8b 5d 30 41 d3 fd 41 83 e5
[ 224.484428][ C0] RSP: 0018:ffffc90000003db8 EFLAGS: 00010282
[ 224.486898][ C0] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: 0000000000000001
[ 224.489955][ C0] RDX: 0000000000000000 RSI: ffffffff83365276 RDI: ffffffff8324665e
[ 224.493134][ C0] RBP: ffff88810aec9bc0 R08: ffff88800bf7c040 R09: 0000000000000001
[ 224.496092][ C0] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88810aec9cf0
[ 224.499257][ C0] R13: 000000000000000c R14: ffff88810b5d8040 R15: ffff88810aec9e48
[ 224.502907][ C0] FS: 0000000000000000(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[ 224.506218][ C0] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 224.508647][ C0] CR2: 00007ff7ddf6101c CR3: 0000000003e3a000 CR4: 00000000000506f0
[ 224.511437][ C0] Kernel panic - not syncing: Fatal exception in interrupt
[ 224.514645][ C0] Kernel Offset: disabled
[ 224.516090][ C0] Rebooting in 10 seconds..
----------------------------------------
Also, I tried to find what bpf_skb_load_helper_8_no_cache() is doing
but I couldn't find the implementation of ____bpf_skb_load_helper_8().
Where is ____bpf_skb_load_helper_8() defined?
----------------------------------------
BPF_CALL_2(bpf_skb_load_helper_8_no_cache, const struct sk_buff *, skb,
int, offset)
{
return ____bpf_skb_load_helper_8(skb, skb->data, skb->len - skb->data_len,
offset);
}
----------------------------------------
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-21 7:17 ` Tetsuo Handa
@ 2022-04-21 11:15 ` Tetsuo Handa
2022-04-22 6:44 ` Tetsuo Handa
0 siblings, 1 reply; 7+ messages in thread
From: Tetsuo Handa @ 2022-04-21 11:15 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
On 2022/04/21 16:17, Tetsuo Handa wrote:
> Also, I tried to find what bpf_skb_load_helper_8_no_cache() is doing
> but I couldn't find the implementation of ____bpf_skb_load_helper_8().
> Where is ____bpf_skb_load_helper_8() defined?
>
> ----------------------------------------
> BPF_CALL_2(bpf_skb_load_helper_8_no_cache, const struct sk_buff *, skb,
> int, offset)
> {
> return ____bpf_skb_load_helper_8(skb, skb->data, skb->len - skb->data_len,
> offset);
> }
> ----------------------------------------
>
Ah, OK. Since BPF_CALL_x macro defines
static __always_inline \
u64 ____##name(__BPF_MAP(x, __BPF_DECL_ARGS, __BPF_V, __VA_ARGS__))
, BPF_CALL_4(bpf_skb_load_helper_8) will define
static __always_inline u64 ____bpf_skb_load_helper_8()
for to be called from BPF_CALL_2(bpf_skb_load_helper_8_no_cache).
> I feel that amount of output above is too short for "char program[2053]".
> How can TCP/IPv6 socket be created from this quite limited operations?
Since bpf_skb_load_helper_8() nothing but reads a byte, I don't think that
bpf(BPF_PROG_LOAD) / setsockopt(SOL_SOCKET, SO_ATTACH_BPF) can affect this
use-after-free bug, unless "char program[2053]" is doing something other
than reading a byte.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-21 11:15 ` Tetsuo Handa
@ 2022-04-22 6:44 ` Tetsuo Handa
0 siblings, 0 replies; 7+ messages in thread
From: Tetsuo Handa @ 2022-04-22 6:44 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
On 2022/04/21 20:15, Tetsuo Handa wrote:
>> I feel that amount of output above is too short for "char program[2053]".
>> How can TCP/IPv6 socket be created from this quite limited operations?
>
> Since bpf_skb_load_helper_8() nothing but reads a byte, I don't think that
> bpf(BPF_PROG_LOAD) / setsockopt(SOL_SOCKET, SO_ATTACH_BPF) can affect this
> use-after-free bug, unless "char program[2053]" is doing something other
> than reading a byte.
It turned out that only first 37 bytes of "char program[2053]" was meaningful.
const union bpf_attr attr = {
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
.insn_cnt = 5,
.insns = (unsigned long long)
"\xbf\x16\x00\x00\x00\x00\x00\x00\xb7\x07\x00\x00\x01\x00\xf0\xff"
"\x50\x70\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x00\x00\xc0\x00"
"\x95\x00\x00\x00\x00",
.license = (unsigned long long) "GPL",
};
Thus, I think that the output of "tools/bpf/bpftool/bpftool prog dump xlat id $NUM"
was correct.
Moreover, it turned out that even no-op program which does only
r0 = 0
exit
can trigger this use-after-free bug.
const union bpf_attr attr = {
.prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
.insn_cnt = 2,
.insns = (unsigned long long) "\xb7\x00\x00\x00\x00\x00\x00\x00\x95\x00\x00\x00\x00\x00\x00\x00",
.license = (unsigned long long) "GPL",
};
That is, it seems that loading a BPF program using bpf(BPF_PROG_LOAD) and attaching
that program using setsockopt(SOL_SOCKET, SO_ATTACH_BPF) itself somehow affects this
use-after-free bug.
Now, I suspect that some refcount is taken by setsockopt(SOL_SOCKET, SO_ATTACH_BPF)
and is causing the socket to stay alive long enough to fire tcp_retransmit_timer().
Any ideas?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to disassemble a BPF program?
2022-04-20 16:48 ` Andrii Nakryiko
2022-04-21 7:17 ` Tetsuo Handa
@ 2022-04-25 13:48 ` Jakub Sitnicki
1 sibling, 0 replies; 7+ messages in thread
From: Jakub Sitnicki @ 2022-04-25 13:48 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Tetsuo Handa, bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko
On Wed, Apr 20, 2022 at 09:48 AM -07, Andrii Nakryiko wrote:
> On Wed, Apr 20, 2022 at 4:38 AM Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>
>> Ping?
>>
>> Since how to fix this "current top five crasher" bug depends on how a kernel
>> socket is created via BPF program, this bug wants help from BPF developers.
>
> If the BPF program is loaded/verified successfully, the easiest way to
> go about this would be to prevent repro from proceeding right after
> successful validation (e.g, do scanf()) and then use bpftool to find
> that program's ID and dump disassembly while that program is in the
> kernel.
>
> $ sudo bpftool prog show
> ...
> 654439: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2022-04-20T06:14:08-0700 uid 0
> xlated 64B jited 54B memlock 4096B
> pids systemd(1)
>
> $ sudo bpftool prog dump xlat id 654439
> 0: (bf) r6 = r1
> 1: (69) r7 = *(u16 *)(r6 +176)
> 2: (b4) w8 = 0
> 3: (44) w8 |= 2
> 4: (b7) r0 = 1
> 5: (55) if r8 != 0x2 goto pc+1
> 6: (b7) r0 = 0
> 7: (95) exit
>
> Hope that helps. I don't know any tool that allows to disassemble raw
> bytes into BPF assembly. Normally I use llvm-objdump to disassemble
> well-formed BPF ELF files. Not sure if you can wrange llvm-objdump to
> disassemble raw bytes without ELF file itself.
You can disassemble raw BPF binaries with GNU objdump, but the assembly
mnemonics are different:
$ sudo bpftool prog dump xlated id 77
0: (bf) r6 = r1
1: (69) r7 = *(u16 *)(r6 +176)
2: (b4) w8 = 0
3: (44) w8 |= 2
4: (b7) r0 = 1
5: (55) if r8 != 0x2 goto pc+1
6: (b7) r0 = 0
7: (95) exit
$ sudo bpftool prog dump xlated id 77 file prog.bin
$ sudo objdump -D -b binary -m bpf prog.bin
prog.bin: file format binary
Disassembly of section .data:
0000000000000000 <.data>:
0: bf 16 00 00 00 00 00 00 mov %r6,%r1
8: 69 67 b0 00 00 00 00 00 ldxh %r7,[%r6+0xb0]
10: b4 08 00 00 00 00 00 00 mov32 %r8,0
18: 44 08 00 00 02 00 00 00 or32 %r8,2
20: b7 00 00 00 01 00 00 00 mov %r0,1
28: 55 08 01 00 02 00 00 00 jne %r8,2,1
30: b7 00 00 00 00 00 00 00 mov %r0,0
38: 95 00 00 00 00 00 00 00 exit
$
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-04-25 13:52 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-12 11:04 How to disassemble a BPF program? Tetsuo Handa
2022-04-20 11:38 ` Tetsuo Handa
2022-04-20 16:48 ` Andrii Nakryiko
2022-04-21 7:17 ` Tetsuo Handa
2022-04-21 11:15 ` Tetsuo Handa
2022-04-22 6:44 ` Tetsuo Handa
2022-04-25 13:48 ` Jakub Sitnicki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).