How to disassemble a BPF program?

How to disassemble a BPF program?

[Tetsuo Handa]

Hello.

I'm not a BPF user, but I want to know what a BPF program stored in
"static const char program[2053] =" at
https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
is doing so that I can parse syzkaller-generated BPF programs like C programs.

Do you have a utility for this purpose?

Regards.

Re: How to disassemble a BPF program?

[Tetsuo Handa]

Ping?

Since how to fix this "current top five crasher" bug depends on how a kernel
socket is created via BPF program, this bug wants help from BPF developers.

On 2022/04/12 20:04, Tetsuo Handa wrote:
> Hello.
> 
> I'm not a BPF user, but I want to know what a BPF program stored in
> "static const char program[2053] =" at
> https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
> is doing so that I can parse syzkaller-generated BPF programs like C programs.
> 
> Do you have a utility for this purpose?
> 
> Regards.

Re: How to disassemble a BPF program?

[Andrii Nakryiko]

On Wed, Apr 20, 2022 at 4:38 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> Ping?
>
> Since how to fix this "current top five crasher" bug depends on how a kernel
> socket is created via BPF program, this bug wants help from BPF developers.

If the BPF program is loaded/verified successfully, the easiest way to
go about this would be to prevent repro from proceeding right after
successful validation (e.g, do scanf()) and then use bpftool to find
that program's ID and dump disassembly while that program is in the
kernel.

$ sudo bpftool prog show
...
654439: cgroup_skb  tag 6deef7357e7b4530  gpl
        loaded_at 2022-04-20T06:14:08-0700  uid 0
        xlated 64B  jited 54B  memlock 4096B
        pids systemd(1)

$ sudo bpftool prog dump xlat id 654439
   0: (bf) r6 = r1
   1: (69) r7 = *(u16 *)(r6 +176)
   2: (b4) w8 = 0
   3: (44) w8 |= 2
   4: (b7) r0 = 1
   5: (55) if r8 != 0x2 goto pc+1
   6: (b7) r0 = 0
   7: (95) exit

Hope that helps. I don't know any tool that allows to disassemble raw
bytes into BPF assembly. Normally I use llvm-objdump to disassemble
well-formed BPF ELF files. Not sure if you can wrange llvm-objdump to
disassemble raw bytes without ELF file itself.

>
> On 2022/04/12 20:04, Tetsuo Handa wrote:
> > Hello.
> >
> > I'm not a BPF user, but I want to know what a BPF program stored in
> > "static const char program[2053] =" at
> > https://lkml.kernel.org/r/c389e47f-8f82-fd62-8c1d-d9481d2f71ff@I-love.SAKURA.ne.jp
> > is doing so that I can parse syzkaller-generated BPF programs like C programs.
> >
> > Do you have a utility for this purpose?
> >
> > Regards.

Re: How to disassemble a BPF program?

[Tetsuo Handa]

On 2022/04/21 1:48, Andrii Nakryiko wrote:
> If the BPF program is loaded/verified successfully, the easiest way to
> go about this would be to prevent repro from proceeding right after
> successful validation (e.g, do scanf()) and then use bpftool to find
> that program's ID and dump disassembly while that program is in the
> kernel.

Thank you for a command line example.

As of commit b253435746d9a4a7 ("Merge tag 'xtensa-20220416' of https://github.com/jcmvbkbc/linux-xtensa")
I got the following output with "tools/bpf/bpftool/bpftool prog dump xlat id $NUM".

----------------------------------------
   0: (bf) r6 = r1
   1: (b7) r7 = -1048575
   2: (bf) r2 = r7
   3: (bf) r1 = r6
   4: (85) call bpf_skb_load_helper_8_no_cache#12742912
   5: (75) if r0 s>= 0x0 goto pc+2
   6: (ac) w0 ^= w0
   7: (95) exit
   8: (b7) r2 = 12582912
   9: (bf) r1 = r6
  10: (85) call bpf_skb_load_helper_8_no_cache#12742912
  11: (75) if r0 s>= 0x0 goto pc+2
  12: (ac) w0 ^= w0
  13: (95) exit
  14: (95) exit
----------------------------------------

I feel that amount of output above is too short for "char program[2053]".
How can TCP/IPv6 socket be created from this quite limited operations?
Since insn_cnt = 5 when bpf(BPF_PROG_LOAD) is called, am I failing to dump
some of programs in "char program[2053]" ?

        const union bpf_attr attr = {
                .prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
                .insn_cnt = 5,
                .insns = (unsigned long long) program,
                .license = (unsigned long long) license,
        };
        const int bpf_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, 72);

----------------------------------------
[   63.156414][ T2733] a.out (2733) used greatest stack depth: 11736 bytes left
[  224.313093][    C0] general protection fault, probably for non-canonical address 0x6b6af3ebe92b6cab: 0000 [#1] PREEMPT SMP
[  224.324686][    C0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.0-rc3-00016-gb253435746d9 #761
[  224.337077][    C0] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  224.349213][    C0] RIP: 0010:tcp_retransmit_timer+0x33c/0xcc0
[  224.357069][    C0] Code: 12 02 01 e8 a0 7c 5b 00 e9 22 fd ff ff 48 c7 c7 76 52 36 83 41 bd 0c 00 00 00 e8 bf 54 66 00 48 8b 45 30 48 8b 80 c0 02 00 00 <65> 48 ff 80 40 01 00 00 0f b6 4d 12 48 8b 5d 30 41 d3 fd 41 83 e5
[  224.367728][    C0] RSP: 0018:ffffc90000003db8 EFLAGS: 00010282
[  224.371159][    C0] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: 0000000000000001
[  224.376266][    C0] RDX: 0000000000000000 RSI: ffffffff83365276 RDI: ffffffff8324665e
[  224.380835][    C0] RBP: ffff88810aec9bc0 R08: ffff88800bf7c040 R09: 0000000000000001
[  224.385458][    C0] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88810aec9cf0
[  224.389093][    C0] R13: 000000000000000c R14: ffff88810b5d8040 R15: ffff88810aec9e48
[  224.392236][    C0] FS:  0000000000000000(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[  224.395757][    C0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.397935][    C0] CR2: 00007ff7ddf6101c CR3: 0000000003e3a000 CR4: 00000000000506f0
[  224.401532][    C0] Call Trace:
[  224.402898][    C0]  <IRQ>
[  224.403963][    C0]  ? mark_held_locks+0x49/0x70
[  224.405609][    C0]  ? ktime_get+0x1cb/0x260
[  224.407147][    C0]  ? lockdep_hardirqs_on+0x79/0x100
[  224.409040][    C0]  ? tcp_write_timer_handler+0x280/0x280
[  224.410906][    C0]  tcp_write_timer_handler+0x1c2/0x280
[  224.412975][    C0]  tcp_write_timer+0xa5/0x110
[  224.414661][    C0]  ? tcp_write_timer_handler+0x280/0x280
[  224.416786][    C0]  call_timer_fn+0xa6/0x300
[  224.418609][    C0]  __run_timers.part.0+0x209/0x320
[  224.420428][    C0]  run_timer_softirq+0x2c/0x60
[  224.422104][    C0]  __do_softirq+0x174/0x53f
[  224.423762][    C0]  __irq_exit_rcu+0xcb/0x120
[  224.425402][    C0]  irq_exit_rcu+0x5/0x20
[  224.427016][    C0]  sysvec_apic_timer_interrupt+0x8e/0xc0
[  224.429080][    C0]  </IRQ>
[  224.430250][    C0]  <TASK>
[  224.431407][    C0]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[  224.436669][    C0] RIP: 0010:default_idle+0xb/0x10
[  224.438742][    C0] Code: 8b 04 25 40 af 01 00 f0 80 60 02 df c3 0f ae f0 0f ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d e3 b6 56 00 fb f4 <c3> cc cc cc cc 53 48 89 fb e8 67 fb fe ff 48 8b 15 a0 91 4e 02 89
[  224.445032][    C0] RSP: 0018:ffffffff83e03ea8 EFLAGS: 00000206
[  224.446996][    C0] RAX: 00000000000234bb RBX: ffffffff83e61a00 RCX: 0000000000000001
[  224.450133][    C0] RDX: 0000000000000000 RSI: ffffffff832e9bc9 RDI: ffffffff8324665e
[  224.453066][    C0] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000001
[  224.455758][    C0] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  224.458531][    C0] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  224.461234][    C0]  default_idle_call+0x54/0x90
[  224.462885][    C0]  do_idle+0x1f3/0x240
[  224.464307][    C0]  cpu_startup_entry+0x14/0x20
[  224.465970][    C0]  start_kernel+0x69c/0x6c1
[  224.467689][    C0]  secondary_startup_64_no_verify+0xc3/0xcb
[  224.470086][    C0]  </TASK>
[  224.471219][    C0] Modules linked in:
[  224.472849][    C0] ---[ end trace 0000000000000000 ]---
[  224.474935][    C0] RIP: 0010:tcp_retransmit_timer+0x33c/0xcc0
[  224.477175][    C0] Code: 12 02 01 e8 a0 7c 5b 00 e9 22 fd ff ff 48 c7 c7 76 52 36 83 41 bd 0c 00 00 00 e8 bf 54 66 00 48 8b 45 30 48 8b 80 c0 02 00 00 <65> 48 ff 80 40 01 00 00 0f b6 4d 12 48 8b 5d 30 41 d3 fd 41 83 e5
[  224.484428][    C0] RSP: 0018:ffffc90000003db8 EFLAGS: 00010282
[  224.486898][    C0] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: 0000000000000001
[  224.489955][    C0] RDX: 0000000000000000 RSI: ffffffff83365276 RDI: ffffffff8324665e
[  224.493134][    C0] RBP: ffff88810aec9bc0 R08: ffff88800bf7c040 R09: 0000000000000001
[  224.496092][    C0] R10: 0000000000000002 R11: 0000000000000000 R12: ffff88810aec9cf0
[  224.499257][    C0] R13: 000000000000000c R14: ffff88810b5d8040 R15: ffff88810aec9e48
[  224.502907][    C0] FS:  0000000000000000(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[  224.506218][    C0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.508647][    C0] CR2: 00007ff7ddf6101c CR3: 0000000003e3a000 CR4: 00000000000506f0
[  224.511437][    C0] Kernel panic - not syncing: Fatal exception in interrupt
[  224.514645][    C0] Kernel Offset: disabled
[  224.516090][    C0] Rebooting in 10 seconds..
----------------------------------------

Also, I tried to find what bpf_skb_load_helper_8_no_cache() is doing
but I couldn't find the implementation of ____bpf_skb_load_helper_8().
Where is ____bpf_skb_load_helper_8() defined?

----------------------------------------
BPF_CALL_2(bpf_skb_load_helper_8_no_cache, const struct sk_buff *, skb,
           int, offset)
{
        return ____bpf_skb_load_helper_8(skb, skb->data, skb->len - skb->data_len,
                                         offset);
}
----------------------------------------

Re: How to disassemble a BPF program?

[Tetsuo Handa]

On 2022/04/21 16:17, Tetsuo Handa wrote:
> Also, I tried to find what bpf_skb_load_helper_8_no_cache() is doing
> but I couldn't find the implementation of ____bpf_skb_load_helper_8().
> Where is ____bpf_skb_load_helper_8() defined?
> 
> ----------------------------------------
> BPF_CALL_2(bpf_skb_load_helper_8_no_cache, const struct sk_buff *, skb,
>            int, offset)
> {
>         return ____bpf_skb_load_helper_8(skb, skb->data, skb->len - skb->data_len,
>                                          offset);
> }
> ----------------------------------------
> 

Ah, OK. Since BPF_CALL_x macro defines

        static __always_inline                                                 \
        u64 ____##name(__BPF_MAP(x, __BPF_DECL_ARGS, __BPF_V, __VA_ARGS__))

, BPF_CALL_4(bpf_skb_load_helper_8) will define

	static __always_inline u64 ____bpf_skb_load_helper_8()

for to be called from BPF_CALL_2(bpf_skb_load_helper_8_no_cache).

> I feel that amount of output above is too short for "char program[2053]".
> How can TCP/IPv6 socket be created from this quite limited operations?

Since bpf_skb_load_helper_8() nothing but reads a byte, I don't think that
bpf(BPF_PROG_LOAD) / setsockopt(SOL_SOCKET, SO_ATTACH_BPF) can affect this
use-after-free bug, unless "char program[2053]" is doing something other
than reading a byte.

Re: How to disassemble a BPF program?

[Tetsuo Handa]

On 2022/04/21 20:15, Tetsuo Handa wrote:
>> I feel that amount of output above is too short for "char program[2053]".
>> How can TCP/IPv6 socket be created from this quite limited operations?
> 
> Since bpf_skb_load_helper_8() nothing but reads a byte, I don't think that
> bpf(BPF_PROG_LOAD) / setsockopt(SOL_SOCKET, SO_ATTACH_BPF) can affect this
> use-after-free bug, unless "char program[2053]" is doing something other
> than reading a byte.

It turned out that only first 37 bytes of "char program[2053]" was meaningful.

        const union bpf_attr attr = {
                .prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
                .insn_cnt = 5,
                .insns = (unsigned long long)
                "\xbf\x16\x00\x00\x00\x00\x00\x00\xb7\x07\x00\x00\x01\x00\xf0\xff"
                "\x50\x70\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x00\x00\xc0\x00"
                "\x95\x00\x00\x00\x00",
                .license = (unsigned long long) "GPL",
        };

Thus, I think that the output of "tools/bpf/bpftool/bpftool prog dump xlat id $NUM"
was correct.

Moreover, it turned out that even no-op program which does only

	r0 = 0
	exit

can trigger this use-after-free bug.

        const union bpf_attr attr = {
                .prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
                .insn_cnt = 2,
                .insns = (unsigned long long) "\xb7\x00\x00\x00\x00\x00\x00\x00\x95\x00\x00\x00\x00\x00\x00\x00",
                .license = (unsigned long long) "GPL",
        };

That is, it seems that loading a BPF program using bpf(BPF_PROG_LOAD) and attaching
that program using setsockopt(SOL_SOCKET, SO_ATTACH_BPF) itself somehow affects this
use-after-free bug.

Now, I suspect that some refcount is taken by setsockopt(SOL_SOCKET, SO_ATTACH_BPF)
and is causing the socket to stay alive long enough to fire tcp_retransmit_timer().
Any ideas?

Re: How to disassemble a BPF program?

[Jakub Sitnicki]

On Wed, Apr 20, 2022 at 09:48 AM -07, Andrii Nakryiko wrote:
> On Wed, Apr 20, 2022 at 4:38 AM Tetsuo Handa
> <penguin-kernel@i-love.sakura.ne.jp> wrote:
>>
>> Ping?
>>
>> Since how to fix this "current top five crasher" bug depends on how a kernel
>> socket is created via BPF program, this bug wants help from BPF developers.
>
> If the BPF program is loaded/verified successfully, the easiest way to
> go about this would be to prevent repro from proceeding right after
> successful validation (e.g, do scanf()) and then use bpftool to find
> that program's ID and dump disassembly while that program is in the
> kernel.
>
> $ sudo bpftool prog show
> ...
> 654439: cgroup_skb  tag 6deef7357e7b4530  gpl
>         loaded_at 2022-04-20T06:14:08-0700  uid 0
>         xlated 64B  jited 54B  memlock 4096B
>         pids systemd(1)
>
> $ sudo bpftool prog dump xlat id 654439
>    0: (bf) r6 = r1
>    1: (69) r7 = *(u16 *)(r6 +176)
>    2: (b4) w8 = 0
>    3: (44) w8 |= 2
>    4: (b7) r0 = 1
>    5: (55) if r8 != 0x2 goto pc+1
>    6: (b7) r0 = 0
>    7: (95) exit
>
> Hope that helps. I don't know any tool that allows to disassemble raw
> bytes into BPF assembly. Normally I use llvm-objdump to disassemble
> well-formed BPF ELF files. Not sure if you can wrange llvm-objdump to
> disassemble raw bytes without ELF file itself.

You can disassemble raw BPF binaries with GNU objdump, but the assembly
mnemonics are different:

$ sudo bpftool prog dump xlated id 77
   0: (bf) r6 = r1
   1: (69) r7 = *(u16 *)(r6 +176)
   2: (b4) w8 = 0
   3: (44) w8 |= 2
   4: (b7) r0 = 1
   5: (55) if r8 != 0x2 goto pc+1
   6: (b7) r0 = 0
   7: (95) exit
$ sudo bpftool prog dump xlated id 77 file prog.bin
$ sudo objdump -D -b binary -m bpf prog.bin

prog.bin:     file format binary


Disassembly of section .data:

0000000000000000 <.data>:
   0:   bf 16 00 00 00 00 00 00         mov %r6,%r1
   8:   69 67 b0 00 00 00 00 00         ldxh %r7,[%r6+0xb0]
  10:   b4 08 00 00 00 00 00 00         mov32 %r8,0
  18:   44 08 00 00 02 00 00 00         or32 %r8,2
  20:   b7 00 00 00 01 00 00 00         mov %r0,1
  28:   55 08 01 00 02 00 00 00         jne %r8,2,1
  30:   b7 00 00 00 00 00 00 00         mov %r0,0
  38:   95 00 00 00 00 00 00 00         exit
$