* verifier backtracking bug
@ 2023-05-12 18:55 Alexei Starovoitov
2023-05-12 22:16 ` Andrii Nakryiko
0 siblings, 1 reply; 3+ messages in thread
From: Alexei Starovoitov @ 2023-05-12 18:55 UTC (permalink / raw)
To: Andrii Nakryiko, bpf
Andrii,
Here is what I see on the latest bpf-next:
./test_progs -t global_funcs
[ 7.969549] bpf_testmod: loading out-of-tree module taints kernel.
[ 7.979444] ------------[ cut here ]------------
[ 7.979812] verifier backtracking bug
[ 7.979828] WARNING: CPU: 1 PID: 2026 at kernel/bpf/verifier.c:3500
__mark_chain_precision+0xd8d/0xda0
[ 7.980818] Modules linked in: bpf_testmod(O)
[ 7.981161] CPU: 1 PID: 2026 Comm: test_progs Tainted: G
O 6.3.0-07968-g7b99f75942da #4614
[ 7.981876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
[ 7.982732] RIP: 0010:__mark_chain_precision+0xd8d/0xda0
[ 7.983140] Code: ff e9 fb f4 ff ff 80 3d e2 c5 50 02 00 0f 85 15
fd ff ff 48 c7 c7 fe 5b 5c 82 4c 89 0c 24 c6 05 ca c5 50 02 01 e8 b3
ed e8 ff <0f> 0b 4c 8b 0c 24 e9 f3 fc ff ff 0f4
[ 7.984523] RSP: 0018:ffffc90002bb78f0 EFLAGS: 00010282
[ 7.984918] RAX: 0000000000000019 RBX: ffff88810137c000 RCX: 0000000000000002
[ 7.985467] RDX: 0000000080000002 RSI: ffffffff825bda2c RDI: 00000000ffffffff
[ 7.986011] RBP: 00000000ffffffff R08: 0000000000000000 R09: c0000000fffeffff
[ 7.986553] R10: 0000000000000001 R11: ffffc90002bb77a8 R12: 000000000000001b
[ 7.987093] R13: 0000000000000002 R14: 0000000000000010 R15: 000000000000001c
[ 7.987641] FS: 00007f7bd27d7400(0000) GS:ffff888237a40000(0000)
knlGS:0000000000000000
[ 7.988254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7.988687] CR2: 000000000511e078 CR3: 000000010512f005 CR4: 00000000003706e0
[ 7.989228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7.989765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 7.990306] Call Trace:
[ 7.990500] <TASK>
[ 7.990668] ? check_helper_mem_access+0xf9/0x2a0
[ 7.991035] ? btf_type_name+0x20/0x20
[ 7.991329] ? find_kfunc_desc_btf.part.106+0x210/0x210
[ 7.991723] check_stack_write_fixed_off+0x437/0x610
[ 7.992113] ? lock_acquire+0x15c/0x290
[ 7.992416] ? adjust_reg_min_max_vals+0xdf/0x1070
[ 7.992778] ? __kmem_cache_alloc_node+0x41/0x530
[ 7.993140] ? check_ptr_alignment+0x7d/0x210
[ 7.993479] ? lock_release+0x1b7/0x250
[ 7.993774] check_mem_access+0x8fc/0x1750
Looks like my earlier suggestion to do:
WARN_ONCE(idx + 1 != subseq_idx, "verifier backtracking bug");
is tripping on something.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: verifier backtracking bug
2023-05-12 18:55 verifier backtracking bug Alexei Starovoitov
@ 2023-05-12 22:16 ` Andrii Nakryiko
2023-05-16 17:32 ` Andrii Nakryiko
0 siblings, 1 reply; 3+ messages in thread
From: Andrii Nakryiko @ 2023-05-12 22:16 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: Andrii Nakryiko, bpf
On Fri, May 12, 2023 at 11:55 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> Andrii,
>
> Here is what I see on the latest bpf-next:
>
> ./test_progs -t global_funcs
> [ 7.969549] bpf_testmod: loading out-of-tree module taints kernel.
> [ 7.979444] ------------[ cut here ]------------
> [ 7.979812] verifier backtracking bug
> [ 7.979828] WARNING: CPU: 1 PID: 2026 at kernel/bpf/verifier.c:3500
> __mark_chain_precision+0xd8d/0xda0
> [ 7.980818] Modules linked in: bpf_testmod(O)
> [ 7.981161] CPU: 1 PID: 2026 Comm: test_progs Tainted: G
> O 6.3.0-07968-g7b99f75942da #4614
> [ 7.981876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
> [ 7.982732] RIP: 0010:__mark_chain_precision+0xd8d/0xda0
> [ 7.983140] Code: ff e9 fb f4 ff ff 80 3d e2 c5 50 02 00 0f 85 15
> fd ff ff 48 c7 c7 fe 5b 5c 82 4c 89 0c 24 c6 05 ca c5 50 02 01 e8 b3
> ed e8 ff <0f> 0b 4c 8b 0c 24 e9 f3 fc ff ff 0f4
> [ 7.984523] RSP: 0018:ffffc90002bb78f0 EFLAGS: 00010282
> [ 7.984918] RAX: 0000000000000019 RBX: ffff88810137c000 RCX: 0000000000000002
> [ 7.985467] RDX: 0000000080000002 RSI: ffffffff825bda2c RDI: 00000000ffffffff
> [ 7.986011] RBP: 00000000ffffffff R08: 0000000000000000 R09: c0000000fffeffff
> [ 7.986553] R10: 0000000000000001 R11: ffffc90002bb77a8 R12: 000000000000001b
> [ 7.987093] R13: 0000000000000002 R14: 0000000000000010 R15: 000000000000001c
> [ 7.987641] FS: 00007f7bd27d7400(0000) GS:ffff888237a40000(0000)
> knlGS:0000000000000000
> [ 7.988254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 7.988687] CR2: 000000000511e078 CR3: 000000010512f005 CR4: 00000000003706e0
> [ 7.989228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 7.989765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 7.990306] Call Trace:
> [ 7.990500] <TASK>
> [ 7.990668] ? check_helper_mem_access+0xf9/0x2a0
> [ 7.991035] ? btf_type_name+0x20/0x20
> [ 7.991329] ? find_kfunc_desc_btf.part.106+0x210/0x210
> [ 7.991723] check_stack_write_fixed_off+0x437/0x610
> [ 7.992113] ? lock_acquire+0x15c/0x290
> [ 7.992416] ? adjust_reg_min_max_vals+0xdf/0x1070
> [ 7.992778] ? __kmem_cache_alloc_node+0x41/0x530
> [ 7.993140] ? check_ptr_alignment+0x7d/0x210
> [ 7.993479] ? lock_release+0x1b7/0x250
> [ 7.993774] check_mem_access+0x8fc/0x1750
>
> Looks like my earlier suggestion to do:
> WARN_ONCE(idx + 1 != subseq_idx, "verifier backtracking bug");
>
> is tripping on something.
Interesting... I did check dmesg after adding this check, strange.
I'll try to repro later today and see what's up, thanks for heads up!
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: verifier backtracking bug
2023-05-12 22:16 ` Andrii Nakryiko
@ 2023-05-16 17:32 ` Andrii Nakryiko
0 siblings, 0 replies; 3+ messages in thread
From: Andrii Nakryiko @ 2023-05-16 17:32 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: Andrii Nakryiko, bpf
On Fri, May 12, 2023 at 3:16 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Fri, May 12, 2023 at 11:55 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > Andrii,
> >
> > Here is what I see on the latest bpf-next:
> >
> > ./test_progs -t global_funcs
> > [ 7.969549] bpf_testmod: loading out-of-tree module taints kernel.
> > [ 7.979444] ------------[ cut here ]------------
> > [ 7.979812] verifier backtracking bug
> > [ 7.979828] WARNING: CPU: 1 PID: 2026 at kernel/bpf/verifier.c:3500
> > __mark_chain_precision+0xd8d/0xda0
> > [ 7.980818] Modules linked in: bpf_testmod(O)
> > [ 7.981161] CPU: 1 PID: 2026 Comm: test_progs Tainted: G
> > O 6.3.0-07968-g7b99f75942da #4614
> > [ 7.981876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
> > [ 7.982732] RIP: 0010:__mark_chain_precision+0xd8d/0xda0
> > [ 7.983140] Code: ff e9 fb f4 ff ff 80 3d e2 c5 50 02 00 0f 85 15
> > fd ff ff 48 c7 c7 fe 5b 5c 82 4c 89 0c 24 c6 05 ca c5 50 02 01 e8 b3
> > ed e8 ff <0f> 0b 4c 8b 0c 24 e9 f3 fc ff ff 0f4
> > [ 7.984523] RSP: 0018:ffffc90002bb78f0 EFLAGS: 00010282
> > [ 7.984918] RAX: 0000000000000019 RBX: ffff88810137c000 RCX: 0000000000000002
> > [ 7.985467] RDX: 0000000080000002 RSI: ffffffff825bda2c RDI: 00000000ffffffff
> > [ 7.986011] RBP: 00000000ffffffff R08: 0000000000000000 R09: c0000000fffeffff
> > [ 7.986553] R10: 0000000000000001 R11: ffffc90002bb77a8 R12: 000000000000001b
> > [ 7.987093] R13: 0000000000000002 R14: 0000000000000010 R15: 000000000000001c
> > [ 7.987641] FS: 00007f7bd27d7400(0000) GS:ffff888237a40000(0000)
> > knlGS:0000000000000000
> > [ 7.988254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 7.988687] CR2: 000000000511e078 CR3: 000000010512f005 CR4: 00000000003706e0
> > [ 7.989228] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 7.989765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [ 7.990306] Call Trace:
> > [ 7.990500] <TASK>
> > [ 7.990668] ? check_helper_mem_access+0xf9/0x2a0
> > [ 7.991035] ? btf_type_name+0x20/0x20
> > [ 7.991329] ? find_kfunc_desc_btf.part.106+0x210/0x210
> > [ 7.991723] check_stack_write_fixed_off+0x437/0x610
> > [ 7.992113] ? lock_acquire+0x15c/0x290
> > [ 7.992416] ? adjust_reg_min_max_vals+0xdf/0x1070
> > [ 7.992778] ? __kmem_cache_alloc_node+0x41/0x530
> > [ 7.993140] ? check_ptr_alignment+0x7d/0x210
> > [ 7.993479] ? lock_release+0x1b7/0x250
> > [ 7.993774] check_mem_access+0x8fc/0x1750
> >
> > Looks like my earlier suggestion to do:
> > WARN_ONCE(idx + 1 != subseq_idx, "verifier backtracking bug");
> >
> > is tripping on something.
>
> Interesting... I did check dmesg after adding this check, strange.
> I'll try to repro later today and see what's up, thanks for heads up!
To close the loop, this was fixed in [0]. The problem was that
subseq_idx is not always preserved correctly when traversing between
states.
[0] https://patchwork.kernel.org/project/netdevbpf/patch/20230515180710.1535018-1-andrii@kernel.org/
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-05-16 17:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-12 18:55 verifier backtracking bug Alexei Starovoitov
2023-05-12 22:16 ` Andrii Nakryiko
2023-05-16 17:32 ` Andrii Nakryiko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).