* [Syzkaller & bisect] There is general protection fault in bpf_prog_offload_verifier_prep in v6.6-rc1 kernel
@ 2023-09-15 3:46 Pengfei Xu
2023-09-15 16:02 ` Martin KaFai Lau
0 siblings, 1 reply; 3+ messages in thread
From: Pengfei Xu @ 2023-09-15 3:46 UTC (permalink / raw)
To: sdf; +Cc: bpf, heng.su, pengfei.xu, lkp
Hi Stanislav,
Greeting!
There is general protection fault in bpf_prog_offload_verifier_prep in
v6.6-rc1 kernel.
All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230914_154711_bpf_prog_offload_verifier_prep
Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.c
Syzkaller reproduced steps: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.prog
Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/bisect_info.log
Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/0bb80ecc33a8fb5a682236443c1e740d5c917d1d_dmesg.log
bzImage: https://github.com/xupengfe/syzkaller_logs/raw/main/230914_154711_bpf_prog_offload_verifier_prep/bzImage_0bb80ecc33a8fb5a682236443c1e740d5c917d1d.tar.gz
Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/kconfig_origin
Bisected and found suspected commit is:
2b3486bc2d23 bpf: Introduce device-bound XDP programs
"
[ 24.157409] general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN NOPTI
[ 24.158244] KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007]
[ 24.158778] CPU: 0 PID: 721 Comm: repro Not tainted 6.6.0-rc1-0bb80ecc33a8 #1
[ 24.159284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[ 24.160075] RIP: 0010:bpf_prog_offload_verifier_prep+0xb6/0x190
[ 24.160510] Code: 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 ae 00 00 00 48 b8 00 00 00 00 00 fc ff df 4c 8b 6b 10 4c 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 a3 00 00 00 48 b8 00 00 00 00 00 fc ff df 4d 8b
[ 24.161793] RSP: 0018:ffff8880103275e8 EFLAGS: 00010246
[ 24.162164] RAX: dffffc0000000000 RBX: ffff88801a707800 RCX: 0000000000000000
[ 24.162661] RDX: 0000000000000000 RSI: ffff8880146b8000 RDI: ffff88801a707810
[ 24.163158] RBP: ffff888010327600 R08: fffffbfff0db8716 R09: fffffbfff0db8716
[ 24.163656] R10: fffffbfff0db8715 R11: ffffffff86dc38af R12: ffffc900008f8000
[ 24.164153] R13: 0000000000000000 R14: ffffc900008f8004 R15: ffffc900008f8038
[ 24.164651] FS: 00007fce2a150740(0000) GS:ffff88806cc00000(0000) knlGS:0000000000000000
[ 24.165212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.165619] CR2: 0000000020000440 CR3: 00000000223ec005 CR4: 0000000000770ef0
[ 24.166118] PKRU: 55555554
[ 24.166317] Call Trace:
[ 24.166497] <TASK>
[ 24.166656] ? show_regs+0xa2/0xb0
[ 24.166920] ? __die_body+0x28/0x80
[ 24.167191] ? die_addr+0x5f/0xb0
[ 24.167447] ? exc_general_protection+0x190/0x340
[ 24.167805] ? asm_exc_general_protection+0x2b/0x30
[ 24.168171] ? bpf_prog_offload_verifier_prep+0xb6/0x190
[ 24.168573] ? bpf_prog_offload_verifier_prep+0x82/0x190
[ 24.168983] bpf_check+0x55ab/0xb270
[ 24.169283] ? __pfx_bpf_check+0x10/0x10
[ 24.169586] ? __pfx___lock_acquire+0x10/0x10
[ 24.169920] ? __this_cpu_preempt_check+0x20/0x30
[ 24.170271] ? lock_release+0x3f8/0x770
[ 24.170557] ? bpf_prog_load+0x1630/0x2370
[ 24.170859] ? __pfx_lock_release+0x10/0x10
[ 24.171174] ? __pfx_lock_acquire+0x10/0x10
[ 24.171490] ? ktime_get_with_offset+0x24a/0x290
[ 24.171836] ? bpf_prog_load+0x1630/0x2370
[ 24.172143] ? write_comp_data+0x2f/0x90
[ 24.172444] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 24.172804] bpf_prog_load+0x1732/0x2370
[ 24.173100] ? bpf_prog_load+0x1732/0x2370
[ 24.173411] ? __pfx_bpf_prog_load+0x10/0x10
[ 24.173738] ? lock_release+0x3f8/0x770
[ 24.174028] ? __this_cpu_preempt_check+0x20/0x30
[ 24.174380] ? __might_fault+0x102/0x1b0
[ 24.174683] ? __pfx_lock_release+0x10/0x10
[ 24.174998] ? __pfx_lock_acquire+0x10/0x10
[ 24.175319] ? write_comp_data+0x2f/0x90
[ 24.175614] ? write_comp_data+0x2f/0x90
[ 24.175913] __sys_bpf+0x18e7/0x66a0
[ 24.176185] ? __kasan_check_read+0x15/0x20
[ 24.176502] ? __pfx___sys_bpf+0x10/0x10
[ 24.176804] ? write_comp_data+0x2f/0x90
[ 24.177108] ? __pfx___lock_acquire+0x10/0x10
[ 24.177429] ? __sanitizer_cov_trace_pc+0x25/0x60
[ 24.177780] ? __this_cpu_preempt_check+0x20/0x30
[ 24.178132] ? lock_release+0x3f8/0x770
[ 24.178423] ? __audit_syscall_entry+0x3d5/0x540
[ 24.178773] ? __pfx_lock_release+0x10/0x10
[ 24.179089] ? __pfx_lock_acquire+0x10/0x10
[ 24.179405] ? ktime_get_coarse_real_ts64+0x181/0x1b0
[ 24.179778] ? __audit_syscall_entry+0x3d5/0x540
[ 24.180126] ? __this_cpu_preempt_check+0x20/0x30
[ 24.180476] ? write_comp_data+0x2f/0x90
[ 24.180776] __x64_sys_bpf+0x7e/0xc0
[ 24.180982] ? syscall_enter_from_user_mode+0x51/0x60
[ 24.181277] do_syscall_64+0x3b/0x90
[ 24.181499] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[ 24.181796] RIP: 0033:0x7fce29e3ee5d
[ 24.182014] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 93 af 1b 00 f7 d8 64 89 01 48
[ 24.183052] RSP: 002b:00007fffb9a02958 EFLAGS: 00000202 ORIG_RAX: 0000000000000141
[ 24.183485] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fce29e3ee5d
[ 24.183892] RDX: 0000000000000090 RSI: 0000000020000380 RDI: 0000000000000005
[ 24.184298] RBP: 00007fffb9a02970 R08: 00007fffb9a02970 R09: 00007fffb9a02970
[ 24.184709] R10: 00007fffb9a02970 R11: 0000000000000202 R12: 00007fffb9a02ae8
[ 24.185121] R13: 0000000000402bf3 R14: 0000000000404e08 R15: 00007fce2a195000
[ 24.185537] </TASK>
[ 24.185670] Modules linked in:
[ 24.185884] ---[ end trace 0000000000000000 ]---
[ 24.186155] RIP: 0010:bpf_prog_offload_verifier_prep+0xb6/0x190
[ 24.186507] Code: 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 ae 00 00 00 48 b8 00 00 00 00 00 fc ff df 4c 8b 6b 10 4c 89 ea 48 c1 ea 03 <80> 3c 02 00 0f 85 a3 00 00 00 48 b8 00 00 00 00 00 fc ff df 4d 8b
[ 24.188245] RSP: 0018:ffff8880103275e8 EFLAGS: 00010246
[ 24.188553] RAX: dffffc0000000000 RBX: ffff88801a707800 RCX: 0000000000000000
[ 24.188965] RDX: 0000000000000000 RSI: ffff8880146b8000 RDI: ffff88801a707810
[ 24.189373] RBP: ffff888010327600 R08: fffffbfff0db8716 R09: fffffbfff0db8716
[ 24.189779] R10: fffffbfff0db8715 R11: ffffffff86dc38af R12: ffffc900008f8000
[ 24.190188] R13: 0000000000000000 R14: ffffc900008f8004 R15: ffffc900008f8038
[ 24.190596] FS: 00007fce2a150740(0000) GS:ffff88806cc00000(0000) knlGS:0000000000000000
[ 24.191079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.191417] CR2: 0000000020000440 CR3: 00000000223ec005 CR4: 0000000000770ef0
[ 24.191829] PKRU: 55555554
"
I hope above info is helpful.
---
If you don't need the following environment to reproduce the problem or if you
already have one, please ignore the following information.
How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0
// start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
// You could change the bzImage_xxx as you want
// Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost
After login vm(virtual machine) successfully, you could transfer reproduced
binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/
Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config
make olddefconfig
make -jx bzImage //x should equal or less than cpu num your pc has
Fill the bzImage file into above start3.sh to load the target kernel in vm.
Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
yum -y install libslirp-devel.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp
make
make install
Best Regards,
Thanks!
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Syzkaller & bisect] There is general protection fault in bpf_prog_offload_verifier_prep in v6.6-rc1 kernel
2023-09-15 3:46 [Syzkaller & bisect] There is general protection fault in bpf_prog_offload_verifier_prep in v6.6-rc1 kernel Pengfei Xu
@ 2023-09-15 16:02 ` Martin KaFai Lau
2023-09-16 3:29 ` Pengfei Xu
0 siblings, 1 reply; 3+ messages in thread
From: Martin KaFai Lau @ 2023-09-15 16:02 UTC (permalink / raw)
To: Pengfei Xu; +Cc: bpf, heng.su, lkp, sdf
On 9/14/23 8:46 PM, Pengfei Xu wrote:
> Hi Stanislav,
>
> Greeting!
>
> There is general protection fault in bpf_prog_offload_verifier_prep in
> v6.6-rc1 kernel.
>
> All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230914_154711_bpf_prog_offload_verifier_prep
> Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.c
> Syzkaller reproduced steps: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.prog
> Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/bisect_info.log
> Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/0bb80ecc33a8fb5a682236443c1e740d5c917d1d_dmesg.log
> bzImage: https://github.com/xupengfe/syzkaller_logs/raw/main/230914_154711_bpf_prog_offload_verifier_prep/bzImage_0bb80ecc33a8fb5a682236443c1e740d5c917d1d.tar.gz
> Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/kconfig_origin
>
> Bisected and found suspected commit is:
> 2b3486bc2d23 bpf: Introduce device-bound XDP programs
Thanks for the report.
It has just been fixed in the following commit in the bpf tree:
commit 1a49f4195d3498fe458a7f5ff7ec5385da70d92e
Author: Eduard Zingerman <eddyz87@gmail.com>
Date: Mon Sep 11 17:55:37 2023
bpf: Avoid dummy bpf_offload_netdev in __bpf_prog_dev_bound_init
Fix for a bug observable under the following sequence of events:
1. Create a network device that does not support XDP offload.
2. Load a device bound XDP program with BPF_F_XDP_DEV_BOUND_ONLY flag
(such programs are not offloaded).
3. Load a device bound XDP program with zero flags
(such programs are offloaded).
At step (2) __bpf_prog_dev_bound_init() associates with device (1)
a dummy bpf_offload_netdev struct with .offdev field set to NULL.
At step (3) __bpf_prog_dev_bound_init() would reuse dummy struct
allocated at step (2).
However, downstream usage of the bpf_offload_netdev assumes that
.offdev field can't be NULL, e.g. in bpf_prog_offload_verifier_prep().
Adjust __bpf_prog_dev_bound_init() to require bpf_offload_netdev
with non-NULL .offdev for offloaded BPF programs.
Fixes: 2b3486bc2d23 ("bpf: Introduce device-bound XDP programs")
Reported-by: syzbot+291100dcb32190ec02a8@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/bpf/000000000000d97f3c060479c4f8@google.com/
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/r/20230912005539.2248244-2-eddyz87@gmail.com
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Syzkaller & bisect] There is general protection fault in bpf_prog_offload_verifier_prep in v6.6-rc1 kernel
2023-09-15 16:02 ` Martin KaFai Lau
@ 2023-09-16 3:29 ` Pengfei Xu
0 siblings, 0 replies; 3+ messages in thread
From: Pengfei Xu @ 2023-09-16 3:29 UTC (permalink / raw)
To: Martin KaFai Lau; +Cc: bpf, heng.su, lkp, sdf
Hi Martin,
On 2023-09-15 at 09:02:18 -0700, Martin KaFai Lau wrote:
> On 9/14/23 8:46 PM, Pengfei Xu wrote:
> > Hi Stanislav,
> >
> > Greeting!
> >
> > There is general protection fault in bpf_prog_offload_verifier_prep in
> > v6.6-rc1 kernel.
> >
> > All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230914_154711_bpf_prog_offload_verifier_prep
> > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.c
> > Syzkaller reproduced steps: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/repro.prog
> > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/bisect_info.log
> > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/0bb80ecc33a8fb5a682236443c1e740d5c917d1d_dmesg.log
> > bzImage: https://github.com/xupengfe/syzkaller_logs/raw/main/230914_154711_bpf_prog_offload_verifier_prep/bzImage_0bb80ecc33a8fb5a682236443c1e740d5c917d1d.tar.gz
> > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230914_154711_bpf_prog_offload_verifier_prep/kconfig_origin
> >
> > Bisected and found suspected commit is:
> > 2b3486bc2d23 bpf: Introduce device-bound XDP programs
>
> Thanks for the report.
>
> It has just been fixed in the following commit in the bpf tree:
Thanks for hints!
I will check Linux kernel community email carefully for the same issue
report next time.
I have tested the below fixed patch on top of v6.6-rc1 by kernel:
6.6.0-rc1-kvm-bpf-dirty, this issue in this email was gone, it's fixed by
below patch.
Best Regards,
Thanks!
>
> commit 1a49f4195d3498fe458a7f5ff7ec5385da70d92e
> Author: Eduard Zingerman <eddyz87@gmail.com>
> Date: Mon Sep 11 17:55:37 2023
>
> bpf: Avoid dummy bpf_offload_netdev in __bpf_prog_dev_bound_init
>
> Fix for a bug observable under the following sequence of events:
> 1. Create a network device that does not support XDP offload.
> 2. Load a device bound XDP program with BPF_F_XDP_DEV_BOUND_ONLY flag
> (such programs are not offloaded).
> 3. Load a device bound XDP program with zero flags
> (such programs are offloaded).
>
> At step (2) __bpf_prog_dev_bound_init() associates with device (1)
> a dummy bpf_offload_netdev struct with .offdev field set to NULL.
> At step (3) __bpf_prog_dev_bound_init() would reuse dummy struct
> allocated at step (2).
> However, downstream usage of the bpf_offload_netdev assumes that
> .offdev field can't be NULL, e.g. in bpf_prog_offload_verifier_prep().
>
> Adjust __bpf_prog_dev_bound_init() to require bpf_offload_netdev
> with non-NULL .offdev for offloaded BPF programs.
>
> Fixes: 2b3486bc2d23 ("bpf: Introduce device-bound XDP programs")
> Reported-by: syzbot+291100dcb32190ec02a8@syzkaller.appspotmail.com
> Closes: https://lore.kernel.org/bpf/000000000000d97f3c060479c4f8@google.com/
> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
> Link: https://lore.kernel.org/r/20230912005539.2248244-2-eddyz87@gmail.com
> Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-09-16 3:35 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-15 3:46 [Syzkaller & bisect] There is general protection fault in bpf_prog_offload_verifier_prep in v6.6-rc1 kernel Pengfei Xu
2023-09-15 16:02 ` Martin KaFai Lau
2023-09-16 3:29 ` Pengfei Xu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.