* [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 @ 2023-05-22 2:07 Pengfei Xu 2023-05-22 6:39 ` Bagas Sanjaya 0 siblings, 1 reply; 14+ messages in thread From: Pengfei Xu @ 2023-05-22 2:07 UTC (permalink / raw) To: djwong; +Cc: linux-xfs, linux-fsdevel, heng.su, dchinner, lkp Hi Darrick, Greeting! There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3: Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: " f6b384631e1e xfs: give xfs_extfree_intent its own perag reference " report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log v6.4-rc3 reproduced info: " [ 91.419498] loop0: detected capacity change from 0 to 65536 [ 91.420095] XFS: attr2 mount option is deprecated. [ 91.420500] XFS: ikeep mount option is deprecated. [ 91.422379] XFS (loop0): Deprecated V4 format (crc=0) will not be supported after September 2030. [ 91.423468] XFS (loop0): Mounting V4 Filesystem d28317a9-9e04-4f2a-be27-e55b4c413ff6 [ 91.428169] XFS (loop0): Ending clean mount [ 91.429120] XFS (loop0): Quotacheck needed: Please wait. [ 91.432182] BUG: kernel NULL pointer dereference, address: 0000000000000008 [ 91.432770] #PF: supervisor read access in kernel mode [ 91.433216] #PF: error_code(0x0000) - not-present page [ 91.433640] PGD 0 P4D 0 [ 91.433864] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 91.434232] CPU: 0 PID: 33 Comm: kworker/u4:2 Not tainted 6.4.0-rc3-kvm #2 [ 91.434793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015 [ 91.435445] Workqueue: xfs_iwalk-393 xfs_pwork_work [ 91.435855] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 [ 91.436312] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 [ 91.437812] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 [ 91.438250] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 [ 91.438840] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 [ 91.439430] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 [ 91.440019] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 [ 91.440610] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 [ 91.441202] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 [ 91.441864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 91.442343] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 [ 91.442941] PKRU: 55555554 [ 91.443178] Call Trace: [ 91.443394] <TASK> [ 91.443585] list_sort+0xb8/0x3a0 [ 91.443885] xfs_extent_free_create_intent+0xb6/0xc0 [ 91.444312] xfs_defer_create_intents+0xc3/0x220 [ 91.444711] ? write_comp_data+0x2f/0x90 [ 91.445056] xfs_defer_finish_noroll+0x9e/0xbc0 [ 91.445449] ? list_sort+0x344/0x3a0 [ 91.445768] __xfs_trans_commit+0x4be/0x630 [ 91.446135] xfs_trans_commit+0x20/0x30 [ 91.446473] xfs_dquot_disk_alloc+0x45d/0x4e0 [ 91.446860] xfs_qm_dqread+0x2f7/0x310 [ 91.447192] xfs_qm_dqget+0xd5/0x300 [ 91.447506] xfs_qm_quotacheck_dqadjust+0x5a/0x230 [ 91.447921] xfs_qm_dqusage_adjust+0x249/0x300 [ 91.448313] xfs_iwalk_ag_recs+0x1bd/0x2e0 [ 91.448671] xfs_iwalk_run_callbacks+0xc3/0x1c0 [ 91.449071] xfs_iwalk_ag+0x32e/0x3f0 [ 91.449398] xfs_iwalk_ag_work+0xbe/0xf0 [ 91.449744] xfs_pwork_work+0x2c/0xc0 [ 91.450064] process_one_work+0x3b1/0x860 [ 91.450416] worker_thread+0x52/0x660 [ 91.450739] ? __pfx_worker_thread+0x10/0x10 [ 91.451113] kthread+0x16d/0x1c0 [ 91.451406] ? __pfx_kthread+0x10/0x10 [ 91.451740] ret_from_fork+0x29/0x50 [ 91.452064] </TASK> [ 91.452261] Modules linked in: [ 91.452530] CR2: 0000000000000008 [ 91.452819] ---[ end trace 0000000000000000 ]--- [ 91.487979] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 [ 91.488463] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 [ 91.490021] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 [ 91.490472] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 [ 91.491080] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 [ 91.491689] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 [ 91.492298] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 [ 91.492909] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 [ 91.493516] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 [ 91.494199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 91.494695] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 [ 91.495306] PKRU: 55555554 [ 91.495549] note: kworker/u4:2[33] exited with irqs disabled " I hope it's helpful. Thanks! --- If you don't need the following environment to reproduce the problem or if you already have one, please ignore the following information. How to reproduce: git clone https://gitlab.com/xupengfe/repro_vm_env.git cd repro_vm_env tar -xvf repro_vm_env.tar.gz cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0 // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel // You could change the bzImage_xxx as you want // Maybe you need to remove line "-drive if=pflash,format=raw,readonly=on,file=./OVMF_CODE.fd \" for different qemu version You could use below command to log in, there is no password for root. ssh -p 10023 root@localhost After login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm: gcc -pthread -o repro repro.c scp -P 10023 repro root@localhost:/root/ Get the bzImage for target kernel: Please use target kconfig and copy it to kernel_src/.config make olddefconfig make -jx bzImage //x should equal or less than cpu num your pc has Fill the bzImage file into above start3.sh to load the target kernel in vm. Tips: If you already have qemu-system-x86_64, please ignore below info. If you want to install qemu v7.1.0 version: git clone https://github.com/qemu/qemu.git cd qemu git checkout -f v7.1.0 mkdir build cd build yum install -y ninja-build.x86_64 yum -y install libslirp-devel.x86_64 ../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl --enable-usb-redir --enable-slirp make make install Thanks! BR. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-22 2:07 [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 Pengfei Xu @ 2023-05-22 6:39 ` Bagas Sanjaya 2023-05-22 16:05 ` Darrick J. Wong 0 siblings, 1 reply; 14+ messages in thread From: Bagas Sanjaya @ 2023-05-22 6:39 UTC (permalink / raw) To: Pengfei Xu, djwong Cc: linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions [-- Attachment #1: Type: text/plain, Size: 6205 bytes --] On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > Hi Darrick, > > Greeting! > There is BUG: unable to handle kernel NULL pointer dereference in > xfs_extent_free_diff_items in v6.4-rc3: > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > " > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > " > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > v6.4-rc3 reproduced info: > " > [ 91.419498] loop0: detected capacity change from 0 to 65536 > [ 91.420095] XFS: attr2 mount option is deprecated. > [ 91.420500] XFS: ikeep mount option is deprecated. > [ 91.422379] XFS (loop0): Deprecated V4 format (crc=0) will not be supported after September 2030. > [ 91.423468] XFS (loop0): Mounting V4 Filesystem d28317a9-9e04-4f2a-be27-e55b4c413ff6 > [ 91.428169] XFS (loop0): Ending clean mount > [ 91.429120] XFS (loop0): Quotacheck needed: Please wait. > [ 91.432182] BUG: kernel NULL pointer dereference, address: 0000000000000008 > [ 91.432770] #PF: supervisor read access in kernel mode > [ 91.433216] #PF: error_code(0x0000) - not-present page > [ 91.433640] PGD 0 P4D 0 > [ 91.433864] Oops: 0000 [#1] PREEMPT SMP NOPTI > [ 91.434232] CPU: 0 PID: 33 Comm: kworker/u4:2 Not tainted 6.4.0-rc3-kvm #2 > [ 91.434793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015 > [ 91.435445] Workqueue: xfs_iwalk-393 xfs_pwork_work > [ 91.435855] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 > [ 91.436312] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 > [ 91.437812] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 > [ 91.438250] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 > [ 91.438840] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 > [ 91.439430] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 > [ 91.440019] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 > [ 91.440610] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 > [ 91.441202] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 > [ 91.441864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 91.442343] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 > [ 91.442941] PKRU: 55555554 > [ 91.443178] Call Trace: > [ 91.443394] <TASK> > [ 91.443585] list_sort+0xb8/0x3a0 > [ 91.443885] xfs_extent_free_create_intent+0xb6/0xc0 > [ 91.444312] xfs_defer_create_intents+0xc3/0x220 > [ 91.444711] ? write_comp_data+0x2f/0x90 > [ 91.445056] xfs_defer_finish_noroll+0x9e/0xbc0 > [ 91.445449] ? list_sort+0x344/0x3a0 > [ 91.445768] __xfs_trans_commit+0x4be/0x630 > [ 91.446135] xfs_trans_commit+0x20/0x30 > [ 91.446473] xfs_dquot_disk_alloc+0x45d/0x4e0 > [ 91.446860] xfs_qm_dqread+0x2f7/0x310 > [ 91.447192] xfs_qm_dqget+0xd5/0x300 > [ 91.447506] xfs_qm_quotacheck_dqadjust+0x5a/0x230 > [ 91.447921] xfs_qm_dqusage_adjust+0x249/0x300 > [ 91.448313] xfs_iwalk_ag_recs+0x1bd/0x2e0 > [ 91.448671] xfs_iwalk_run_callbacks+0xc3/0x1c0 > [ 91.449071] xfs_iwalk_ag+0x32e/0x3f0 > [ 91.449398] xfs_iwalk_ag_work+0xbe/0xf0 > [ 91.449744] xfs_pwork_work+0x2c/0xc0 > [ 91.450064] process_one_work+0x3b1/0x860 > [ 91.450416] worker_thread+0x52/0x660 > [ 91.450739] ? __pfx_worker_thread+0x10/0x10 > [ 91.451113] kthread+0x16d/0x1c0 > [ 91.451406] ? __pfx_kthread+0x10/0x10 > [ 91.451740] ret_from_fork+0x29/0x50 > [ 91.452064] </TASK> > [ 91.452261] Modules linked in: > [ 91.452530] CR2: 0000000000000008 > [ 91.452819] ---[ end trace 0000000000000000 ]--- > [ 91.487979] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 > [ 91.488463] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 > [ 91.490021] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 > [ 91.490472] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 > [ 91.491080] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 > [ 91.491689] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 > [ 91.492298] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 > [ 91.492909] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 > [ 91.493516] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 > [ 91.494199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 91.494695] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 > [ 91.495306] PKRU: 55555554 > [ 91.495549] note: kworker/u4:2[33] exited with irqs disabled > " > Thanks for the regression report. I'm adding it to regzbot: #regzbot ^introduced: f6b384631e1e34 #regzbot title: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items (due to xfs_extfree_intent perag change) #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=217470 -- An old man doll... just what I always wanted! - Clara [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-22 6:39 ` Bagas Sanjaya @ 2023-05-22 16:05 ` Darrick J. Wong 2023-05-22 17:05 ` Linux regression tracking (Thorsten Leemhuis) 2023-05-23 0:00 ` Eric Biggers 0 siblings, 2 replies; 14+ messages in thread From: Darrick J. Wong @ 2023-05-22 16:05 UTC (permalink / raw) To: Bagas Sanjaya Cc: Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > Hi Darrick, > > > > Greeting! > > There is BUG: unable to handle kernel NULL pointer dereference in > > xfs_extent_free_diff_items in v6.4-rc3: > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > " > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > " > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > v6.4-rc3 reproduced info: Diagnosis and patches welcomed. Or are we doing the usual syzbot bullshit where you all assume that I'm going to do all the fucking work for you? --D > > [ 91.419498] loop0: detected capacity change from 0 to 65536 > > [ 91.420095] XFS: attr2 mount option is deprecated. > > [ 91.420500] XFS: ikeep mount option is deprecated. > > [ 91.422379] XFS (loop0): Deprecated V4 format (crc=0) will not be supported after September 2030. > > [ 91.423468] XFS (loop0): Mounting V4 Filesystem d28317a9-9e04-4f2a-be27-e55b4c413ff6 > > [ 91.428169] XFS (loop0): Ending clean mount > > [ 91.429120] XFS (loop0): Quotacheck needed: Please wait. > > [ 91.432182] BUG: kernel NULL pointer dereference, address: 0000000000000008 > > [ 91.432770] #PF: supervisor read access in kernel mode > > [ 91.433216] #PF: error_code(0x0000) - not-present page > > [ 91.433640] PGD 0 P4D 0 > > [ 91.433864] Oops: 0000 [#1] PREEMPT SMP NOPTI > > [ 91.434232] CPU: 0 PID: 33 Comm: kworker/u4:2 Not tainted 6.4.0-rc3-kvm #2 > > [ 91.434793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015 > > [ 91.435445] Workqueue: xfs_iwalk-393 xfs_pwork_work > > [ 91.435855] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 > > [ 91.436312] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 > > [ 91.437812] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 > > [ 91.438250] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 > > [ 91.438840] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 > > [ 91.439430] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 > > [ 91.440019] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 > > [ 91.440610] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 > > [ 91.441202] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 > > [ 91.441864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 91.442343] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 > > [ 91.442941] PKRU: 55555554 > > [ 91.443178] Call Trace: > > [ 91.443394] <TASK> > > [ 91.443585] list_sort+0xb8/0x3a0 > > [ 91.443885] xfs_extent_free_create_intent+0xb6/0xc0 > > [ 91.444312] xfs_defer_create_intents+0xc3/0x220 > > [ 91.444711] ? write_comp_data+0x2f/0x90 > > [ 91.445056] xfs_defer_finish_noroll+0x9e/0xbc0 > > [ 91.445449] ? list_sort+0x344/0x3a0 > > [ 91.445768] __xfs_trans_commit+0x4be/0x630 > > [ 91.446135] xfs_trans_commit+0x20/0x30 > > [ 91.446473] xfs_dquot_disk_alloc+0x45d/0x4e0 > > [ 91.446860] xfs_qm_dqread+0x2f7/0x310 > > [ 91.447192] xfs_qm_dqget+0xd5/0x300 > > [ 91.447506] xfs_qm_quotacheck_dqadjust+0x5a/0x230 > > [ 91.447921] xfs_qm_dqusage_adjust+0x249/0x300 > > [ 91.448313] xfs_iwalk_ag_recs+0x1bd/0x2e0 > > [ 91.448671] xfs_iwalk_run_callbacks+0xc3/0x1c0 > > [ 91.449071] xfs_iwalk_ag+0x32e/0x3f0 > > [ 91.449398] xfs_iwalk_ag_work+0xbe/0xf0 > > [ 91.449744] xfs_pwork_work+0x2c/0xc0 > > [ 91.450064] process_one_work+0x3b1/0x860 > > [ 91.450416] worker_thread+0x52/0x660 > > [ 91.450739] ? __pfx_worker_thread+0x10/0x10 > > [ 91.451113] kthread+0x16d/0x1c0 > > [ 91.451406] ? __pfx_kthread+0x10/0x10 > > [ 91.451740] ret_from_fork+0x29/0x50 > > [ 91.452064] </TASK> > > [ 91.452261] Modules linked in: > > [ 91.452530] CR2: 0000000000000008 > > [ 91.452819] ---[ end trace 0000000000000000 ]--- > > [ 91.487979] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 > > [ 91.488463] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 > > [ 91.490021] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 > > [ 91.490472] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 > > [ 91.491080] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 > > [ 91.491689] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 > > [ 91.492298] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 > > [ 91.492909] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 > > [ 91.493516] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 > > [ 91.494199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 91.494695] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 > > [ 91.495306] PKRU: 55555554 > > [ 91.495549] note: kworker/u4:2[33] exited with irqs disabled > > " > > > > Thanks for the regression report. I'm adding it to regzbot: > > #regzbot ^introduced: f6b384631e1e34 > #regzbot title: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items (due to xfs_extfree_intent perag change) > #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=217470 > > -- > An old man doll... just what I always wanted! - Clara ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-22 16:05 ` Darrick J. Wong @ 2023-05-22 17:05 ` Linux regression tracking (Thorsten Leemhuis) 2023-05-23 6:08 ` Bagas Sanjaya 2023-05-23 0:00 ` Eric Biggers 1 sibling, 1 reply; 14+ messages in thread From: Linux regression tracking (Thorsten Leemhuis) @ 2023-05-22 17:05 UTC (permalink / raw) To: Darrick J. Wong, Bagas Sanjaya Cc: Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On 22.05.23 18:05, Darrick J. Wong wrote: > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: >> On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: >>> Greeting! >>> There is BUG: unable to handle kernel NULL pointer dereference in >>> xfs_extent_free_diff_items in v6.4-rc3: >>> >>> Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. >>> >>> Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: >>> " >>> f6b384631e1e xfs: give xfs_extfree_intent its own perag reference >>> " >>> >>> report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items >>> Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c >>> Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog >>> Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin >>> Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log >>> Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log >>> >>> v6.4-rc3 reproduced info: > > Diagnosis and patches welcomed. > > Or are we doing the usual syzbot bullshit where you all assume that I'm > going to do all the fucking work for you? Darrick, sorry for the trouble. Bagas recently out of the blue started to help with adding regressions to the tracking. That's great, but OTOH it means that it's likely time to write a few things up that are obvious to some of us and myself. Bagas, please for the foreseeable future don't add regressions found by syzkaller to the regression tracking, unless some well known developer actually looked into the issue and indicated that it's something that needs to be fixed. Syzbot is great. But it occasionally does odd things or goes of the rails. And in can easily find problems that didn't happen in an earlier version, but are unlikely to be encountered by users in practice (aka "in the wild"). And we normally don't consider those regressions that needs to be fixed. Ciao, Thorsten #regzbot inconclusive: syzbot regression that would need further analysis >>> [ 91.419498] loop0: detected capacity change from 0 to 65536 >>> [ 91.420095] XFS: attr2 mount option is deprecated. >>> [ 91.420500] XFS: ikeep mount option is deprecated. >>> [ 91.422379] XFS (loop0): Deprecated V4 format (crc=0) will not be supported after September 2030. >>> [ 91.423468] XFS (loop0): Mounting V4 Filesystem d28317a9-9e04-4f2a-be27-e55b4c413ff6 >>> [ 91.428169] XFS (loop0): Ending clean mount >>> [ 91.429120] XFS (loop0): Quotacheck needed: Please wait. >>> [ 91.432182] BUG: kernel NULL pointer dereference, address: 0000000000000008 >>> [ 91.432770] #PF: supervisor read access in kernel mode >>> [ 91.433216] #PF: error_code(0x0000) - not-present page >>> [ 91.433640] PGD 0 P4D 0 >>> [ 91.433864] Oops: 0000 [#1] PREEMPT SMP NOPTI >>> [ 91.434232] CPU: 0 PID: 33 Comm: kworker/u4:2 Not tainted 6.4.0-rc3-kvm #2 >>> [ 91.434793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015 >>> [ 91.435445] Workqueue: xfs_iwalk-393 xfs_pwork_work >>> [ 91.435855] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 >>> [ 91.436312] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 >>> [ 91.437812] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 >>> [ 91.438250] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 >>> [ 91.438840] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 >>> [ 91.439430] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 >>> [ 91.440019] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 >>> [ 91.440610] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 >>> [ 91.441202] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 >>> [ 91.441864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 91.442343] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 >>> [ 91.442941] PKRU: 55555554 >>> [ 91.443178] Call Trace: >>> [ 91.443394] <TASK> >>> [ 91.443585] list_sort+0xb8/0x3a0 >>> [ 91.443885] xfs_extent_free_create_intent+0xb6/0xc0 >>> [ 91.444312] xfs_defer_create_intents+0xc3/0x220 >>> [ 91.444711] ? write_comp_data+0x2f/0x90 >>> [ 91.445056] xfs_defer_finish_noroll+0x9e/0xbc0 >>> [ 91.445449] ? list_sort+0x344/0x3a0 >>> [ 91.445768] __xfs_trans_commit+0x4be/0x630 >>> [ 91.446135] xfs_trans_commit+0x20/0x30 >>> [ 91.446473] xfs_dquot_disk_alloc+0x45d/0x4e0 >>> [ 91.446860] xfs_qm_dqread+0x2f7/0x310 >>> [ 91.447192] xfs_qm_dqget+0xd5/0x300 >>> [ 91.447506] xfs_qm_quotacheck_dqadjust+0x5a/0x230 >>> [ 91.447921] xfs_qm_dqusage_adjust+0x249/0x300 >>> [ 91.448313] xfs_iwalk_ag_recs+0x1bd/0x2e0 >>> [ 91.448671] xfs_iwalk_run_callbacks+0xc3/0x1c0 >>> [ 91.449071] xfs_iwalk_ag+0x32e/0x3f0 >>> [ 91.449398] xfs_iwalk_ag_work+0xbe/0xf0 >>> [ 91.449744] xfs_pwork_work+0x2c/0xc0 >>> [ 91.450064] process_one_work+0x3b1/0x860 >>> [ 91.450416] worker_thread+0x52/0x660 >>> [ 91.450739] ? __pfx_worker_thread+0x10/0x10 >>> [ 91.451113] kthread+0x16d/0x1c0 >>> [ 91.451406] ? __pfx_kthread+0x10/0x10 >>> [ 91.451740] ret_from_fork+0x29/0x50 >>> [ 91.452064] </TASK> >>> [ 91.452261] Modules linked in: >>> [ 91.452530] CR2: 0000000000000008 >>> [ 91.452819] ---[ end trace 0000000000000000 ]--- >>> [ 91.487979] RIP: 0010:xfs_extent_free_diff_items+0x27/0x40 >>> [ 91.488463] Code: 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 e8 05 73 7d ff 49 8b 44 24 28 48 8b 53 28 5b 41 5c <8b> 40 08 5d 2b 42 08 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 >>> [ 91.490021] RSP: 0000:ffffc9000012b8c0 EFLAGS: 00010246 >>> [ 91.490472] RAX: 0000000000000000 RBX: ffff8880015826c8 RCX: ffffffff81d71e41 >>> [ 91.491080] RDX: 0000000000000000 RSI: ffff888001ca4800 RDI: 0000000000000002 >>> [ 91.491689] RBP: ffffc9000012b8c0 R08: ffffc9000012b8e0 R09: 0000000000000000 >>> [ 91.492298] R10: ffff88800613f290 R11: ffffffff83e426c0 R12: ffff888001582230 >>> [ 91.492909] R13: ffff888001582428 R14: ffffffff81b042c0 R15: ffffc9000012b908 >>> [ 91.493516] FS: 0000000000000000(0000) GS:ffff88807ec00000(0000) knlGS:0000000000000000 >>> [ 91.494199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 91.494695] CR2: 0000000000000008 CR3: 000000000ed22006 CR4: 0000000000770ef0 >>> [ 91.495306] PKRU: 55555554 >>> [ 91.495549] note: kworker/u4:2[33] exited with irqs disabled >>> " >>> >> >> Thanks for the regression report. I'm adding it to regzbot: >> >> #regzbot ^introduced: f6b384631e1e34 >> #regzbot title: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items (due to xfs_extfree_intent perag change) >> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=217470 >> >> -- >> An old man doll... just what I always wanted! - Clara > > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-22 17:05 ` Linux regression tracking (Thorsten Leemhuis) @ 2023-05-23 6:08 ` Bagas Sanjaya 2023-05-23 6:44 ` Pengfei Xu 0 siblings, 1 reply; 14+ messages in thread From: Bagas Sanjaya @ 2023-05-23 6:08 UTC (permalink / raw) To: Linux regressions mailing list, Darrick J. Wong Cc: Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp On 5/23/23 00:05, Linux regression tracking (Thorsten Leemhuis) wrote: > Darrick, sorry for the trouble. Bagas recently out of the blue started > to help with adding regressions to the tracking. That's great, but OTOH > it means that it's likely time to write a few things up that are obvious > to some of us and myself. > > Bagas, please for the foreseeable future don't add regressions found by > syzkaller to the regression tracking, unless some well known developer > actually looked into the issue and indicated that it's something that > needs to be fixed. > > Syzbot is great. But it occasionally does odd things or goes of the > rails. And in can easily find problems that didn't happen in an earlier > version, but are unlikely to be encountered by users in practice (aka > "in the wild"). And we normally don't consider those regressions that > needs to be fixed. > Oops, at the moment I didn't know how to distinguish true regressions and issues found by the bot, so I thought that both are regressions. Thanks for the tip! -- An old man doll... just what I always wanted! - Clara ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 6:08 ` Bagas Sanjaya @ 2023-05-23 6:44 ` Pengfei Xu 0 siblings, 0 replies; 14+ messages in thread From: Pengfei Xu @ 2023-05-23 6:44 UTC (permalink / raw) To: Bagas Sanjaya Cc: Linux regressions mailing list, Darrick J. Wong, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp Hi Bagas Sanjaya, On 2023-05-23 at 13:08:32 +0700, Bagas Sanjaya wrote: > On 5/23/23 00:05, Linux regression tracking (Thorsten Leemhuis) wrote: > > Darrick, sorry for the trouble. Bagas recently out of the blue started > > to help with adding regressions to the tracking. That's great, but OTOH > > it means that it's likely time to write a few things up that are obvious > > to some of us and myself. > > > > Bagas, please for the foreseeable future don't add regressions found by > > syzkaller to the regression tracking, unless some well known developer > > actually looked into the issue and indicated that it's something that > > needs to be fixed. > > > > Syzbot is great. But it occasionally does odd things or goes of the > > rails. And in can easily find problems that didn't happen in an earlier > > version, but are unlikely to be encountered by users in practice (aka > > "in the wild"). And we normally don't consider those regressions that > > needs to be fixed. > > > > Oops, at the moment I didn't know how to distinguish true regressions > and issues found by the bot, so I thought that both are regressions. > > Thanks for the tip! > The bisect used keyword "xfs_extent_free_diff_items" to bisect, and seems it's not accurate this time. I will consider improving it. Thanks! BR. > -- > An old man doll... just what I always wanted! - Clara > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-22 16:05 ` Darrick J. Wong 2023-05-22 17:05 ` Linux regression tracking (Thorsten Leemhuis) @ 2023-05-23 0:00 ` Eric Biggers 2023-05-23 7:31 ` Dave Chinner 1 sibling, 1 reply; 14+ messages in thread From: Eric Biggers @ 2023-05-23 0:00 UTC (permalink / raw) To: Darrick J. Wong Cc: Bagas Sanjaya, Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Mon, May 22, 2023 at 09:05:25AM -0700, Darrick J. Wong wrote: > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > > Hi Darrick, > > > > > > Greeting! > > > There is BUG: unable to handle kernel NULL pointer dereference in > > > xfs_extent_free_diff_items in v6.4-rc3: > > > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > > " > > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > > " > > > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > > > v6.4-rc3 reproduced info: > > Diagnosis and patches welcomed. > > Or are we doing the usual syzbot bullshit where you all assume that I'm > going to do all the fucking work for you? > It looks like Pengfei already took the time to manually bisect this issue to a very recent commit authored by you. Is that not helpful? (Apologies if I didn't include enough profanities for this email to be suitable for linux-xfs@.) - Eric ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 0:00 ` Eric Biggers @ 2023-05-23 7:31 ` Dave Chinner 2023-05-23 9:14 ` Pengfei Xu 2023-05-23 16:50 ` Eric Biggers 0 siblings, 2 replies; 14+ messages in thread From: Dave Chinner @ 2023-05-23 7:31 UTC (permalink / raw) To: Eric Biggers Cc: Darrick J. Wong, Bagas Sanjaya, Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Tue, May 23, 2023 at 12:00:29AM +0000, Eric Biggers wrote: > On Mon, May 22, 2023 at 09:05:25AM -0700, Darrick J. Wong wrote: > > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > > > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > > > Hi Darrick, > > > > > > > > Greeting! > > > > There is BUG: unable to handle kernel NULL pointer dereference in > > > > xfs_extent_free_diff_items in v6.4-rc3: > > > > > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > > > " > > > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > > > " > > > > > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > > > > > v6.4-rc3 reproduced info: > > > > Diagnosis and patches welcomed. > > > > Or are we doing the usual syzbot bullshit where you all assume that I'm > > going to do all the fucking work for you? > > > > It looks like Pengfei already took the time to manually bisect this issue to a > very recent commit authored by you. Is that not helpful? No. The bisect is completely meaningless. The cause of the problem is going to be some piece of corrupted metadata has got through a verifier check or log recovery and has resulted in a perag lookup failing. The bisect landed on the commit where the perag dependency was introduced; whatever is letting unchecked corrupted metadata throught he verifiers has existed long before this recent change was made. I've already spent two hours analysing this report - I've got to the point where I've isolated the transaction in the trace, I see the allocation being run as expected, I see all the right things happening, and then it goes splat after the allocation has committed and it starts processing defered extent free operations. Neither the code nor the trace actually tell me anything about the nature of the failure that has occurred. At this point, I still don't know where the corrupted metadata is coming from. That's the next thing I need to look at, and then I realised that this bug report *doesn't include a pointer to the corrupted filesystem image that is being mounted*. IOWs, the bug report is deficient and not complete, and so I'm forced to spend unnecessary time trying to work out how to extract the filesystem image from a weird syzkaller report that is basically just a bunch of undocumented blobs in a github tree. This is the same sort of shit we've been having to deal rigth from teh start with syzkaller. It doesn't matter that syzbot might have improved it's reporting a bit these days, we still have to deal with this sort of poor reporting from all the private syzkaller bot crank handles that are being turned by people who know little more than how to turn a crank handle. To make matters worse, this is a v4 filesystem which has known unfixable issues when handling corrupted filesystems in both log replay and in runtime detection of corruption. We've repeatedly told people running syzkaller (including Pengfei) to stop running it on v4 filesystems and only report bugs on V5 format filesystems. This is to avoid wasting time triaging these problems back down to v4 specific format bugs that ican only be fixed by moving to the v5 format. ..... And now after 4 hours, I have found several corruptions in the on disk format that v5 filesystems will have caught and v4 filesystems will not. The AGFL indexes in the AGF have been corrupted. They are within valid bounds, but first + last != count. On a V5 filesystem we catch this and trigger an AGFL reset that is done of the first allocation. v4 filesystems do not do this last - first = count validation at all. Further, the AGFL has also been corrupted - it is full of null blocks. This is another problem that V5 filesystems can catch and report, but v4 filesystems don't because they don't have headers in the AGFL that enable verification. Yes, there's definitely scope for further improvements in validation here, but the unhandled corruptions that I've found still don't explain how we got a null perag in the xefi created from a referenced perag that is causing the crash. So, yeah, the bisect is completely useless, and I've got half a day into triage and I still don't have any clue what the corruption is that is causing the kernel to crash.... ---- Do you see the problem now, Eric? Performing root-cause analysis of syzkaller based malicious filesystem corruption bugs is anything but simple. It takes hours to days just to work through triage of a single bug report, and we're getting a couple of these sorts of bug reported every week. People who do nothing but turn the bot crank handle throw stuff like this over the wall at usi are easy to find. Bots and bot crank turners scale really easily. Engineers who can find and fix the problems, OTOH, don't. And just to rub salt into the wounds, we now have people who turn crank handles on other bots to tell everyone else how important they think the problem is without having performed any triage at all. And then we're expected to make an untriaged bug report our highest priority and immediately spend hours of time to make sense of the steaming pile that has just been dumped on us. Worse, we've had people who track regressions imply that if we don't prioritise fixing regressions ahead of anything else we might be working on, then we might not get new work merged until the regressions have been fixed. In my book, that's akin to extortion, and it might give you some insight to why Darrick reacted so vigorously to having an untriaged syzkaller bug tracked as a high visibility, must fix regression. What we really need is more people who are capable to triaging bug reports like this instead of having lots of people cranking on bot handles and dumping untriaged bug reports on the maintainer. Further, if you aren't capable of triaging the bug report, then you aren't qualified to classify it as a "must fix" regression. It's like people don't have any common sense or decency anymore: it's not very nice to classify a bug as a "must fix" regression without first having consulted the engineers responsible for that code. If you don't know what the cause of the bug is, then don't crank handles that cause people to have to address it immediately! If nothing changes, then the ever increasing amount of bot cranking is going to burn us out completely. Nobody wins when that happens.... -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 7:31 ` Dave Chinner @ 2023-05-23 9:14 ` Pengfei Xu 2023-05-23 21:52 ` Dave Chinner 2023-05-23 16:50 ` Eric Biggers 1 sibling, 1 reply; 14+ messages in thread From: Pengfei Xu @ 2023-05-23 9:14 UTC (permalink / raw) To: Dave Chinner Cc: Eric Biggers, Darrick J. Wong, Bagas Sanjaya, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions Hi Dave, On 2023-05-23 at 17:31:23 +1000, Dave Chinner wrote: > On Tue, May 23, 2023 at 12:00:29AM +0000, Eric Biggers wrote: > > On Mon, May 22, 2023 at 09:05:25AM -0700, Darrick J. Wong wrote: > > > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > > > > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > > > > Hi Darrick, > > > > > > > > > > Greeting! > > > > > There is BUG: unable to handle kernel NULL pointer dereference in > > > > > xfs_extent_free_diff_items in v6.4-rc3: > > > > > > > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > > > > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > > > > " > > > > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > > > > " > > > > > > > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > > > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > > > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > > > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > > > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > > > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > > > > > > > v6.4-rc3 reproduced info: > > > > > > Diagnosis and patches welcomed. > > > > > > Or are we doing the usual syzbot bullshit where you all assume that I'm > > > going to do all the fucking work for you? > > > > > > > It looks like Pengfei already took the time to manually bisect this issue to a > > very recent commit authored by you. Is that not helpful? > > No. The bisect is completely meaningless. > > The cause of the problem is going to be some piece of corrupted > metadata has got through a verifier check or log recovery and has > resulted in a perag lookup failing. The bisect landed on the commit > where the perag dependency was introduced; whatever is letting > unchecked corrupted metadata throught he verifiers has existed long > before this recent change was made. > > I've already spent two hours analysing this report - I've got to the > point where I've isolated the transaction in the trace, I see the > allocation being run as expected, I see all the right things > happening, and then it goes splat after the allocation has committed > and it starts processing defered extent free operations. Neither the > code nor the trace actually tell me anything about the nature of the > failure that has occurred. > > At this point, I still don't know where the corrupted metadata is > coming from. That's the next thing I need to look at, and then I > realised that this bug report *doesn't include a pointer to the > corrupted filesystem image that is being mounted*. > > IOWs, the bug report is deficient and not complete, and so I'm > forced to spend unnecessary time trying to work out how to extract > the filesystem image from a weird syzkaller report that is basically > just a bunch of undocumented blobs in a github tree. > > This is the same sort of shit we've been having to deal rigth from > teh start with syzkaller. It doesn't matter that syzbot might have > improved it's reporting a bit these days, we still have to deal with > this sort of poor reporting from all the private syzkaller bot crank > handles that are being turned by people who know little more than > how to turn a crank handle. > > To make matters worse, this is a v4 filesystem which has known > unfixable issues when handling corrupted filesystems in both log > replay and in runtime detection of corruption. We've repeatedly told > people running syzkaller (including Pengfei) to stop running it on > v4 filesystems and only report bugs on V5 format filesystems. This > is to avoid wasting time triaging these problems back down to v4 > specific format bugs that ican only be fixed by moving to the v5 > format. > > ..... > > And now after 4 hours, I have found several corruptions in the on > disk format that v5 filesystems will have caught and v4 filesystems > will not. > > The AGFL indexes in the AGF have been corrupted. They are within > valid bounds, but first + last != count. On a V5 filesystem we catch > this and trigger an AGFL reset that is done of the first allocation. > v4 filesystems do not do this last - first = count validation at > all. > > Further, the AGFL has also been corrupted - it is full of null > blocks. This is another problem that V5 filesystems can catch and > report, but v4 filesystems don't because they don't have headers in > the AGFL that enable verification. > > Yes, there's definitely scope for further improvements in validation > here, but the unhandled corruptions that I've found still don't > explain how we got a null perag in the xefi created from a > referenced perag that is causing the crash. > > So, yeah, the bisect is completely useless, and I've got half a day > into triage and I still don't have any clue what the corruption is > that is causing the kernel to crash.... > > ---- > > Do you see the problem now, Eric? > > Performing root-cause analysis of syzkaller based malicious > filesystem corruption bugs is anything but simple. It takes hours to > days just to work through triage of a single bug report, and we're > getting a couple of these sorts of bug reported every week. > > People who do nothing but turn the bot crank handle throw stuff like > this over the wall at usi are easy to find. Bots and bot crank > turners scale really easily. Engineers who can find and fix the > problems, OTOH, don't. > > And just to rub salt into the wounds, we now have people who turn > crank handles on other bots to tell everyone else how important > they think the problem is without having performed any triage at > all. And then we're expected to make an untriaged bug report our > highest priority and immediately spend hours of time to make sense > of the steaming pile that has just been dumped on us. > > Worse, we've had people who track regressions imply that if we don't > prioritise fixing regressions ahead of anything else we might be > working on, then we might not get new work merged until the > regressions have been fixed. In my book, that's akin to extortion, > and it might give you some insight to why Darrick reacted so > vigorously to having an untriaged syzkaller bug tracked as a high > visibility, must fix regression. > > What we really need is more people who are capable to triaging bug > reports like this instead of having lots of people cranking on bot > handles and dumping untriaged bug reports on the maintainer. > Further, if you aren't capable of triaging the bug report, then you > aren't qualified to classify it as a "must fix" regression. > > It's like people don't have any common sense or decency anymore: > it's not very nice to classify a bug as a "must fix" regression > without first having consulted the engineers responsible for that > code. If you don't know what the cause of the bug is, then don't > crank handles that cause people to have to address it immediately! > > If nothing changes, then the ever increasing amount of bot cranking > is going to burn us out completely. Nobody wins when that > happens.... > I did not do well in two points, which led to the problem of this useless bisect info: 1. Should double check "V4 Filesystem" related issue carefully, and should give reason of problem. 2. Double check the bisect bad and good dmesg info, this time actually "good(actually not good)" dmesg also contains "BUG" related dmesg, but it doesn't contain the keyword "xfs_extent_free_diff_items" dmesg info, and give the wrong bisect info. Sorry for inconvenience... For above 2 points, I will solve the above two problems thoroughly a.s.a.p. I won't report useless bisection information caused by the above 2 points. Thanks! BR. > -Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 9:14 ` Pengfei Xu @ 2023-05-23 21:52 ` Dave Chinner 2023-05-24 2:20 ` Pengfei Xu 0 siblings, 1 reply; 14+ messages in thread From: Dave Chinner @ 2023-05-23 21:52 UTC (permalink / raw) To: Pengfei Xu Cc: Eric Biggers, Darrick J. Wong, Bagas Sanjaya, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Tue, May 23, 2023 at 05:14:24PM +0800, Pengfei Xu wrote: > I did not do well in two points, which led to the problem of this useless > bisect info: > 1. Should double check "V4 Filesystem" related issue carefully, and should > give reason of problem. > 2. Double check the bisect bad and good dmesg info, this time actually > "good(actually not good)" dmesg also contains "BUG" related > dmesg, but it doesn't contain the keyword "xfs_extent_free_diff_items" > dmesg info, and give the wrong bisect info. > Sorry for inconvenience... I think you misunderstand. The bisect you did was correct - the commit it identified was certainly does expose the underlying issue. The reason the bisect, while correct, is actually useless is that it the underlying issue that the commit tripped over is not caused by the change in the commit. The underlying issue has been there for a long while - probably a decade - and it's that old, underlying issue that has caused the new code to fail. IOWs, the problem is not the new code (i.e. it is not a regression in the new code identified by the bisect), the problem is in other code that has been silently propagating undetected corruption for years. Hence the bisect is not actually useful in diagnosing the root cause of the problem. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 21:52 ` Dave Chinner @ 2023-05-24 2:20 ` Pengfei Xu 0 siblings, 0 replies; 14+ messages in thread From: Pengfei Xu @ 2023-05-24 2:20 UTC (permalink / raw) To: Dave Chinner Cc: Eric Biggers, Darrick J. Wong, Bagas Sanjaya, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On 2023-05-24 at 07:52:59 +1000, Dave Chinner wrote: > On Tue, May 23, 2023 at 05:14:24PM +0800, Pengfei Xu wrote: > > I did not do well in two points, which led to the problem of this useless > > bisect info: > > 1. Should double check "V4 Filesystem" related issue carefully, and should > > give reason of problem. > > 2. Double check the bisect bad and good dmesg info, this time actually > > "good(actually not good)" dmesg also contains "BUG" related > > dmesg, but it doesn't contain the keyword "xfs_extent_free_diff_items" > > dmesg info, and give the wrong bisect info. > > Sorry for inconvenience... > > I think you misunderstand. > > The bisect you did was correct - the commit it > identified was certainly does expose the underlying issue. > > The reason the bisect, while correct, is actually useless is that it > the underlying issue that the commit tripped over is not caused by > the change in the commit. The underlying issue has been there for a > long while - probably a decade - and it's that old, underlying issue > that has caused the new code to fail. > > IOWs, the problem is not the new code (i.e. it is not a regression > in the new code identified by the bisect), the problem is in other > code that has been silently propagating undetected corruption for > years. Hence the bisect is not actually useful in diagnosing the > root cause of the problem. > Thanks a lot Dave's description! It's clear. Anyway I will remove "CONFIG_XFS_SUPPORT_V4" in syzkaller fuzzing test next time to avoid the noise. Thanks also to Eric Biggers, Bagas Sanjaya and all community's help! Thanks! BR. > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 7:31 ` Dave Chinner 2023-05-23 9:14 ` Pengfei Xu @ 2023-05-23 16:50 ` Eric Biggers 2023-05-23 22:16 ` Dave Chinner 1 sibling, 1 reply; 14+ messages in thread From: Eric Biggers @ 2023-05-23 16:50 UTC (permalink / raw) To: Dave Chinner Cc: Darrick J. Wong, Bagas Sanjaya, Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions Hi Dave, On Tue, May 23, 2023 at 05:31:23PM +1000, Dave Chinner wrote: > On Tue, May 23, 2023 at 12:00:29AM +0000, Eric Biggers wrote: > > On Mon, May 22, 2023 at 09:05:25AM -0700, Darrick J. Wong wrote: > > > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > > > > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > > > > Hi Darrick, > > > > > > > > > > Greeting! > > > > > There is BUG: unable to handle kernel NULL pointer dereference in > > > > > xfs_extent_free_diff_items in v6.4-rc3: > > > > > > > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > > > > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > > > > " > > > > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > > > > " > > > > > > > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > > > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > > > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > > > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > > > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > > > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > > > > > > > v6.4-rc3 reproduced info: > > > > > > Diagnosis and patches welcomed. > > > > > > Or are we doing the usual syzbot bullshit where you all assume that I'm > > > going to do all the fucking work for you? > > > > > > > It looks like Pengfei already took the time to manually bisect this issue to a > > very recent commit authored by you. Is that not helpful? > > No. The bisect is completely meaningless. > > The cause of the problem is going to be some piece of corrupted > metadata has got through a verifier check or log recovery and has > resulted in a perag lookup failing. The bisect landed on the commit > where the perag dependency was introduced; whatever is letting > unchecked corrupted metadata throught he verifiers has existed long > before this recent change was made. > > I've already spent two hours analysing this report - I've got to the > point where I've isolated the transaction in the trace, I see the > allocation being run as expected, I see all the right things > happening, and then it goes splat after the allocation has committed > and it starts processing defered extent free operations. Neither the > code nor the trace actually tell me anything about the nature of the > failure that has occurred. > > At this point, I still don't know where the corrupted metadata is > coming from. That's the next thing I need to look at, and then I > realised that this bug report *doesn't include a pointer to the > corrupted filesystem image that is being mounted*. > > IOWs, the bug report is deficient and not complete, and so I'm > forced to spend unnecessary time trying to work out how to extract > the filesystem image from a weird syzkaller report that is basically > just a bunch of undocumented blobs in a github tree. > > This is the same sort of shit we've been having to deal rigth from > teh start with syzkaller. It doesn't matter that syzbot might have > improved it's reporting a bit these days, we still have to deal with > this sort of poor reporting from all the private syzkaller bot crank > handles that are being turned by people who know little more than > how to turn a crank handle. > > To make matters worse, this is a v4 filesystem which has known > unfixable issues when handling corrupted filesystems in both log > replay and in runtime detection of corruption. We've repeatedly told > people running syzkaller (including Pengfei) to stop running it on > v4 filesystems and only report bugs on V5 format filesystems. This > is to avoid wasting time triaging these problems back down to v4 > specific format bugs that ican only be fixed by moving to the v5 > format. > > ..... > > And now after 4 hours, I have found several corruptions in the on > disk format that v5 filesystems will have caught and v4 filesystems > will not. > > The AGFL indexes in the AGF have been corrupted. They are within > valid bounds, but first + last != count. On a V5 filesystem we catch > this and trigger an AGFL reset that is done of the first allocation. > v4 filesystems do not do this last - first = count validation at > all. > > Further, the AGFL has also been corrupted - it is full of null > blocks. This is another problem that V5 filesystems can catch and > report, but v4 filesystems don't because they don't have headers in > the AGFL that enable verification. > > Yes, there's definitely scope for further improvements in validation > here, but the unhandled corruptions that I've found still don't > explain how we got a null perag in the xefi created from a > referenced perag that is causing the crash. > > So, yeah, the bisect is completely useless, and I've got half a day > into triage and I still don't have any clue what the corruption is > that is causing the kernel to crash.... > > ---- > > Do you see the problem now, Eric? > > Performing root-cause analysis of syzkaller based malicious > filesystem corruption bugs is anything but simple. It takes hours to > days just to work through triage of a single bug report, and we're > getting a couple of these sorts of bug reported every week. > > People who do nothing but turn the bot crank handle throw stuff like > this over the wall at usi are easy to find. Bots and bot crank > turners scale really easily. Engineers who can find and fix the > problems, OTOH, don't. > > And just to rub salt into the wounds, we now have people who turn > crank handles on other bots to tell everyone else how important > they think the problem is without having performed any triage at > all. And then we're expected to make an untriaged bug report our > highest priority and immediately spend hours of time to make sense > of the steaming pile that has just been dumped on us. > > Worse, we've had people who track regressions imply that if we don't > prioritise fixing regressions ahead of anything else we might be > working on, then we might not get new work merged until the > regressions have been fixed. In my book, that's akin to extortion, > and it might give you some insight to why Darrick reacted so > vigorously to having an untriaged syzkaller bug tracked as a high > visibility, must fix regression. > > What we really need is more people who are capable to triaging bug > reports like this instead of having lots of people cranking on bot > handles and dumping untriaged bug reports on the maintainer. > Further, if you aren't capable of triaging the bug report, then you > aren't qualified to classify it as a "must fix" regression. > > It's like people don't have any common sense or decency anymore: > it's not very nice to classify a bug as a "must fix" regression > without first having consulted the engineers responsible for that > code. If you don't know what the cause of the bug is, then don't > crank handles that cause people to have to address it immediately! > > If nothing changes, then the ever increasing amount of bot cranking > is going to burn us out completely. Nobody wins when that > happens.... > Thanks for the explanation. I personally didn't need such a long explanation, but it should be helpful for Pengfei and others. I was mostly just concerned that a report of a bug with a reproducer and a bisection to a recent commit just got a response from the maintainer with profanities and no helpful information. ("It's like people don't have any common sense or decency anymore...") I think that makes it hard for people like Pengfei to understand what they did wrong, especially when they might have gotten very different responses from other kernel subsystems. So, thank you for providing a more detailed explanation, though honestly, something much shorter would have sufficed. (Maybe you should even have a write-up on the XFS wiki or in Documentation/ that you point to whenever this sort of thing comes up?) BTW, given that XFS has a policy of not fixing bugs in XFS v4 filesystems, I suggest adding a kconfig option that disables the support for mounting XFS v4 filesystems. Then you could just tell the people fuzzing XFS filesystem images that they need to use that option. That would save everyone a lot of time. (To be clear, I'm not arguing for the XFS policy on v4 filesystems being right or wrong; that's really not something I'd like to get into again... I'm just saying that if that's indeed your policy, this is what you should do.) - Eric ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 16:50 ` Eric Biggers @ 2023-05-23 22:16 ` Dave Chinner 2023-05-23 23:46 ` Eric Biggers 0 siblings, 1 reply; 14+ messages in thread From: Dave Chinner @ 2023-05-23 22:16 UTC (permalink / raw) To: Eric Biggers Cc: Darrick J. Wong, Bagas Sanjaya, Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Tue, May 23, 2023 at 04:50:44PM +0000, Eric Biggers wrote: > On Tue, May 23, 2023 at 05:31:23PM +1000, Dave Chinner wrote: > > On Tue, May 23, 2023 at 12:00:29AM +0000, Eric Biggers wrote: > > > On Mon, May 22, 2023 at 09:05:25AM -0700, Darrick J. Wong wrote: > > > > On Mon, May 22, 2023 at 01:39:27PM +0700, Bagas Sanjaya wrote: > > > > > On Mon, May 22, 2023 at 10:07:28AM +0800, Pengfei Xu wrote: > > > > > > Hi Darrick, > > > > > > > > > > > > Greeting! > > > > > > There is BUG: unable to handle kernel NULL pointer dereference in > > > > > > xfs_extent_free_diff_items in v6.4-rc3: > > > > > > > > > > > > Above issue could be reproduced in v6.4-rc3 and v6.4-rc2 kernel in guest. > > > > > > > > > > > > Bisected this issue between v6.4-rc2 and v5.11, found the problem commit is: > > > > > > " > > > > > > f6b384631e1e xfs: give xfs_extfree_intent its own perag reference > > > > > > " > > > > > > > > > > > > report0, repro.stat and so on detailed info is link: https://github.com/xupengfe/syzkaller_logs/tree/main/230521_043336_xfs_extent_free_diff_items > > > > > > Syzkaller reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.c > > > > > > Syzkaller reproduced prog: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/repro.prog > > > > > > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/kconfig_origin > > > > > > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/bisect_info.log > > > > > > Issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230521_043336_xfs_extent_free_diff_items/v6.4-rc3_reproduce_dmesg.log > > > > > > > > > > > > v6.4-rc3 reproduced info: > > > > > > > > Diagnosis and patches welcomed. > > > > > > > > Or are we doing the usual syzbot bullshit where you all assume that I'm > > > > going to do all the fucking work for you? > > > > > > > > > > It looks like Pengfei already took the time to manually bisect this issue to a > > > very recent commit authored by you. Is that not helpful? > > > > No. The bisect is completely meaningless. > > > > The cause of the problem is going to be some piece of corrupted > > metadata has got through a verifier check or log recovery and has > > resulted in a perag lookup failing. The bisect landed on the commit > > where the perag dependency was introduced; whatever is letting > > unchecked corrupted metadata throught he verifiers has existed long > > before this recent change was made. > > > > I've already spent two hours analysing this report - I've got to the > > point where I've isolated the transaction in the trace, I see the > > allocation being run as expected, I see all the right things > > happening, and then it goes splat after the allocation has committed > > and it starts processing defered extent free operations. Neither the > > code nor the trace actually tell me anything about the nature of the > > failure that has occurred. > > > > At this point, I still don't know where the corrupted metadata is > > coming from. That's the next thing I need to look at, and then I > > realised that this bug report *doesn't include a pointer to the > > corrupted filesystem image that is being mounted*. > > > > IOWs, the bug report is deficient and not complete, and so I'm > > forced to spend unnecessary time trying to work out how to extract > > the filesystem image from a weird syzkaller report that is basically > > just a bunch of undocumented blobs in a github tree. > > > > This is the same sort of shit we've been having to deal rigth from > > teh start with syzkaller. It doesn't matter that syzbot might have > > improved it's reporting a bit these days, we still have to deal with > > this sort of poor reporting from all the private syzkaller bot crank > > handles that are being turned by people who know little more than > > how to turn a crank handle. > > > > To make matters worse, this is a v4 filesystem which has known > > unfixable issues when handling corrupted filesystems in both log > > replay and in runtime detection of corruption. We've repeatedly told > > people running syzkaller (including Pengfei) to stop running it on > > v4 filesystems and only report bugs on V5 format filesystems. This > > is to avoid wasting time triaging these problems back down to v4 > > specific format bugs that ican only be fixed by moving to the v5 > > format. > > > > ..... > > > > And now after 4 hours, I have found several corruptions in the on > > disk format that v5 filesystems will have caught and v4 filesystems > > will not. > > > > The AGFL indexes in the AGF have been corrupted. They are within > > valid bounds, but first + last != count. On a V5 filesystem we catch > > this and trigger an AGFL reset that is done of the first allocation. > > v4 filesystems do not do this last - first = count validation at > > all. > > > > Further, the AGFL has also been corrupted - it is full of null > > blocks. This is another problem that V5 filesystems can catch and > > report, but v4 filesystems don't because they don't have headers in > > the AGFL that enable verification. > > > > Yes, there's definitely scope for further improvements in validation > > here, but the unhandled corruptions that I've found still don't > > explain how we got a null perag in the xefi created from a > > referenced perag that is causing the crash. > > > > So, yeah, the bisect is completely useless, and I've got half a day > > into triage and I still don't have any clue what the corruption is > > that is causing the kernel to crash.... > > > > ---- > > > > Do you see the problem now, Eric? > > > > Performing root-cause analysis of syzkaller based malicious > > filesystem corruption bugs is anything but simple. It takes hours to > > days just to work through triage of a single bug report, and we're > > getting a couple of these sorts of bug reported every week. > > > > People who do nothing but turn the bot crank handle throw stuff like > > this over the wall at usi are easy to find. Bots and bot crank > > turners scale really easily. Engineers who can find and fix the > > problems, OTOH, don't. > > > > And just to rub salt into the wounds, we now have people who turn > > crank handles on other bots to tell everyone else how important > > they think the problem is without having performed any triage at > > all. And then we're expected to make an untriaged bug report our > > highest priority and immediately spend hours of time to make sense > > of the steaming pile that has just been dumped on us. > > > > Worse, we've had people who track regressions imply that if we don't > > prioritise fixing regressions ahead of anything else we might be > > working on, then we might not get new work merged until the > > regressions have been fixed. In my book, that's akin to extortion, > > and it might give you some insight to why Darrick reacted so > > vigorously to having an untriaged syzkaller bug tracked as a high > > visibility, must fix regression. > > > > What we really need is more people who are capable to triaging bug > > reports like this instead of having lots of people cranking on bot > > handles and dumping untriaged bug reports on the maintainer. > > Further, if you aren't capable of triaging the bug report, then you > > aren't qualified to classify it as a "must fix" regression. > > > > It's like people don't have any common sense or decency anymore: > > it's not very nice to classify a bug as a "must fix" regression > > without first having consulted the engineers responsible for that > > code. If you don't know what the cause of the bug is, then don't > > crank handles that cause people to have to address it immediately! > > > > If nothing changes, then the ever increasing amount of bot cranking > > is going to burn us out completely. Nobody wins when that > > happens.... > > > > Thanks for the explanation. I personally didn't need such a long explanation, > but it should be helpful for Pengfei and others. > > I was mostly just concerned that a report of a bug with a reproducer and a > bisection to a recent commit just got a response from the maintainer with > profanities and no helpful information. ("It's like people don't have any > common sense or decency anymore...") I think that makes it hard for people like > Pengfei to understand what they did wrong, especially when they might have > gotten very different responses from other kernel subsystems. So, thank you for > providing a more detailed explanation, though honestly, something much shorter > would have sufficed. (Maybe you should even have a write-up on the XFS wiki or > in Documentation/ that you point to whenever this sort of thing comes up?) > > BTW, given that XFS has a policy of not fixing bugs in XFS v4 filesystems, I No, I *did not say that*. That is most definitely not our policy. I said that *V4 filesystems have known bugs that can only be fixed by moving to the V5 format*. Some of the checks that V5 filesystems do can also be done on v4 filesystems. There are some missing checks for both v4 and v5 filesystems. Once I get to the root cause and understand how this all blew up, I'll fix the problems in common v4/v5 code. I might not test the v4 code that much, because.... > suggest adding a kconfig option that disables the support for mounting XFS v4 > filesystems. fs/xfs/Kconfig: config XFS_SUPPORT_V4 bool "Support deprecated V4 (crc=0) format" depends on XFS_FS default y help The V4 filesystem format lacks certain features that are supported by the V5 format, such as metadata checksumming, strengthened metadata verification, and the ability to store timestamps past the year 2038. Because of this, the V4 format is deprecated. All users should upgrade by backing up their files, reformatting, and restoring from the backup. Administrators and users can detect a V4 filesystem by running xfs_info against a filesystem mountpoint and checking for a string beginning with "crc=". If the string "crc=0" is found, the filesystem is a V4 filesystem. If no such string is found, please upgrade xfsprogs to the latest version and try again. This option will become default N in September 2025. Support for the V4 format will be removed entirely in September 2030. Distributors can say N here to withdraw support earlier. To continue supporting the old V4 format (crc=0), say Y. To close off an attack surface, say N. This was added almost 3 years ago in mid-2020. We're more than half way through the deprecation period and then we're going to turn off v4 support by default. At this point, nobody should be using v4 filesystems in new production systems, and those that are should be preparing for upstream and distro support to be withdraw in the next couple of years... > Then you could just tell the people fuzzing XFS filesystem images > that they need to use that option. That would save everyone a lot of time. > (To be clear, I'm not arguing for the XFS policy on v4 filesystems being right > or wrong; that's really not something I'd like to get into again... I'm just > saying that if that's indeed your policy, this is what you should do.) It should be obvious by now that we've already done this. 3 years ago, in fact. And yet we are still having the same problems. Maybe this helps you understand the level of frustration we have with all the people running fuzzing bots out there.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 2023-05-23 22:16 ` Dave Chinner @ 2023-05-23 23:46 ` Eric Biggers 0 siblings, 0 replies; 14+ messages in thread From: Eric Biggers @ 2023-05-23 23:46 UTC (permalink / raw) To: Dave Chinner Cc: Darrick J. Wong, Bagas Sanjaya, Pengfei Xu, linux-xfs, linux-fsdevel, heng.su, dchinner, lkp, Linux Regressions On Wed, May 24, 2023 at 08:16:58AM +1000, Dave Chinner wrote: > config XFS_SUPPORT_V4 > bool "Support deprecated V4 (crc=0) format" > depends on XFS_FS > default y > help > The V4 filesystem format lacks certain features that are supported > by the V5 format, such as metadata checksumming, strengthened > metadata verification, and the ability to store timestamps past the > year 2038. Because of this, the V4 format is deprecated. All users > should upgrade by backing up their files, reformatting, and restoring > from the backup. > > Administrators and users can detect a V4 filesystem by running > xfs_info against a filesystem mountpoint and checking for a string > beginning with "crc=". If the string "crc=0" is found, the > filesystem is a V4 filesystem. If no such string is found, please > upgrade xfsprogs to the latest version and try again. > > This option will become default N in September 2025. Support for the > V4 format will be removed entirely in September 2030. Distributors > can say N here to withdraw support earlier. > > To continue supporting the old V4 format (crc=0), say Y. > To close off an attack surface, say N. > > This was added almost 3 years ago in mid-2020. We're more than half > way through the deprecation period and then we're going to turn off > v4 support by default. At this point, nobody should be using v4 > filesystems in new production systems, and those that are should be > preparing for upstream and distro support to be withdraw in the next > couple of years... Great to see that this exists now and there is a specific deprecation plan! > > Then you could just tell the people fuzzing XFS filesystem images > > that they need to use that option. That would save everyone a lot of time. > > (To be clear, I'm not arguing for the XFS policy on v4 filesystems being right > > or wrong; that's really not something I'd like to get into again... I'm just > > saying that if that's indeed your policy, this is what you should do.) > > It should be obvious by now that we've already done this. 3 years > ago, in fact. And yet we are still having the same problems. Maybe > this helps you understand the level of frustration we have with all > the people running fuzzing bots out there.... I don't see evidence that this actually happened, though perhaps we are not looking in the same places. https://lore.kernel.org/all/?q=XFS_SUPPORT_V4 brings up little except the original patch thread, nor did https://github.com/search?q=XFS_SUPPORT_V4&type=issues find anything. Anyway, I took 3 minutes to file an issue in the syzkaller repo (https://github.com/google/syzkaller/issues/3918), so at least this should get resolved for syzbot soon. - Eric ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-05-24 2:18 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-05-22 2:07 [Syzkaller & bisect] There is BUG: unable to handle kernel NULL pointer dereference in xfs_extent_free_diff_items in v6.4-rc3 Pengfei Xu 2023-05-22 6:39 ` Bagas Sanjaya 2023-05-22 16:05 ` Darrick J. Wong 2023-05-22 17:05 ` Linux regression tracking (Thorsten Leemhuis) 2023-05-23 6:08 ` Bagas Sanjaya 2023-05-23 6:44 ` Pengfei Xu 2023-05-23 0:00 ` Eric Biggers 2023-05-23 7:31 ` Dave Chinner 2023-05-23 9:14 ` Pengfei Xu 2023-05-23 21:52 ` Dave Chinner 2023-05-24 2:20 ` Pengfei Xu 2023-05-23 16:50 ` Eric Biggers 2023-05-23 22:16 ` Dave Chinner 2023-05-23 23:46 ` Eric Biggers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).