* Pending bugs for 5.7
@ 2020-05-12 14:12 David Sterba
2020-05-12 14:14 ` Bug 5.7-rc: root leak, eb leak David Sterba
` (3 more replies)
0 siblings, 4 replies; 24+ messages in thread
From: David Sterba @ 2020-05-12 14:12 UTC (permalink / raw)
To: linux-btrfs
Hi,
there are some pending bugs that are either regressions or exposed after
changes in the 5.7 development cycle. At least the root leaks need to be fixed,
so patches with features are lower priority in case you wonder why yours hasn't
been merged or reviewed.
I'll post each as reply to this mail as they contain dumps of logs. The short
summary:
* root leaks
Recent updates switched from SRCU to refcounting and adde a tree root leak
detector, that reports leak in tests btrfs/125.
* corrupted leaf
For this one it's not clear if it's a regression or an old bug, reported by test
generic/457
* memory leaks
With KMEMLEAK turned on, there are several reports that there are leaks. It's
not clear if it's related to the root leaks.
* lockdep warnings
reproducible reports, our locks vs fs_reclaim, or just our locks
^ permalink raw reply [flat|nested] 24+ messages in thread* Bug 5.7-rc: root leak, eb leak 2020-05-12 14:12 Pending bugs for 5.7 David Sterba @ 2020-05-12 14:14 ` David Sterba 2020-05-12 23:03 ` David Sterba 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba ` (2 subsequent siblings) 3 siblings, 1 reply; 24+ messages in thread From: David Sterba @ 2020-05-12 14:14 UTC (permalink / raw) To: linux-btrfs Root an eb leak. Johannes bisected the problem, the first bad commit is one of 0e996e7fcf2e3a "btrfs: move ino_cache_inode dropping out of btrfs_free_fs_root" 3fd6372758d91d "btrfs: make the extent buffer leak check per fs info" 8c38938c7bb096 "btrfs: move the root freeing stuff into btrfs_put_root" Reproduced on btrfs/028, I was able to reproduce it once on btrfs/125 (log below). This is most likely a regression. btrfs/125 [19:31:06][ 7063.961647] run fstests btrfs/125 at 2020-02-21 19:31:06 [ 7064.376156] BTRFS info (device vda): disk space caching is enabled [ 7064.379500] BTRFS info (device vda): has skinny extents [ 7067.890863] BTRFS: device fsid f88a762d-a155-40a2-b259-1a9331583726 devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (28576) [ 7067.895033] BTRFS: device fsid f88a762d-a155-40a2-b259-1a9331583726 devid 2 transid 5 /dev/vdc scanned by mkfs.btrfs (28576) [ 7067.899388] BTRFS: device fsid f88a762d-a155-40a2-b259-1a9331583726 devid 3 transid 5 /dev/vdd scanned by mkfs.btrfs (28576) [ 7067.919633] BTRFS info (device vdb): disk space caching is enabled [ 7067.922349] BTRFS info (device vdb): has skinny extents [ 7067.924430] BTRFS info (device vdb): flagging fs with big metadata feature [ 7067.931360] BTRFS info (device vdb): checking UUID tree [ 7068.117606] BTRFS: device fsid f88a762d-a155-40a2-b259-1a9331583726 devid 2 transid 7 /dev/vdc scanned by mount (28618) [ 7068.121547] BTRFS: device fsid f88a762d-a155-40a2-b259-1a9331583726 devid 1 transid 7 /dev/vdb scanned by mount (28618) [ 7068.125450] BTRFS info (device vdb): allowing degraded mounts [ 7068.127457] BTRFS info (device vdb): disk space caching is enabled [ 7068.129299] BTRFS info (device vdb): has skinny extents [ 7068.132805] BTRFS warning (device vdb): devid 3 uuid 9049eb45-e68b-41d4-9fa1-e1b0e08fa552 is missing [ 7068.135869] BTRFS warning (device vdb): devid 3 uuid 9049eb45-e68b-41d4-9fa1-e1b0e08fa552 is missing [ 7070.095641] BTRFS: device fsid cf4bb9b1-6c5c-46d2-8fd6-b9f427137ade devid 1 transid 250 /dev/vda scanned by btrfs (28646) [ 7070.118262] BTRFS info (device vdb): disk space caching is enabled [ 7070.120662] BTRFS info (device vdb): has skinny extents [ 7080.724838] BTRFS info (device vdb): balance: start -d -m -s [ 7080.733528] BTRFS info (device vdb): relocating block group 217710592 flags data|raid5 [ 7081.001862] btrfs_print_data_csum_error: 16829 callbacks suppressed [ 7081.001870] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2228224 csum 0xa9a2deb9 expected csum 0x8941f998 mirror 1 [ 7081.013321] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2236416 csum 0x85a94242 expected csum 0x8941f998 mirror 1 [ 7081.018428] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2240512 csum 0x95ef75cb expected csum 0x8941f998 mirror 1 [ 7081.018843] repair_io_failure: 17181 callbacks suppressed [ 7081.018847] BTRFS info (device vdb): read error corrected: ino 257 off 2236416 (dev /dev/vdd sector 195344) [ 7081.022838] BTRFS info (device vdb): read error corrected: ino 257 off 2228224 (dev /dev/vdd sector 195328) [ 7081.022915] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2244608 csum 0x4f96cecf expected csum 0x8941f998 mirror 1 [ 7081.023009] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2248704 csum 0xa8fc61d5 expected csum 0x8941f998 mirror 1 [ 7081.023090] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2252800 csum 0xedb728d1 expected csum 0x8941f998 mirror 1 [ 7081.023134] BTRFS info (device vdb): read error corrected: ino 257 off 2240512 (dev /dev/vdd sector 195352) [ 7081.023168] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2256896 csum 0xeb11484f expected csum 0x8941f998 mirror 1 [ 7081.023248] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2260992 csum 0xc31d9456 expected csum 0x8941f998 mirror 1 [ 7081.023324] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2265088 csum 0xff9caf51 expected csum 0x8941f998 mirror 1 [ 7081.023328] BTRFS info (device vdb): read error corrected: ino 257 off 2244608 (dev /dev/vdd sector 195360) [ 7081.023399] BTRFS warning (device vdb): csum failed root -9 ino 257 off 2269184 csum 0x866f9727 expected csum 0x8941f998 mirror 1 [ 7081.023497] BTRFS info (device vdb): read error corrected: ino 257 off 2248704 (dev /dev/vdd sector 195368) [ 7081.023680] BTRFS info (device vdb): read error corrected: ino 257 off 2252800 (dev /dev/vdd sector 195376) [ 7081.023846] BTRFS info (device vdb): read error corrected: ino 257 off 2256896 (dev /dev/vdd sector 195384) [ 7081.024009] BTRFS info (device vdb): read error corrected: ino 257 off 2265088 (dev /dev/vdd sector 195400) [ 7081.024012] BTRFS info (device vdb): read error corrected: ino 257 off 2260992 (dev /dev/vdd sector 195392) [ 7081.024217] BTRFS info (device vdb): read error corrected: ino 257 off 2269184 (dev /dev/vdd sector 195408) [ 7082.760255] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.760267] BTRFS error (device vdb): bad tree block start, want 39075840 have 30556160 [ 7082.764139] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.770746] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.770769] BTRFS error (device vdb): bad tree block start, want 39075840 have 30556160 [ 7082.773961] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.779340] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.779357] BTRFS error (device vdb): bad tree block start, want 39075840 have 30556160 [ 7082.782367] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7082.788031] BTRFS error (device vdb): bad tree block start, want 39059456 have 30539776 [ 7084.736674] BTRFS info (device vdb): balance: ended with status: -5 [failed, exit status 1][ 7084.791281] BTRFS info (device vdb): at unmount delalloc count 1994752 [ 7084.891149] ------------[ cut here ]------------ [ 7084.893431] WARNING: CPU: 0 PID: 28679 at fs/btrfs/block-group.c:3367 btrfs_free_block_groups+0x225/0x2b0 [btrfs] [ 7084.898426] Modules linked in: dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [ 7084.904875] CPU: 0 PID: 28679 Comm: umount Not tainted 5.6.0-rc2-default+ #1003 [ 7084.908207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 7084.912188] RIP: 0010:btrfs_free_block_groups+0x225/0x2b0 [btrfs] [ 7084.914049] Code: 49 be 22 01 00 00 00 00 ad de e8 66 b5 21 cc e8 e1 3d ba cb 48 89 df e8 f9 7e ff ff 48 8b 83 78 12 00 00 49 39 c5 75 4a eb 76 <0f> 0b 31 c9 31 d2 4c 89 e6 48 89 df e8 aa 6c ff ff 48 89 ef e8 b2 [ 7084.918227] RSP: 0018:ffffba52492ffdd8 EFLAGS: 00010206 [ 7084.919331] RAX: ffff9a44ecdbd908 RBX: ffff9a44e6a2c000 RCX: 0000000000000001 [ 7084.920735] RDX: 0000000000000000 RSI: ffffffffc056ae01 RDI: ffff9a44ecdbcc00 [ 7084.922886] RBP: ffff9a44ecdbd908 R08: 0000000000000000 R09: 0000000000000000 [ 7084.925048] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9a44ecdbd800 [ 7084.927217] R13: ffff9a44e6a2d278 R14: dead000000000122 R15: dead000000000100 [ 7084.929430] FS: 00007f1a27c27800(0000) GS:ffff9a44fe600000(0000) knlGS:0000000000000000 [ 7084.932270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7084.934179] CR2: 00007ffdda6eac48 CR3: 0000000018df5001 CR4: 0000000000160ef0 [ 7084.935958] Call Trace: [ 7084.936818] close_ctree+0x24b/0x2a0 [btrfs] [ 7084.948688] generic_shutdown_super+0x69/0x100 [ 7084.950163] kill_anon_super+0x14/0x30 [ 7084.951525] btrfs_kill_super+0x12/0x20 [btrfs] [ 7084.953001] deactivate_locked_super+0x2c/0x70 [ 7084.954495] cleanup_mnt+0x100/0x160 [ 7084.955810] task_work_run+0x90/0xc0 [ 7084.957364] exit_to_usermode_loop+0x96/0xa0 [ 7084.958890] do_syscall_64+0x1df/0x210 [ 7084.960255] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 7084.961756] RIP: 0033:0x7f1a27e6a3f7 [ 7084.962903] Code: fa 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 fa 0b 00 f7 d8 64 89 01 48 [ 7084.967021] RSP: 002b:00007ffdda6ebc58 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 7084.968916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f1a27e6a3f7 [ 7084.971111] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005588b931cb70 [ 7084.973181] RBP: 00005588b931c960 R08: 0000000000000000 R09: 00007ffdda6ea9d0 [ 7084.975183] R10: 00005588b931cb90 R11: 0000000000000246 R12: 00005588b931cb70 [ 7084.976700] R13: 0000000000000000 R14: 00005588b931ca58 R15: 0000000000000000 [ 7084.978291] irq event stamp: 0 [ 7084.979126] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 7084.980404] hardirqs last disabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7084.983042] softirqs last enabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7084.985822] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 7084.987748] ---[ end trace 4ff66fac35aae386 ]--- [ 7084.989590] BTRFS info (device vdb): space_info 1 has 3983872000 free, is not full [ 7084.992242] BTRFS info (device vdb): space_info total=4294967296, used=212926464, pinned=0, reserved=0, may_use=98168832, readonly=0 [ 7084.995211] BTRFS info (device vdb): global_block_rsv: size 0 reserved 0 [ 7084.996814] BTRFS info (device vdb): trans_block_rsv: size 0 reserved 0 [ 7084.998581] BTRFS info (device vdb): chunk_block_rsv: size 0 reserved 0 [ 7084.999979] BTRFS info (device vdb): delayed_block_rsv: size 0 reserved 0 [ 7085.001941] BTRFS info (device vdb): delayed_refs_rsv: size 0 reserved 0 [ 7085.003886] ------------[ cut here ]------------ [ 7085.005362] WARNING: CPU: 0 PID: 28679 at fs/btrfs/block-group.c:3367 btrfs_free_block_groups+0x225/0x2b0 [btrfs] [ 7085.007985] Modules linked in: dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [ 7085.011615] CPU: 0 PID: 28679 Comm: umount Tainted: G W 5.6.0-rc2-default+ #1003 [ 7085.014036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 7085.017209] RIP: 0010:btrfs_free_block_groups+0x225/0x2b0 [btrfs] [ 7085.018961] Code: 49 be 22 01 00 00 00 00 ad de e8 66 b5 21 cc e8 e1 3d ba cb 48 89 df e8 f9 7e ff ff 48 8b 83 78 12 00 00 49 39 c5 75 4a eb 76 <0f> 0b 31 c9 31 d2 4c 89 e6 48 89 df e8 aa 6c ff ff 48 89 ef e8 b2 [ 7085.024282] RSP: 0018:ffffba52492ffdd8 EFLAGS: 00010206 [ 7085.025741] RAX: ffff9a44ecdbcd08 RBX: ffff9a44e6a2c000 RCX: 0000000000170009 [ 7085.027170] RDX: 0000000000000000 RSI: 0000000000170009 RDI: ffffffff8c254b8e [ 7085.028565] RBP: ffff9a44ecdbcd08 R08: 0000000000000001 R09: 0000000000000000 [ 7085.030727] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9a44ecdbcc00 [ 7085.032258] R13: ffff9a44e6a2d278 R14: dead000000000122 R15: dead000000000100 [ 7085.033991] FS: 00007f1a27c27800(0000) GS:ffff9a44fe600000(0000) knlGS:0000000000000000 [ 7085.035879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7085.037420] CR2: 00007ffdda6eac48 CR3: 0000000018df5001 CR4: 0000000000160ef0 [ 7085.039250] Call Trace: [ 7085.040374] close_ctree+0x24b/0x2a0 [btrfs] [ 7085.041745] generic_shutdown_super+0x69/0x100 [ 7085.042982] kill_anon_super+0x14/0x30 [ 7085.043963] btrfs_kill_super+0x12/0x20 [btrfs] [ 7085.045184] deactivate_locked_super+0x2c/0x70 [ 7085.046319] cleanup_mnt+0x100/0x160 [ 7085.047229] task_work_run+0x90/0xc0 [ 7085.048130] exit_to_usermode_loop+0x96/0xa0 [ 7085.049689] do_syscall_64+0x1df/0x210 [ 7085.050849] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 7085.052025] RIP: 0033:0x7f1a27e6a3f7 [ 7085.053230] Code: fa 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 fa 0b 00 f7 d8 64 89 01 48 [ 7085.057154] RSP: 002b:00007ffdda6ebc58 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 7085.059659] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f1a27e6a3f7 [ 7085.061346] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005588b931cb70 [ 7085.062762] RBP: 00005588b931c960 R08: 0000000000000000 R09: 00007ffdda6ea9d0 [ 7085.064108] R10: 00005588b931cb90 R11: 0000000000000246 R12: 00005588b931cb70 [ 7085.066292] R13: 0000000000000000 R14: 00005588b931ca58 R15: 0000000000000000 [ 7085.068449] irq event stamp: 0 [ 7085.069512] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 7085.071228] hardirqs last disabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7085.073611] softirqs last enabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7085.076362] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 7085.077966] ---[ end trace 4ff66fac35aae387 ]--- [ 7085.079100] BTRFS info (device vdb): space_info 4 has 176652288 free, is not full [ 7085.081031] BTRFS info (device vdb): space_info total=178913280, used=294912, pinned=0, reserved=0, may_use=1703936, readonly=262144 [ 7085.083715] BTRFS info (device vdb): global_block_rsv: size 0 reserved 0 [ 7085.085270] BTRFS info (device vdb): trans_block_rsv: size 0 reserved 0 [ 7085.087169] BTRFS info (device vdb): chunk_block_rsv: size 0 reserved 0 [ 7085.088634] BTRFS info (device vdb): delayed_block_rsv: size 0 reserved 0 [ 7085.090714] BTRFS info (device vdb): delayed_refs_rsv: size 0 reserved 0 [ 7085.092457] BTRFS warning (device vdb): page private not zero on page 39485440 [ 7085.094946] BTRFS warning (device vdb): page private not zero on page 39489536 [ 7085.097735] BTRFS warning (device vdb): page private not zero on page 39493632 [ 7085.100235] BTRFS warning (device vdb): page private not zero on page 39497728 [ 7085.102806] VFS: Busy inodes after unmount of vdb. Self-destruct in 5 seconds. Have a nice day... [ 7085.109816] BTRFS error (device vdb): leaked root 18446744073709551607-0 refcount 1 [ 7085.112211] ------------[ cut here ]------------ [ 7085.113493] WARNING: CPU: 0 PID: 28679 at fs/btrfs/disk-io.c:2040 btrfs_put_root+0x11a/0x130 [btrfs] [ 7085.115537] Modules linked in: dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [ 7085.119049] CPU: 0 PID: 28679 Comm: umount Tainted: G W 5.6.0-rc2-default+ #1003 [ 7085.121446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 7085.123866] RIP: 0010:btrfs_put_root+0x11a/0x130 [btrfs] [ 7085.125532] Code: e8 fb a0 2c cc 48 89 ef 5b 5d 41 5c e9 bf fd d8 cb 5b be 03 00 00 00 5d 41 5c e9 d1 1d f7 cb c3 e8 cb be da cb e9 3f ff ff ff <0f> 0b e9 2a ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 [ 7085.130769] RSP: 0018:ffffba52492ffe38 EFLAGS: 00010282 [ 7085.132146] RAX: ffff9a44d8ee7720 RBX: ffff9a44e6a2c000 RCX: 0000000000000001 [ 7085.134361] RDX: 0000000000000000 RSI: ffffffff8c1015d9 RDI: ffff9a44f9f5c828 [ 7085.136552] RBP: ffff9a44f9f5c000 R08: 0000000000000000 R09: 0000000000000000 [ 7085.138720] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9a44f9f5cb70 [ 7085.140819] R13: ffff9a44f9f5c000 R14: 0000000000000000 R15: ffff9a44df0ca388 [ 7085.142991] FS: 00007f1a27c27800(0000) GS:ffff9a44fe600000(0000) knlGS:0000000000000000 [ 7085.145164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7085.146560] CR2: 00007ffdda6eac48 CR3: 0000000018df5001 CR4: 0000000000160ef0 [ 7085.147920] Call Trace: [ 7085.148780] btrfs_check_leaked_roots+0x88/0xa0 [btrfs] [ 7085.150186] btrfs_free_fs_info+0xcf/0x100 [btrfs] [ 7085.151319] deactivate_locked_super+0x2c/0x70 [ 7085.152361] cleanup_mnt+0x100/0x160 [ 7085.153533] task_work_run+0x90/0xc0 [ 7085.154532] exit_to_usermode_loop+0x96/0xa0 [ 7085.155547] do_syscall_64+0x1df/0x210 [ 7085.156513] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 7085.157868] RIP: 0033:0x7f1a27e6a3f7 [ 7085.158883] Code: fa 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 69 fa 0b 00 f7 d8 64 89 01 48 [ 7085.162740] RSP: 002b:00007ffdda6ebc58 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 7085.164658] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f1a27e6a3f7 [ 7085.166939] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005588b931cb70 [ 7085.169078] RBP: 00005588b931c960 R08: 0000000000000000 R09: 00007ffdda6ea9d0 [ 7085.171238] R10: 00005588b931cb90 R11: 0000000000000246 R12: 00005588b931cb70 [ 7085.173154] R13: 0000000000000000 R14: 00005588b931ca58 R15: 0000000000000000 [ 7085.175417] irq event stamp: 0 [ 7085.176768] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 7085.178525] hardirqs last disabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7085.180485] softirqs last enabled at (0): [<ffffffff8c07f423>] copy_process+0x653/0x1b40 [ 7085.183751] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 7085.185701] ---[ end trace 4ff66fac35aae388 ]--- [ 7085.187428] BTRFS: buffer leak start 39485440 len 16384 refs 1 bflags 529 owner 18446744073709551607 [19:31:27]- output mismatch (see /tmp/fstests/results//btrfs/125.out.bad) --- tests/btrfs/125.out 2018-04-12 16:57:00.616225550 +0000 +++ /tmp/fstests/results//btrfs/125.out.bad 2020-02-21 19:31:27.732000000 +0000 @@ -3,5 +3,5 @@ Write data with degraded mount Mount normal and balance - -Mount degraded but with other dev +failed: '/sbin/btrfs balance start /tmp/scratch' +(see /tmp/fstests/results//btrfs/125.full for details) ... (Run 'diff -u /tmp/fstests/tests/btrfs/125.out /tmp/fstests/results//btrfs/125.out.bad' to see the entire diff) ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-12 14:14 ` Bug 5.7-rc: root leak, eb leak David Sterba @ 2020-05-12 23:03 ` David Sterba 2020-05-13 11:54 ` Johannes Thumshirn 0 siblings, 1 reply; 24+ messages in thread From: David Sterba @ 2020-05-12 23:03 UTC (permalink / raw) To: David Sterba; +Cc: linux-btrfs, Johannes.Thumshirn On Tue, May 12, 2020 at 04:14:23PM +0200, David Sterba wrote: > Root an eb leak. > > Johannes bisected the problem, the first bad commit is one of > > 0e996e7fcf2e3a "btrfs: move ino_cache_inode dropping out of btrfs_free_fs_root" Unlikely as inode_cache mount option is not used neither by the tests nor by my testing setup. > 3fd6372758d91d "btrfs: make the extent buffer leak check per fs info" Unlikely as it's just moving extent buffer leak detector around. > 8c38938c7bb096 "btrfs: move the root freeing stuff into btrfs_put_root" That's most likely the cause. > Reproduced on btrfs/028, Johannes, do you have logs from the test? > [failed, exit status 1][ 7084.791281] BTRFS info (device vdb): at unmount delalloc count 1994752 delalloc leak at umount > [ 7085.109816] BTRFS error (device vdb): leaked root 18446744073709551607-0 refcount 1 18446744073709551607 == -9, which is BTRFS_DATA_RELOC_TREE_OBJECTID So some post-reloc cleanup is maybe missing, the test btrfs/028 does relocation and btrfs/125 as well. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-12 23:03 ` David Sterba @ 2020-05-13 11:54 ` Johannes Thumshirn 2020-05-13 11:57 ` Qu Wenruo 0 siblings, 1 reply; 24+ messages in thread From: Johannes Thumshirn @ 2020-05-13 11:54 UTC (permalink / raw) To: dsterba, David Sterba; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 202 bytes --] On 13/05/2020 01:04, David Sterba wrote: [...] > > Johannes, do you have logs from the test? I recreated the logs for btrfs/028 (dmesg, kmemleak and fstests log). Please find them attached. [-- Attachment #2: 028.dmesg --] [-- Type: text/plain, Size: 55401 bytes --] [ 0.000000] Linux version 5.7.0-rc5+ (johannes@redsun60) (gcc version 9.2.1 20191209 [gcc-9-branch revision 279114] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.33.1.20191023-2) #599 SMP Wed May 13 19:16:40 JST 2020 [ 0.000000] Command line: rapido.vm_num=1 ip=none rd.systemd.unit=emergency rd.shell=1 console=ttyS0 rd.lvm=0 rd.luks=0 loglevel=9 [ 0.000000] x86/fpu: x87 FPU will use FXSAVE [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2200.029 MHz processor [ 0.003025] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.003028] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.003036] last_pfn = 0x240000 max_arch_pfn = 0x400000000 [ 0.003066] MTRR default type: write-back [ 0.003068] MTRR fixed ranges enabled: [ 0.003070] 00000-9FFFF write-back [ 0.003072] A0000-BFFFF uncachable [ 0.003074] C0000-FFFFF write-protect [ 0.003075] MTRR variable ranges enabled: [ 0.003078] 0 base 000C0000000 mask 3FFC0000000 uncachable [ 0.003079] 1 disabled [ 0.003081] 2 disabled [ 0.003082] 3 disabled [ 0.003083] 4 disabled [ 0.003084] 5 disabled [ 0.003085] 6 disabled [ 0.003087] 7 disabled [ 0.003096] x86/PAT: PAT not supported by the CPU. [ 0.003106] x86/PAT: Configuration [0-7]: WB WT UC- UC WB WT UC- UC [ 0.003108] last_pfn = 0xbffe0 max_arch_pfn = 0x400000000 [ 0.013238] found SMP MP-table at [mem 0x000f5c80-0x000f5c8f] [ 0.013286] BRK [0x03801000, 0x03801fff] PGTABLE [ 0.013288] BRK [0x03802000, 0x03802fff] PGTABLE [ 0.013290] BRK [0x03803000, 0x03803fff] PGTABLE [ 0.013406] BRK [0x03804000, 0x03804fff] PGTABLE [ 0.014243] BRK [0x03805000, 0x03805fff] PGTABLE [ 0.014263] BRK [0x03806000, 0x03806fff] PGTABLE [ 0.014408] RAMDISK: [mem 0x7874c000-0x7fffffff] [ 0.014424] ACPI: Early table checksum verification disabled [ 0.014439] ACPI: RSDP 0x00000000000F5A40 000014 (v00 BOCHS ) [ 0.014448] ACPI: RSDT 0x00000000BFFE15C9 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.014461] ACPI: FACP 0x00000000BFFE149D 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.014471] ACPI: DSDT 0x00000000BFFE0040 00145D (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.014477] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.014482] ACPI: APIC 0x00000000BFFE1511 000080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.014489] ACPI: HPET 0x00000000BFFE1591 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) [ 0.014504] ACPI: Local APIC address 0xfee00000 [ 0.014556] Zone ranges: [ 0.014558] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.014560] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.014562] Normal [mem 0x0000000100000000-0x000000023fffffff] [ 0.014564] Movable zone start for each node [ 0.014565] Early memory node ranges [ 0.014567] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.014572] node 0: [mem 0x0000000000100000-0x00000000bffdffff] [ 0.014575] node 0: [mem 0x0000000100000000-0x000000023fffffff] [ 0.015385] Zeroed struct page in unavailable ranges: 130 pages [ 0.015387] Initmem setup node 0 [mem 0x0000000000001000-0x000000023fffffff] [ 0.015390] On node 0 totalpages: 2097022 [ 0.015394] DMA zone: 64 pages used for memmap [ 0.015396] DMA zone: 21 pages reserved [ 0.015398] DMA zone: 3998 pages, LIFO batch:0 [ 0.015630] DMA32 zone: 12224 pages used for memmap [ 0.015632] DMA32 zone: 782304 pages, LIFO batch:63 [ 0.065708] Normal zone: 20480 pages used for memmap [ 0.065713] Normal zone: 1310720 pages, LIFO batch:63 [ 0.417194] kasan: KernelAddressSanitizer initialized [ 0.417475] ACPI: PM-Timer IO Port: 0x608 [ 0.417479] ACPI: Local APIC address 0xfee00000 [ 0.417491] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.417532] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.417538] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.417540] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.417542] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.417550] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.417552] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.417555] ACPI: IRQ0 used by override. [ 0.417557] ACPI: IRQ5 used by override. [ 0.417558] ACPI: IRQ9 used by override. [ 0.417560] ACPI: IRQ10 used by override. [ 0.417561] ACPI: IRQ11 used by override. [ 0.417565] Using ACPI (MADT) for SMP configuration information [ 0.417568] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.417578] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.417608] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.417615] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns [ 0.422546] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1 [ 0.423166] percpu: Embedded 51 pages/cpu s171480 r8192 d29224 u1048576 [ 0.423191] pcpu-alloc: s171480 r8192 d29224 u1048576 alloc=1*2097152 [ 0.423193] pcpu-alloc: [0] 0 1 [ 0.423255] Built 1 zonelists, mobility grouping on. Total pages: 2064233 [ 0.423258] Kernel command line: rapido.vm_num=1 ip=none rd.systemd.unit=emergency rd.shell=1 console=ttyS0 rd.lvm=0 rd.luks=0 loglevel=9 [ 0.425583] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear) [ 0.426679] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 0.426762] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.597569] Memory: 6959648K/8388088K available (12291K kernel code, 2631K rwdata, 3160K rodata, 1084K init, 15416K bss, 1428440K reserved, 0K cma-reserved) [ 0.598623] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.598712] Kernel/User page tables isolation: enabled [ 0.599500] rcu: Hierarchical RCU implementation. [ 0.599502] rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2. [ 0.599506] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.599508] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 [ 0.599557] NR_IRQS: 4352, nr_irqs: 56, preallocated irqs: 16 [ 0.600206] random: get_random_bytes called from start_kernel+0x3c0/0x5b2 with crng_init=0 [ 0.610266] Console: colour VGA+ 80x25 [ 0.677556] printk: console [ttyS0] enabled [ 0.678044] ACPI: Core revision 20200326 [ 0.679169] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.680218] APIC: Switch to symmetric I/O mode setup [ 0.682357] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.700205] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x1fb64e4135c, max_idle_ns: 440795292232 ns [ 0.702602] Calibrating delay loop (skipped), value calculated using timer frequency.. 4400.05 BogoMIPS (lpj=8800116) [ 0.703647] pid_max: default: 32768 minimum: 301 [ 0.707144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.707946] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear) [ 0.711031] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.711591] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.712236] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.713101] Spectre V2 : Mitigation: Full generic retpoline [ 0.713668] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.714510] Speculative Store Bypass: Vulnerable [ 0.714598] MDS: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.715597] Freeing SMP alternatives memory: 20K [ 0.824933] smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0x6, model: 0x6, stepping: 0x3) [ 0.826588] Performance Events: PMU not available due to virtualization, using software events only. [ 0.835331] rcu: Hierarchical SRCU implementation. [ 0.841798] NMI watchdog: Perf NMI watchdog permanently disabled [ 0.843067] smp: Bringing up secondary CPUs ... [ 0.844811] x86: Booting SMP configuration: [ 0.845288] .... node #0, CPUs: #1 [ 0.082460] smpboot: CPU 1 Converting physical 0 to logical die 1 [ 0.930693] smp: Brought up 1 node, 2 CPUs [ 0.931105] smpboot: Max logical packages: 2 [ 0.931529] smpboot: Total of 2 processors activated (8800.08 BogoMIPS) [ 0.934116] devtmpfs: initialized [ 0.939908] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.940847] futex hash table entries: 512 (order: 3, 32768 bytes, linear) [ 0.942181] thermal_sys: Registered thermal governor 'step_wise' [ 0.943084] NET: Registered protocol family 16 [ 0.947237] cpuidle: using governor ladder [ 0.947776] ACPI: bus type PCI registered [ 0.948198] PCI: Using configuration type 1 for base access [ 0.991085] ACPI: Added _OSI(Module Device) [ 0.991490] ACPI: Added _OSI(Processor Device) [ 0.991906] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.992352] ACPI: Added _OSI(Processor Aggregator Device) [ 0.992886] ACPI: Added _OSI(Linux-Dell-Video) [ 0.993328] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) [ 0.993847] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics) [ 1.054669] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 1.068921] ACPI: Interpreter enabled [ 1.069374] ACPI: (supports S0 S5) [ 1.069727] ACPI: Using IOAPIC for interrupt routing [ 1.070490] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.075665] ACPI: Enabled 2 GPEs in block 00 to 0F [ 1.205983] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 1.206648] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments HPX-Type3] [ 1.208662] PCI host bridge to bus 0000:00 [ 1.209115] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 1.209846] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 1.210612] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.211409] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 1.212210] pci_bus 0000:00: root bus resource [mem 0x240000000-0x2bfffffff window] [ 1.213019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 1.213870] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 1.219099] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.224273] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.226467] pci 0000:00:01.1: reg 0x20: [io 0xc040-0xc04f] [ 1.227372] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.228130] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.228822] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.229575] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.234806] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.235820] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.236583] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.241632] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000 [ 1.243100] pci 0000:00:02.0: reg 0x10: [mem 0xfd000000-0xfdffffff pref] [ 1.245696] pci 0000:00:02.0: reg 0x18: [mem 0xfebf0000-0xfebf0fff] [ 1.250179] pci 0000:00:02.0: reg 0x30: [mem 0xfebe0000-0xfebeffff pref] [ 1.255865] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 [ 1.257699] pci 0000:00:03.0: reg 0x10: [io 0xc000-0xc01f] [ 1.259134] pci 0000:00:03.0: reg 0x14: [mem 0xfebf1000-0xfebf1fff] [ 1.263153] pci 0000:00:03.0: reg 0x20: [mem 0xfe000000-0xfe003fff 64bit pref] [ 1.269443] pci 0000:00:04.0: [1af4:1005] type 00 class 0x00ff00 [ 1.270595] pci 0000:00:04.0: reg 0x10: [io 0xc020-0xc03f] [ 1.274093] pci 0000:00:04.0: reg 0x20: [mem 0xfe004000-0xfe007fff 64bit pref] [ 1.280734] pci_bus 0000:00: on NUMA node 0 [ 1.294459] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.298012] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.301993] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.305991] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.307973] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.321220] pci 0000:00:02.0: vgaarb: setting as boot VGA device [ 1.321851] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none [ 1.328725] pci 0000:00:02.0: vgaarb: bridge control possible [ 1.329322] vgaarb: loaded [ 1.332461] SCSI subsystem initialized [ 1.335472] PCI: Using ACPI for IRQ routing [ 1.335945] PCI: pci_cache_line_size set to 64 bytes [ 1.336550] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.337193] e820: reserve RAM buffer [mem 0xbffe0000-0xbfffffff] [ 1.352252] clocksource: Switched to clocksource tsc-early [ 1.354516] FS-Cache: Loaded [ 1.355102] pnp: PnP ACPI init [ 1.356209] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.357706] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.359154] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.360129] pnp 00:03: [dma 2] [ 1.360734] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.362737] pnp 00:04: Plug and Play ACPI device, IDs PNP0400 (active) [ 1.364737] pnp 00:05: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.373485] pnp: PnP ACPI: found 6 devices [ 1.405228] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.406173] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.406815] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.407441] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.408109] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.408814] pci_bus 0000:00: resource 8 [mem 0x240000000-0x2bfffffff window] [ 1.410929] NET: Registered protocol family 2 [ 1.413083] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear) [ 1.414072] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 1.415316] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear) [ 1.416861] TCP: Hash tables configured (established 65536 bind 65536) [ 1.417751] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear) [ 1.418562] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear) [ 1.419885] NET: Registered protocol family 1 [ 1.420906] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.421516] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.422150] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.422824] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 1.423706] PCI: CLS 0 bytes, default 64 [ 1.424712] Trying to unpack rootfs image as initramfs... [ 2.192302] Freeing initrd memory: 123600K [ 2.193149] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.193816] software IO TLB: mapped [mem 0xbbfe0000-0xbffe0000] (64MB) [ 2.197110] Initialise system trusted keyrings [ 2.198286] workingset: timestamp_bits=62 max_order=21 bucket_order=0 [ 2.250032] SGI XFS with ACLs, security attributes, no debug enabled [ 2.263575] 9p: Installing v9fs 9p2000 file system support [ 2.264677] FS-Cache: Netfs '9p' registered for caching [ 2.268760] ceph: loaded (mds proto 32) [ 2.277100] Key type asymmetric registered [ 2.277552] Asymmetric key parser 'x509' registered [ 2.278199] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 2.278964] io scheduler mq-deadline registered [ 2.280657] io scheduler bfq registered [ 2.285338] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.286140] ACPI: Power Button [PWRF] [ 2.286878] Warning: Processor Platform Limit event detected, but not handled. [ 2.287583] Consider compiling CPUfreq support into your kernel. [ 5.565706] PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 8.845953] PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 8.848450] tsc: Refined TSC clocksource calibration: 2200.024 MHz [ 8.849102] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fb64996fe0, max_idle_ns: 440795277019 ns [ 8.850206] clocksource: Switched to clocksource tsc [ 8.850804] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 8.851797] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.859715] random: fast init done [ 8.860214] random: crng init done [ 8.907820] brd: module loaded [ 8.910454] rbd: loaded (major 253) [ 8.910937] Loading iSCSI transport class v2.0-870. [ 8.924742] iscsi: registered transport (tcp) [ 8.928843] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI [ 8.929782] e1000: Copyright (c) 1999-2006 Intel Corporation. [ 8.931703] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.934352] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.935175] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.939431] device-mapper: ioctl: 4.42.0-ioctl (2020-02-27) initialised: dm-devel@redhat.com [ 8.946918] NET: Registered protocol family 10 [ 8.952221] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.958247] Segment Routing with IPv6 [ 8.958958] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 8.963019] NET: Registered protocol family 17 [ 8.964432] 9pnet: Installing 9P2000 support [ 8.966187] Key type dns_resolver registered [ 8.966809] Key type ceph registered [ 8.969539] libceph: loaded (mon/osd proto 15/24) [ 8.970285] IPI shorthand broadcast: enabled [ 8.970942] sched_clock: Marking stable (8889931296, 78460886)->(8994523547, -26131365) [ 8.972589] Loading compiled-in X.509 certificates [ 8.974328] kmemleak: Kernel memory leak detector initialized (mem pool available: 14433) [ 8.981186] Freeing unused kernel image (initmem) memory: 1084K [ 8.982047] kmemleak: Automatic memory scanning thread started [ 9.438099] Write protecting the kernel read-only data: 18432k [ 9.444258] Freeing unused kernel image (text/rodata gap) memory: 2044K [ 9.446524] Freeing unused kernel image (rodata/data gap) memory: 936K [ 9.447452] rodata_test: all tests were successful [ 9.448118] Run /init as init process [ 9.448631] with arguments: [ 9.449047] /init [ 9.449367] with environment: [ 9.449805] HOME=/ [ 9.450152] TERM=linux [ 9.793324] dracut: dracut-049+git115.c2d8d6fb-1.1 [ 9.856624] dracut: FATAL: No or empty root= argument [ 9.857387] dracut: Refusing to continue [ 9.867175] dracut Warning: [ 9.883318] dracut Warning: dracut: FATAL: No or empty root= argument [ 9.884862] dracut Warning: dracut: Refusing to continue [ 10.348557] zram: Added device: zram0 [ 10.352322] zram: Added device: zram1 [ 10.357538] zram: Added device: zram2 [ 10.363696] zram: Added device: zram3 [ 10.367574] zram: Added device: zram4 [ 10.370238] zram: Added device: zram5 [ 10.419688] zram0: detected capacity change from 0 to 3221225472 [ 10.429761] zram1: detected capacity change from 0 to 3221225472 [ 10.439662] zram2: detected capacity change from 0 to 3221225472 [ 10.449521] zram3: detected capacity change from 0 to 3221225472 [ 10.458582] zram4: detected capacity change from 0 to 3221225472 [ 10.467675] zram5: detected capacity change from 0 to 3221225472 [ 10.590900] raid6: skip pq benchmark and using algorithm sse2x4 [ 10.591525] raid6: using intx1 recovery algorithm [ 10.604946] xor: measuring software checksum speed [ 10.644353] prefetch64-sse: 16683.000 MB/sec [ 10.684352] generic_sse: 12352.000 MB/sec [ 10.684793] xor: using function: prefetch64-sse (16683.000 MB/sec) [ 10.762364] Btrfs loaded, crc32c=crc32c-generic, debug=on, assert=on, ref-verify=on [ 10.763542] BTRFS: selftest: sectorsize: 4096 nodesize: 4096 [ 10.764125] BTRFS: selftest: running btrfs free space cache tests [ 10.765016] BTRFS: selftest: running extent only tests [ 10.765883] BTRFS: selftest: running bitmap only tests [ 10.766534] BTRFS: selftest: running bitmap and extent tests [ 10.767385] BTRFS: selftest: running space stealing from bitmap to extent tests [ 10.768729] BTRFS: selftest: running extent buffer operation tests [ 10.769360] BTRFS: selftest: running btrfs_split_item tests [ 10.770163] BTRFS: selftest: running extent I/O tests [ 10.770681] BTRFS: selftest: running find delalloc tests [ 10.943587] BTRFS: selftest: running find_first_clear_extent_bit test [ 10.944389] BTRFS: selftest: running extent buffer bitmap tests [ 11.011671] BTRFS: selftest: running inode tests [ 11.012110] BTRFS: selftest: running btrfs_get_extent tests [ 11.013735] BTRFS: selftest: running hole first btrfs_get_extent test [ 11.014670] BTRFS: selftest: running outstanding_extents tests [ 11.015952] BTRFS: selftest: running qgroup tests [ 11.016439] BTRFS: selftest: running qgroup add/remove tests [ 11.017837] BTRFS: selftest: running qgroup multiple refs test [ 11.019956] BTRFS: selftest: running free space tree tests [ 11.061631] BTRFS: selftest: sectorsize: 4096 nodesize: 8192 [ 11.062233] BTRFS: selftest: running btrfs free space cache tests [ 11.062969] BTRFS: selftest: running extent only tests [ 11.063541] BTRFS: selftest: running bitmap only tests [ 11.064154] BTRFS: selftest: running bitmap and extent tests [ 11.065034] BTRFS: selftest: running space stealing from bitmap to extent tests [ 11.066413] BTRFS: selftest: running extent buffer operation tests [ 11.067055] BTRFS: selftest: running btrfs_split_item tests [ 11.067763] BTRFS: selftest: running extent I/O tests [ 11.068255] BTRFS: selftest: running find delalloc tests [ 11.238357] BTRFS: selftest: running find_first_clear_extent_bit test [ 11.239153] BTRFS: selftest: running extent buffer bitmap tests [ 11.305129] BTRFS: selftest: running inode tests [ 11.305614] BTRFS: selftest: running btrfs_get_extent tests [ 11.307322] BTRFS: selftest: running hole first btrfs_get_extent test [ 11.308185] BTRFS: selftest: running outstanding_extents tests [ 11.309381] BTRFS: selftest: running qgroup tests [ 11.309885] BTRFS: selftest: running qgroup add/remove tests [ 11.311240] BTRFS: selftest: running qgroup multiple refs test [ 11.313332] BTRFS: selftest: running free space tree tests [ 11.356765] BTRFS: selftest: sectorsize: 4096 nodesize: 16384 [ 11.357393] BTRFS: selftest: running btrfs free space cache tests [ 11.358146] BTRFS: selftest: running extent only tests [ 11.358813] BTRFS: selftest: running bitmap only tests [ 11.360689] BTRFS: selftest: running bitmap and extent tests [ 11.363285] BTRFS: selftest: running space stealing from bitmap to extent tests [ 11.367156] BTRFS: selftest: running extent buffer operation tests [ 11.368886] BTRFS: selftest: running btrfs_split_item tests [ 11.370686] BTRFS: selftest: running extent I/O tests [ 11.371378] BTRFS: selftest: running find delalloc tests [ 11.584883] BTRFS: selftest: running find_first_clear_extent_bit test [ 11.585900] BTRFS: selftest: running extent buffer bitmap tests [ 11.664444] BTRFS: selftest: running inode tests [ 11.664922] BTRFS: selftest: running btrfs_get_extent tests [ 11.666539] BTRFS: selftest: running hole first btrfs_get_extent test [ 11.667436] BTRFS: selftest: running outstanding_extents tests [ 11.668712] BTRFS: selftest: running qgroup tests [ 11.669189] BTRFS: selftest: running qgroup add/remove tests [ 11.670556] BTRFS: selftest: running qgroup multiple refs test [ 11.672721] BTRFS: selftest: running free space tree tests [ 11.713644] BTRFS: selftest: sectorsize: 4096 nodesize: 32768 [ 11.714260] BTRFS: selftest: running btrfs free space cache tests [ 11.714992] BTRFS: selftest: running extent only tests [ 11.715645] BTRFS: selftest: running bitmap only tests [ 11.716959] BTRFS: selftest: running bitmap and extent tests [ 11.718598] BTRFS: selftest: running space stealing from bitmap to extent tests [ 11.721218] BTRFS: selftest: running extent buffer operation tests [ 11.722408] BTRFS: selftest: running btrfs_split_item tests [ 11.723831] BTRFS: selftest: running extent I/O tests [ 11.724811] BTRFS: selftest: running find delalloc tests [ 11.886515] BTRFS: selftest: running find_first_clear_extent_bit test [ 11.887300] BTRFS: selftest: running extent buffer bitmap tests [ 11.955221] BTRFS: selftest: running inode tests [ 11.955696] BTRFS: selftest: running btrfs_get_extent tests [ 11.957294] BTRFS: selftest: running hole first btrfs_get_extent test [ 11.958234] BTRFS: selftest: running outstanding_extents tests [ 11.959409] BTRFS: selftest: running qgroup tests [ 11.959861] BTRFS: selftest: running qgroup add/remove tests [ 11.961207] BTRFS: selftest: running qgroup multiple refs test [ 11.963361] BTRFS: selftest: running free space tree tests [ 12.005711] BTRFS: selftest: sectorsize: 4096 nodesize: 65536 [ 12.006321] BTRFS: selftest: running btrfs free space cache tests [ 12.007061] BTRFS: selftest: running extent only tests [ 12.007622] BTRFS: selftest: running bitmap only tests [ 12.008239] BTRFS: selftest: running bitmap and extent tests [ 12.009223] BTRFS: selftest: running space stealing from bitmap to extent tests [ 12.010592] BTRFS: selftest: running extent buffer operation tests [ 12.011223] BTRFS: selftest: running btrfs_split_item tests [ 12.011936] BTRFS: selftest: running extent I/O tests [ 12.012455] BTRFS: selftest: running find delalloc tests [ 12.175015] BTRFS: selftest: running find_first_clear_extent_bit test [ 12.175796] BTRFS: selftest: running extent buffer bitmap tests [ 12.243650] BTRFS: selftest: running inode tests [ 12.244125] BTRFS: selftest: running btrfs_get_extent tests [ 12.245730] BTRFS: selftest: running hole first btrfs_get_extent test [ 12.246655] BTRFS: selftest: running outstanding_extents tests [ 12.247804] BTRFS: selftest: running qgroup tests [ 12.248264] BTRFS: selftest: running qgroup add/remove tests [ 12.249640] BTRFS: selftest: running qgroup multiple refs test [ 12.251740] BTRFS: selftest: running free space tree tests [ 12.293852] BTRFS: selftest: running extent_map tests [ 12.294807] BTRFS: selftest: running rmap tests [ 12.327257] BTRFS: device fsid ee267483-a433-4b1b-b876-4b63dc196002 devid 1 transid 5 /dev/zram0 scanned by mkfs.btrfs (213) [ 12.347590] BTRFS info (device zram0): disk space caching is enabled [ 12.349138] BTRFS info (device zram0): has skinny extents [ 12.350292] BTRFS info (device zram0): flagging fs with big metadata feature [ 12.366656] BTRFS info (device zram0): enabling ssd optimizations [ 12.368934] BTRFS info (device zram0): checking UUID tree [ 91.856300] kmemleak: Automatic memory scanning thread ended [ 100.340620] BTRFS: device fsid facfaba2-1682-4b38-b1e2-7c2fdcfc0e37 devid 1 transid 5 /dev/zram1 scanned by mkfs.btrfs (435) [ 100.407707] BTRFS info (device zram1): disk space caching is enabled [ 100.409505] BTRFS info (device zram1): has skinny extents [ 100.410242] BTRFS info (device zram1): flagging fs with big metadata feature [ 100.431160] BTRFS info (device zram1): enabling ssd optimizations [ 100.434768] BTRFS info (device zram1): checking UUID tree [ 100.603013] BTRFS info (device zram0): disk space caching is enabled [ 100.603878] BTRFS info (device zram0): has skinny extents [ 100.625292] BTRFS info (device zram0): enabling ssd optimizations [ 100.732105] run fstests btrfs/028 at 2020-05-13 11:47:40 [ 101.558205] BTRFS: device fsid 99328c0b-e5c6-4294-b029-52349bd82617 devid 1 transid 5 /dev/zram1 scanned by mkfs.btrfs (684) [ 101.614335] BTRFS info (device zram1): disk space caching is enabled [ 101.615018] BTRFS info (device zram1): has skinny extents [ 101.615575] BTRFS info (device zram1): flagging fs with big metadata feature [ 101.632747] BTRFS info (device zram1): enabling ssd optimizations [ 101.634937] BTRFS info (device zram1): checking UUID tree [ 101.655198] BTRFS info (device zram1): qgroup scan completed (inconsistency flag cleared) [ 101.669886] BTRFS info (device zram1): qgroup scan completed (inconsistency flag cleared) [ 101.691198] BTRFS info (device zram1): balance: start -d [ 101.693380] BTRFS info (device zram1): relocating block group 13631488 flags data [ 101.735597] BTRFS info (device zram1): balance: ended with status: 0 [ 101.794444] BTRFS info (device zram1): balance: start -d [ 101.803402] BTRFS info (device zram1): relocating block group 22020096 flags data [ 101.926736] BTRFS info (device zram1): found 10 extents, stage: move data extents [ 102.012792] BTRFS info (device zram1): found 2 extents, stage: update data pointers [ 102.115506] BTRFS info (device zram1): balance: ended with status: 0 [ 102.190146] BTRFS info (device zram1): balance: start -d [ 102.194042] BTRFS info (device zram1): relocating block group 357564416 flags data [ 102.289709] BTRFS info (device zram1): found 11 extents, stage: move data extents [ 102.354188] BTRFS info (device zram1): balance: ended with status: 0 [ 102.416422] BTRFS info (device zram1): balance: start -d [ 102.421795] BTRFS info (device zram1): relocating block group 693108736 flags data [ 102.517431] BTRFS info (device zram1): found 10 extents, stage: move data extents [ 102.547929] BTRFS info (device zram1): found 1 extents, stage: update data pointers [ 102.583036] BTRFS info (device zram1): balance: ended with status: 0 [ 102.616377] BTRFS info (device zram1): balance: start -d [ 102.621194] BTRFS info (device zram1): relocating block group 1028653056 flags data [ 102.702836] BTRFS info (device zram1): found 3 extents, stage: move data extents [ 102.771673] BTRFS info (device zram1): balance: ended with status: 0 [ 102.818480] BTRFS info (device zram1): balance: start -d [ 102.822449] BTRFS info (device zram1): relocating block group 1364197376 flags data [ 102.861911] BTRFS info (device zram1): found 1 extents, stage: move data extents [ 102.893575] BTRFS info (device zram1): balance: ended with status: 0 [ 102.923675] BTRFS info (device zram1): balance: start -d [ 102.929569] BTRFS info (device zram1): relocating block group 1699741696 flags data [ 102.954953] BTRFS info (device zram1): balance: ended with status: 0 [ 103.025334] BTRFS info (device zram1): balance: start -d [ 103.037426] BTRFS info (device zram1): relocating block group 2035286016 flags data [ 103.122232] BTRFS info (device zram1): found 5 extents, stage: move data extents [ 103.226419] BTRFS info (device zram1): balance: ended with status: 0 [ 103.279285] BTRFS info (device zram1): balance: start -d [ 103.281939] BTRFS info (device zram1): relocating block group 2370830336 flags data [ 103.357618] BTRFS info (device zram1): found 9 extents, stage: move data extents [ 103.398757] BTRFS info (device zram1): found 3 extents, stage: update data pointers [ 103.438958] BTRFS info (device zram1): balance: ended with status: 0 [ 103.470934] BTRFS info (device zram1): balance: start -d [ 103.475474] BTRFS info (device zram1): relocating block group 2706374656 flags data [ 103.522741] BTRFS info (device zram1): found 1 extents, stage: move data extents [ 103.557964] BTRFS info (device zram1): balance: ended with status: 0 [ 103.593913] BTRFS info (device zram1): balance: start -d [ 103.605763] BTRFS info (device zram1): relocating block group 3041918976 flags data [ 103.671578] BTRFS info (device zram1): found 7 extents, stage: move data extents [ 103.715241] BTRFS info (device zram1): balance: ended with status: 0 [ 103.745683] BTRFS info (device zram1): balance: start -d [ 103.751039] BTRFS info (device zram1): relocating block group 3377463296 flags data [ 103.776387] BTRFS info (device zram1): balance: ended with status: 0 [ 103.806670] BTRFS info (device zram1): balance: start -d [ 103.815855] BTRFS info (device zram1): relocating block group 3713007616 flags data [ 103.870624] BTRFS info (device zram1): found 3 extents, stage: move data extents [ 103.914588] BTRFS info (device zram1): balance: ended with status: 0 [ 103.953775] BTRFS info (device zram1): balance: start -d [ 103.959377] BTRFS info (device zram1): relocating block group 4048551936 flags data [ 104.037340] BTRFS info (device zram1): found 5 extents, stage: move data extents [ 104.079935] BTRFS info (device zram1): found 2 extents, stage: update data pointers [ 104.112588] BTRFS info (device zram1): balance: ended with status: 0 [ 104.148486] BTRFS info (device zram1): balance: start -d [ 104.151146] BTRFS info (device zram1): relocating block group 4384096256 flags data [ 104.370288] BTRFS info (device zram1): found 21 extents, stage: move data extents [ 104.412890] BTRFS info (device zram1): balance: ended with status: 0 [ 104.446927] BTRFS info (device zram1): balance: start -d [ 104.453099] BTRFS info (device zram1): relocating block group 4719640576 flags data [ 104.585025] BTRFS info (device zram1): found 6 extents, stage: move data extents [ 104.684846] BTRFS info (device zram1): balance: ended with status: 0 [ 104.756936] BTRFS info (device zram1): balance: start -d [ 104.773942] BTRFS info (device zram1): relocating block group 5055184896 flags data [ 104.950791] BTRFS info (device zram1): found 14 extents, stage: move data extents [ 105.090886] BTRFS info (device zram1): found 3 extents, stage: update data pointers [ 105.207900] BTRFS info (device zram1): balance: ended with status: 0 [ 105.250306] BTRFS info (device zram1): balance: start -d [ 105.253014] BTRFS info (device zram1): relocating block group 5390729216 flags data [ 105.322589] BTRFS info (device zram1): found 6 extents, stage: move data extents [ 105.372642] BTRFS info (device zram1): found 1 extents, stage: update data pointers [ 105.414903] BTRFS info (device zram1): balance: ended with status: 0 [ 105.446151] BTRFS info (device zram1): balance: start -d [ 105.451814] BTRFS info (device zram1): relocating block group 5726273536 flags data [ 105.614075] BTRFS info (device zram1): found 14 extents, stage: move data extents [ 105.666515] BTRFS info (device zram1): found 2 extents, stage: update data pointers [ 105.708743] BTRFS info (device zram1): balance: ended with status: 0 [ 105.749616] BTRFS info (device zram1): balance: start -d [ 105.754530] BTRFS info (device zram1): relocating block group 6061817856 flags data [ 105.814052] BTRFS info (device zram1): found 6 extents, stage: move data extents [ 105.867463] BTRFS info (device zram1): found 3 extents, stage: update data pointers [ 105.970152] BTRFS info (device zram1): balance: ended with status: 0 [ 106.037924] BTRFS info (device zram1): balance: start -d [ 106.049718] BTRFS info (device zram1): relocating block group 6397362176 flags data [ 106.131016] BTRFS info (device zram1): found 5 extents, stage: move data extents [ 106.182276] BTRFS info (device zram1): balance: ended with status: 0 [ 106.221259] BTRFS info (device zram1): balance: start -d [ 106.225156] BTRFS info (device zram1): relocating block group 6732906496 flags data [ 106.300434] BTRFS info (device zram1): found 2 extents, stage: move data extents [ 106.396793] BTRFS info (device zram1): balance: ended with status: 0 [ 106.431796] BTRFS info (device zram1): balance: start -d [ 106.444708] BTRFS info (device zram1): relocating block group 7068450816 flags data [ 106.476413] BTRFS info (device zram1): found 1 extents, stage: move data extents [ 106.504342] BTRFS info (device zram1): balance: ended with status: 0 [ 106.530801] BTRFS info (device zram1): balance: start -d [ 106.542725] BTRFS info (device zram1): relocating block group 7403995136 flags data [ 106.609953] BTRFS info (device zram1): balance: ended with status: 0 [ 106.675311] BTRFS info (device zram1): balance: start -d [ 106.687304] BTRFS info (device zram1): relocating block group 7739539456 flags data [ 106.807621] BTRFS info (device zram1): found 11 extents, stage: move data extents [ 106.899079] BTRFS info (device zram1): balance: ended with status: 0 [ 106.954156] BTRFS info (device zram1): balance: start -d [ 106.958271] BTRFS info (device zram1): relocating block group 8075083776 flags data [ 107.040905] BTRFS info (device zram1): found 14 extents, stage: move data extents [ 107.102913] BTRFS info (device zram1): found 12 extents, stage: update data pointers [ 107.150381] BTRFS info (device zram1): balance: ended with status: 0 [ 107.185289] BTRFS info (device zram1): balance: start -d [ 107.203224] BTRFS info (device zram1): relocating block group 8410628096 flags data [ 107.341105] BTRFS info (device zram1): found 22 extents, stage: move data extents [ 107.457973] BTRFS info (device zram1): found 12 extents, stage: update data pointers [ 107.552365] BTRFS info (device zram1): balance: ended with status: 0 [ 107.650969] BTRFS info (device zram1): balance: start -d [ 107.659650] BTRFS info (device zram1): relocating block group 8746172416 flags data [ 107.865106] BTRFS info (device zram1): found 19 extents, stage: move data extents [ 107.964861] BTRFS info (device zram1): found 4 extents, stage: update data pointers [ 108.058675] BTRFS info (device zram1): balance: ended with status: 0 [ 108.093332] BTRFS info (device zram1): balance: start -d [ 108.098199] BTRFS info (device zram1): relocating block group 9081716736 flags data [ 108.194178] BTRFS info (device zram1): found 17 extents, stage: move data extents [ 108.314179] BTRFS info (device zram1): found 11 extents, stage: update data pointers [ 108.407668] BTRFS info (device zram1): balance: ended with status: 0 [ 108.462258] BTRFS info (device zram1): balance: start -d [ 108.464899] BTRFS info (device zram1): relocating block group 9417261056 flags data [ 108.624913] BTRFS info (device zram1): found 23 extents, stage: move data extents [ 108.694130] BTRFS info (device zram1): found 14 extents, stage: update data pointers [ 108.736944] BTRFS info (device zram1): balance: ended with status: 0 [ 108.765029] BTRFS info (device zram1): balance: start -d [ 108.770189] BTRFS info (device zram1): relocating block group 9752805376 flags data [ 108.932352] BTRFS info (device zram1): found 28 extents, stage: move data extents [ 108.981375] BTRFS info (device zram1): found 2 extents, stage: update data pointers [ 109.016169] BTRFS info (device zram1): balance: ended with status: 0 [ 109.062165] BTRFS info (device zram1): balance: start -d [ 109.068630] BTRFS info (device zram1): relocating block group 10088349696 flags data [ 109.231787] BTRFS info (device zram1): found 12 extents, stage: move data extents [ 109.381323] BTRFS info (device zram1): found 4 extents, stage: update data pointers [ 109.485989] BTRFS info (device zram1): balance: ended with status: 0 [ 109.538976] BTRFS info (device zram1): balance: start -d [ 109.542823] BTRFS info (device zram1): relocating block group 10423894016 flags data [ 109.709069] BTRFS info (device zram1): found 23 extents, stage: move data extents [ 109.844119] BTRFS info (device zram1): found 16 extents, stage: update data pointers [ 109.946432] BTRFS info (device zram1): balance: ended with status: 0 [ 110.005083] BTRFS info (device zram1): balance: start -d [ 110.009205] BTRFS info (device zram1): relocating block group 10759438336 flags data [ 110.211262] BTRFS info (device zram1): found 21 extents, stage: move data extents [ 110.311553] BTRFS info (device zram1): found 7 extents, stage: update data pointers [ 110.397339] BTRFS info (device zram1): balance: ended with status: 0 [ 110.449486] BTRFS info (device zram1): balance: start -d [ 110.463676] BTRFS info (device zram1): relocating block group 11094982656 flags data [ 110.602316] BTRFS info (device zram1): found 16 extents, stage: move data extents [ 110.727579] BTRFS info (device zram1): found 10 extents, stage: update data pointers [ 110.857247] BTRFS info (device zram1): balance: ended with status: 0 [ 110.917966] BTRFS info (device zram1): balance: start -d [ 110.923576] BTRFS info (device zram1): relocating block group 11430526976 flags data [ 111.174154] BTRFS info (device zram1): found 37 extents, stage: move data extents [ 111.326162] BTRFS info (device zram1): found 21 extents, stage: update data pointers [ 111.480798] BTRFS info (device zram1): balance: ended with status: 0 [ 111.536928] BTRFS info (device zram1): balance: start -d [ 111.541594] BTRFS info (device zram1): relocating block group 11766071296 flags data [ 111.827456] BTRFS info (device zram1): found 36 extents, stage: move data extents [ 112.049714] BTRFS info (device zram1): found 29 extents, stage: update data pointers [ 112.198988] BTRFS info (device zram1): balance: ended with status: 0 [ 112.261825] BTRFS info (device zram1): balance: start -d [ 112.267170] BTRFS info (device zram1): relocating block group 12101615616 flags data [ 112.563540] BTRFS info (device zram1): found 39 extents, stage: move data extents [ 112.753981] BTRFS info (device zram1): found 15 extents, stage: update data pointers [ 112.902709] BTRFS info (device zram1): balance: ended with status: 0 [ 112.965764] BTRFS info (device zram1): balance: start -d [ 112.969805] BTRFS info (device zram1): relocating block group 12437159936 flags data [ 113.268425] BTRFS info (device zram1): found 44 extents, stage: move data extents [ 113.571189] BTRFS info (device zram1): found 39 extents, stage: update data pointers [ 113.841803] BTRFS info (device zram1): balance: ended with status: 0 [ 113.901172] BTRFS info (device zram1): balance: start -d [ 113.906105] BTRFS info (device zram1): relocating block group 12772704256 flags data [ 114.228217] BTRFS info (device zram1): found 56 extents, stage: move data extents [ 114.441556] BTRFS info (device zram1): found 42 extents, stage: update data pointers [ 114.589625] BTRFS info (device zram1): balance: ended with status: 0 [ 114.656714] BTRFS info (device zram1): balance: start -d [ 114.663372] BTRFS info (device zram1): relocating block group 13108248576 flags data [ 115.091216] BTRFS info (device zram1): found 49 extents, stage: move data extents [ 115.328552] BTRFS info (device zram1): found 6 extents, stage: update data pointers [ 115.483049] BTRFS info (device zram1): balance: ended with status: 0 [ 115.575523] BTRFS info (device zram1): balance: start -d [ 115.579596] BTRFS info (device zram1): relocating block group 13443792896 flags data [ 115.843025] BTRFS info (device zram1): found 41 extents, stage: move data extents [ 115.995682] BTRFS info (device zram1): found 22 extents, stage: update data pointers [ 116.114182] BTRFS info (device zram1): balance: ended with status: 0 [ 116.208764] BTRFS info (device zram1): balance: start -d [ 116.224506] BTRFS info (device zram1): relocating block group 13779337216 flags data [ 116.571208] BTRFS info (device zram1): found 43 extents, stage: move data extents [ 116.807278] BTRFS info (device zram1): found 9 extents, stage: update data pointers [ 116.986508] BTRFS info (device zram1): balance: ended with status: 0 [ 117.095661] BTRFS info (device zram1): balance: start -d [ 117.115637] BTRFS info (device zram1): relocating block group 14114881536 flags data [ 117.384672] BTRFS info (device zram1): found 51 extents, stage: move data extents [ 117.641877] BTRFS info (device zram1): found 38 extents, stage: update data pointers [ 117.801795] BTRFS info (device zram1): found 1 extents, stage: update data pointers [ 117.931877] BTRFS info (device zram1): balance: ended with status: 0 [ 118.004841] BTRFS info (device zram1): balance: start -d [ 118.011516] BTRFS info (device zram1): relocating block group 14450425856 flags data [ 118.278850] BTRFS info (device zram1): found 25 extents, stage: move data extents [ 118.357162] BTRFS info (device zram1): found 3 extents, stage: update data pointers [ 118.430276] BTRFS info (device zram1): balance: ended with status: 0 [ 118.473592] BTRFS info (device zram1): balance: start -d [ 118.484678] BTRFS info (device zram1): relocating block group 14785970176 flags data [ 118.597953] BTRFS info (device zram1): found 13 extents, stage: move data extents [ 118.728094] BTRFS info (device zram1): found 9 extents, stage: update data pointers [ 118.851539] BTRFS info (device zram1): balance: ended with status: 0 [ 118.890532] BTRFS info (device zram1): balance: start -d [ 118.895567] BTRFS info (device zram1): relocating block group 15121514496 flags data [ 119.028695] BTRFS info (device zram1): found 20 extents, stage: move data extents [ 119.141682] BTRFS info (device zram1): found 12 extents, stage: update data pointers [ 119.271458] BTRFS info (device zram1): balance: ended with status: 0 [ 119.327474] BTRFS info (device zram1): balance: start -d [ 119.332754] BTRFS info (device zram1): relocating block group 15457058816 flags data [ 119.581570] BTRFS info (device zram1): found 24 extents, stage: move data extents [ 119.816340] BTRFS info (device zram1): found 18 extents, stage: update data pointers [ 119.932668] BTRFS info (device zram1): balance: ended with status: 0 [ 120.017701] BTRFS info (device zram1): balance: start -d [ 120.029622] BTRFS info (device zram1): relocating block group 15792603136 flags data [ 120.286311] BTRFS info (device zram1): found 33 extents, stage: move data extents [ 120.444792] BTRFS info (device zram1): found 23 extents, stage: update data pointers [ 120.508598] BTRFS info (device zram1): balance: ended with status: 0 [ 120.555342] BTRFS info (device zram1): balance: start -d [ 120.559267] BTRFS info (device zram1): relocating block group 16128147456 flags data [ 120.809781] BTRFS info (device zram1): found 40 extents, stage: move data extents [ 120.959677] BTRFS info (device zram1): found 23 extents, stage: update data pointers [ 121.097418] BTRFS info (device zram1): balance: ended with status: 0 [ 121.161688] BTRFS info (device zram1): balance: start -d [ 121.166957] BTRFS info (device zram1): relocating block group 16463691776 flags data [ 121.339976] BTRFS info (device zram1): found 20 extents, stage: move data extents [ 121.513268] BTRFS info (device zram1): found 15 extents, stage: update data pointers [ 121.653531] BTRFS info (device zram1): balance: ended with status: 0 [ 121.732587] BTRFS info (device zram1): balance: start -d [ 121.736477] BTRFS info (device zram1): relocating block group 16799236096 flags data [ 122.049906] BTRFS info (device zram1): found 34 extents, stage: move data extents [ 122.247059] BTRFS info (device zram1): found 11 extents, stage: update data pointers [ 122.411034] BTRFS info (device zram1): balance: ended with status: 0 [ 122.468237] BTRFS info (device zram1): balance: start -d [ 122.473058] BTRFS info (device zram1): relocating block group 17134780416 flags data [ 122.627619] BTRFS info (device zram1): found 21 extents, stage: move data extents [ 122.749555] BTRFS info (device zram1): found 20 extents, stage: update data pointers [ 122.848919] BTRFS info (device zram1): balance: ended with status: 0 [ 122.893378] BTRFS info (device zram1): balance: start -d [ 122.897941] BTRFS info (device zram1): relocating block group 17470324736 flags data [ 123.111299] BTRFS info (device zram1): found 26 extents, stage: move data extents [ 123.261099] BTRFS info (device zram1): found 7 extents, stage: update data pointers [ 123.362856] BTRFS info (device zram1): balance: ended with status: 0 [ 123.410215] BTRFS info (device zram1): balance: start -d [ 123.421133] BTRFS info (device zram1): relocating block group 17805869056 flags data [ 123.608956] BTRFS info (device zram1): found 24 extents, stage: move data extents [ 123.732247] BTRFS info (device zram1): found 12 extents, stage: update data pointers [ 123.831218] BTRFS info (device zram1): balance: ended with status: 0 [ 123.907535] BTRFS info (device zram1): balance: start -d [ 123.914332] BTRFS info (device zram1): relocating block group 18141413376 flags data [ 124.132307] BTRFS info (device zram1): found 23 extents, stage: move data extents [ 124.249633] BTRFS info (device zram1): found 5 extents, stage: update data pointers [ 124.351459] BTRFS info (device zram1): balance: ended with status: 0 [ 124.399812] BTRFS info (device zram1): balance: start -d [ 124.402535] BTRFS info (device zram1): relocating block group 18476957696 flags data [ 124.659231] BTRFS info (device zram1): found 26 extents, stage: move data extents [ 124.840265] BTRFS info (device zram1): found 17 extents, stage: update data pointers [ 125.005601] BTRFS info (device zram1): balance: ended with status: 0 [ 125.112638] BTRFS info (device zram1): balance: start -d [ 125.116227] BTRFS info (device zram1): relocating block group 18812502016 flags data [ 125.403383] BTRFS info (device zram1): found 46 extents, stage: move data extents [ 125.512397] BTRFS info (device zram1): found 3 extents, stage: update data pointers [ 125.604700] BTRFS info (device zram1): balance: ended with status: 0 [ 125.642788] BTRFS info (device zram1): balance: start -d [ 125.649286] BTRFS info (device zram1): relocating block group 19148046336 flags data [ 125.849670] BTRFS info (device zram1): found 29 extents, stage: move data extents [ 126.045467] BTRFS info (device zram1): found 15 extents, stage: update data pointers [ 126.182621] BTRFS info (device zram1): balance: ended with status: 0 [ 126.267549] BTRFS info (device zram1): balance: start -d [ 126.280835] BTRFS info (device zram1): relocating block group 19483590656 flags data [ 126.566546] BTRFS info (device zram1): found 29 extents, stage: move data extents [ 126.708179] BTRFS info (device zram1): found 8 extents, stage: update data pointers [ 126.878932] BTRFS info (device zram1): balance: ended with status: 0 [ 126.979435] BTRFS info (device zram1): balance: start -d [ 126.984788] BTRFS info (device zram1): relocating block group 19819134976 flags data [ 127.319787] BTRFS info (device zram1): found 49 extents, stage: move data extents [ 127.516268] BTRFS info (device zram1): found 21 extents, stage: update data pointers [ 127.667637] BTRFS info (device zram1): balance: ended with status: 0 [ 127.725129] BTRFS info (device zram1): balance: start -d [ 127.730342] BTRFS info (device zram1): relocating block group 20154679296 flags data [ 127.964326] BTRFS info (device zram1): found 36 extents, stage: move data extents [ 128.112685] BTRFS info (device zram1): found 18 extents, stage: update data pointers [ 128.261205] BTRFS info (device zram1): balance: ended with status: 0 [ 128.352856] BTRFS info (device zram1): balance: start -d [ 128.356841] BTRFS info (device zram1): relocating block group 20490223616 flags data [ 128.578815] BTRFS info (device zram1): found 28 extents, stage: move data extents [ 128.727703] BTRFS info (device zram1): found 16 extents, stage: update data pointers [ 128.873412] BTRFS info (device zram1): balance: ended with status: 0 [ 128.915392] BTRFS info (device zram1): balance: start -d [ 128.921869] BTRFS info (device zram1): relocating block group 20825767936 flags data [ 129.042418] BTRFS info (device zram1): found 21 extents, stage: move data extents [ 129.147856] BTRFS info (device zram1): found 14 extents, stage: update data pointers [ 129.240346] BTRFS info (device zram1): balance: ended with status: 0 [ 129.320126] BTRFS info (device zram1): balance: start -d [ 129.324849] BTRFS info (device zram1): relocating block group 21161312256 flags data [ 129.575682] BTRFS info (device zram1): found 23 extents, stage: move data extents [ 129.732482] BTRFS info (device zram1): found 19 extents, stage: update data pointers [ 129.867402] BTRFS info (device zram1): balance: ended with status: 0 [ 129.957238] BTRFS info (device zram1): balance: start -d [ 129.976981] BTRFS info (device zram1): relocating block group 21496856576 flags data [ 130.258870] BTRFS info (device zram1): found 47 extents, stage: move data extents [ 130.519069] BTRFS info (device zram1): found 39 extents, stage: update data pointers [ 130.691861] BTRFS info (device zram1): balance: ended with status: 0 [ 130.781972] BTRFS info (device zram1): balance: start -d [ 130.788070] BTRFS info (device zram1): relocating block group 21832400896 flags data [ 131.170865] BTRFS info (device zram1): found 47 extents, stage: move data extents [ 131.403068] BTRFS info (device zram1): found 9 extents, stage: update data pointers [ 131.540487] BTRFS info (device zram1): balance: ended with status: 0 [ 131.634335] BTRFS info (device zram1): balance: start -d [ 131.639344] BTRFS info (device zram1): relocating block group 22167945216 flags data [ 131.836136] BTRFS info (device zram1): balance: canceled [ 132.005718] BTRFS warning (device zram1): page private not zero on page 7110656 [ 132.007230] BTRFS warning (device zram1): page private not zero on page 7114752 [ 132.008496] BTRFS warning (device zram1): page private not zero on page 7118848 [ 132.010104] BTRFS warning (device zram1): page private not zero on page 7122944 [ 132.011258] BTRFS warning (device zram1): page private not zero on page 7176192 [ 132.012410] BTRFS warning (device zram1): page private not zero on page 7180288 [ 132.013555] BTRFS warning (device zram1): page private not zero on page 7184384 [ 132.014701] BTRFS warning (device zram1): page private not zero on page 7188480 [ 132.017214] BTRFS error (device zram1): leaked root 18446744073709551608-5 refcount 1 [ 132.018063] BTRFS error (device zram1): leaked root 18446744073709551608-18446744073709551607 refcount 1 [ 132.019058] BTRFS: buffer leak start 7176192 len 16384 refs 1 bflags 529 owner 18446744073709551607 [ 132.019991] BTRFS: buffer leak start 7110656 len 16384 refs 1 bflags 529 owner 5 [ 140.112118] kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [-- Attachment #3: 028.full --] [-- Type: text/plain, Size: 356 bytes --] # /home/johannes/src/btrfs-progs/btrfs quota enable /mnt/scratch # /home/johannes/src/btrfs-progs/btrfs quota rescan -w /mnt/scratch quota rescan started Run fsstress -z -f write=10 -f unlink=10 -f creat=10 -f fsync=10 -f fsync=10 -n 100000 -p 2 -d /mnt/scratch/stress_dir Start balance # /home/johannes/src/btrfs-progs/btrfs filesystem sync /mnt/scratch [-- Attachment #4: 028.kmemleak --] [-- Type: text/plain, Size: 2582 bytes --] EXPERIMENTAL kmemleak reported some memory leaks! Due to the way kmemleak works, the leak might be from an earlier test, or something totally unrelated. unreferenced object 0xffff8881f646e940 (size 576): comm "mount", pid 695, jiffies 4294917533 (age 38.508s) hex dump (first 32 bytes): 06 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 68 2a 19 e0 81 88 ff ff 58 e9 46 f6 81 88 ff ff h*......X.F..... backtrace: [<00000000dff11222>] radix_tree_node_alloc.constprop.0+0xd7/0x160 [<000000000d1e18ac>] radix_tree_insert+0x117/0x2c0 [<00000000d19a3dbb>] alloc_extent_buffer+0x56a/0x6d0 [btrfs] [<0000000018b71b0f>] read_tree_block+0x15/0x60 [btrfs] [<0000000010a55e05>] open_ctree+0x137c/0x277a [btrfs] [<0000000036160822>] btrfs_mount_root.cold+0x12/0xec [btrfs] [<00000000388e9139>] legacy_get_tree+0x6b/0xa0 [<00000000e21e39e7>] vfs_get_tree+0x42/0xf0 [<000000002689bf53>] fc_mount+0xa/0x40 [<000000002a92a0e8>] vfs_kern_mount.part.0+0x6c/0x80 [<0000000069d5b91e>] btrfs_mount+0x1ba/0x5d0 [btrfs] [<00000000388e9139>] legacy_get_tree+0x6b/0xa0 [<00000000e21e39e7>] vfs_get_tree+0x42/0xf0 [<0000000041afc687>] do_mount+0xa90/0xe00 [<00000000e2f69bd0>] __x64_sys_mount+0xb9/0xf0 [<00000000667f0b0a>] do_syscall_64+0x5e/0x1c0 unreferenced object 0xffff8881cad8ec00 (size 576): comm "fsstress", pid 723, jiffies 4294921132 (age 24.112s) hex dump (first 32 bytes): 00 1b 02 00 00 00 00 00 40 e9 46 f6 81 88 ff ff ........@.F..... 68 2a 19 e0 81 88 ff ff 18 ec d8 ca 81 88 ff ff h*.............. backtrace: [<00000000dff11222>] radix_tree_node_alloc.constprop.0+0xd7/0x160 [<000000000d1e18ac>] radix_tree_insert+0x117/0x2c0 [<00000000d19a3dbb>] alloc_extent_buffer+0x56a/0x6d0 [btrfs] [<00000000e36da7fe>] btrfs_init_new_buffer+0x3e/0x3b0 [btrfs] [<00000000a40b3d56>] btrfs_alloc_tree_block+0x1f7/0x4b0 [btrfs] [<00000000d37cefa6>] alloc_tree_block_no_bg_flush+0xba/0xe0 [btrfs] [<0000000049aac05e>] __btrfs_cow_block+0x22f/0x840 [btrfs] [<000000004b88f003>] btrfs_cow_block+0x1f1/0x290 [btrfs] [<000000008f1ed34a>] btrfs_search_slot+0x878/0xef0 [btrfs] [<0000000066c12130>] btrfs_insert_empty_items+0x77/0xd0 [btrfs] [<0000000002f3d164>] btrfs_new_inode+0x52c/0x9c0 [btrfs] [<00000000741f8f60>] btrfs_create+0x171/0x310 [btrfs] [<00000000ed9bffe6>] path_openat+0x840/0x1510 [<000000002e5e9c08>] do_filp_open+0x10c/0x180 [<0000000035e673af>] do_sys_openat2+0x28f/0x370 [<000000007e6b0b42>] do_sys_open+0x75/0xb0 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-13 11:54 ` Johannes Thumshirn @ 2020-05-13 11:57 ` Qu Wenruo 2020-05-13 12:06 ` Johannes Thumshirn 0 siblings, 1 reply; 24+ messages in thread From: Qu Wenruo @ 2020-05-13 11:57 UTC (permalink / raw) To: Johannes Thumshirn, dsterba, David Sterba; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 345 bytes --] On 2020/5/13 下午7:54, Johannes Thumshirn wrote: > On 13/05/2020 01:04, David Sterba wrote: > [...] >> >> Johannes, do you have logs from the test? > > > > I recreated the logs for btrfs/028 (dmesg, kmemleak and fstests log). Please find them attached. > BTW, what's the line of open_ctree+0x137c/0x277a? Thanks, Qu [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-13 11:57 ` Qu Wenruo @ 2020-05-13 12:06 ` Johannes Thumshirn 2020-05-13 12:11 ` Qu Wenruo 0 siblings, 1 reply; 24+ messages in thread From: Johannes Thumshirn @ 2020-05-13 12:06 UTC (permalink / raw) To: Qu Wenruo, dsterba, David Sterba; +Cc: linux-btrfs On 13/05/2020 13:57, Qu Wenruo wrote: > > > On 2020/5/13 下午7:54, Johannes Thumshirn wrote: >> On 13/05/2020 01:04, David Sterba wrote: >> [...] >>> >>> Johannes, do you have logs from the test? >> >> >> >> I recreated the logs for btrfs/028 (dmesg, kmemleak and fstests log). Please find them attached. >> > > BTW, what's the line of open_ctree+0x137c/0x277a? Here we go: (gdb) l *(open_ctree+0x137c/0x277a) 0x122acd is in open_ctree (fs/btrfs/disk-io.c:2826). 2821 u64 generation; 2822 u64 features; 2823 u16 csum_type; 2824 struct btrfs_key location; 2825 struct btrfs_super_block *disk_super; 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); 2827 struct btrfs_root *tree_root; 2828 struct btrfs_root *chunk_root; 2829 int ret; 2830 int err = -EINVAL; So its: 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-13 12:06 ` Johannes Thumshirn @ 2020-05-13 12:11 ` Qu Wenruo 2020-05-13 12:17 ` Johannes Thumshirn 0 siblings, 1 reply; 24+ messages in thread From: Qu Wenruo @ 2020-05-13 12:11 UTC (permalink / raw) To: Johannes Thumshirn, dsterba, David Sterba; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 1238 bytes --] On 2020/5/13 下午8:06, Johannes Thumshirn wrote: > On 13/05/2020 13:57, Qu Wenruo wrote: >> >> >> On 2020/5/13 下午7:54, Johannes Thumshirn wrote: >>> On 13/05/2020 01:04, David Sterba wrote: >>> [...] >>>> >>>> Johannes, do you have logs from the test? >>> >>> >>> >>> I recreated the logs for btrfs/028 (dmesg, kmemleak and fstests log). Please find them attached. >>> >> >> BTW, what's the line of open_ctree+0x137c/0x277a? > > > Here we go: > (gdb) l *(open_ctree+0x137c/0x277a) > 0x122acd is in open_ctree (fs/btrfs/disk-io.c:2826). > 2821 u64 generation; > 2822 u64 features; > 2823 u16 csum_type; > 2824 struct btrfs_key location; > 2825 struct btrfs_super_block *disk_super; > 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); > 2827 struct btrfs_root *tree_root; > 2828 struct btrfs_root *chunk_root; > 2829 int ret; > 2830 int err = -EINVAL; > > So its: > 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); > This doesn't make sense. That line doesn't even call read_tree_block() nor even any function call. This looks really strange. Thanks, Qu [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-13 12:11 ` Qu Wenruo @ 2020-05-13 12:17 ` Johannes Thumshirn 2020-05-13 12:29 ` Johannes Thumshirn 0 siblings, 1 reply; 24+ messages in thread From: Johannes Thumshirn @ 2020-05-13 12:17 UTC (permalink / raw) To: Qu Wenruo, dsterba, David Sterba; +Cc: linux-btrfs On 13/05/2020 14:11, Qu Wenruo wrote: > > > On 2020/5/13 下午8:06, Johannes Thumshirn wrote: >> On 13/05/2020 13:57, Qu Wenruo wrote: >>> >>> >>> On 2020/5/13 下午7:54, Johannes Thumshirn wrote: >>>> On 13/05/2020 01:04, David Sterba wrote: >>>> [...] >>>>> >>>>> Johannes, do you have logs from the test? >>>> >>>> >>>> >>>> I recreated the logs for btrfs/028 (dmesg, kmemleak and fstests log). Please find them attached. >>>> >>> >>> BTW, what's the line of open_ctree+0x137c/0x277a? >> >> >> Here we go: >> (gdb) l *(open_ctree+0x137c/0x277a) >> 0x122acd is in open_ctree (fs/btrfs/disk-io.c:2826). >> 2821 u64 generation; >> 2822 u64 features; >> 2823 u16 csum_type; >> 2824 struct btrfs_key location; >> 2825 struct btrfs_super_block *disk_super; >> 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); >> 2827 struct btrfs_root *tree_root; >> 2828 struct btrfs_root *chunk_root; >> 2829 int ret; >> 2830 int err = -EINVAL; >> >> So its: >> 2826 struct btrfs_fs_info *fs_info = btrfs_sb(sb); >> > This doesn't make sense. > > That line doesn't even call read_tree_block() nor even any function call. > > This looks really strange. Indeed, it does. I have no clue what's going on here. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: root leak, eb leak 2020-05-13 12:17 ` Johannes Thumshirn @ 2020-05-13 12:29 ` Johannes Thumshirn 0 siblings, 0 replies; 24+ messages in thread From: Johannes Thumshirn @ 2020-05-13 12:29 UTC (permalink / raw) To: Qu Wenruo, dsterba, David Sterba; +Cc: linux-btrfs On 13/05/2020 14:17, Johannes Thumshirn wrote: [...] >>> >> This doesn't make sense. >> >> That line doesn't even call read_tree_block() nor even any function call. >> >> This looks really strange. > > Indeed, it does. I have no clue what's going on here. > I just retested without KASAN enabled (to rule out any problems in that area) and the test (btrfs/028) passes. Turning KASAN back on fails the test again. So I /suspect/ it's a race somewhere and changed timing (due to KASAN) triggers it. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Bug 5.7-rc: write-time leaf corruption detected 2020-05-12 14:12 Pending bugs for 5.7 David Sterba 2020-05-12 14:14 ` Bug 5.7-rc: root leak, eb leak David Sterba @ 2020-05-12 14:15 ` David Sterba 2020-05-12 14:26 ` Filipe Manana ` (2 more replies) 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex David Sterba 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, fs_reclaim David Sterba 3 siblings, 3 replies; 24+ messages in thread From: David Sterba @ 2020-05-12 14:15 UTC (permalink / raw) To: linux-btrfs Happened once in a VM, defaul mkfs/mount options. generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 [17607.625840] BTRFS: device fsid ae51523a-71e7-4730-9097-faeca36779be devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (27036) [17607.664903] BTRFS info (device vdb): disk space caching is enabled [17607.667422] BTRFS info (device vdb): has skinny extents [17607.669552] BTRFS info (device vdb): flagging fs with big metadata feature [17607.681613] BTRFS info (device vdb): checking UUID tree [17607.921588] BTRFS: device fsid 16238d8a-1af3-4fa1-b3bc-fd6f04469192 devid 1 transid 5 /dev/mapper/logwrites-test scanned by mkfs.btrfs (27101) [17607.948085] BTRFS info (device dm-0): turning on sync discard [17607.952647] BTRFS info (device dm-0): disk space caching is enabled [17607.955088] BTRFS info (device dm-0): has skinny extents [17607.956924] BTRFS info (device dm-0): flagging fs with big metadata feature [17607.966128] BTRFS info (device dm-0): checking UUID tree [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 [17608.106426] item 0 key (257 1 0) itemoff 16123 itemsize 160 [17608.108766] inode generation 6 size 262144 mode 100600 [17608.110947] item 1 key (257 12 256) itemoff 16103 itemsize 20 [17608.112698] item 2 key (257 108 0) itemoff 16050 itemsize 53 [17608.114568] extent data disk bytenr 13631488 nr 262144 [17608.116352] extent data offset 0 nr 262144 ram 262144 [17608.118082] item 3 key (258 1 0) itemoff 15890 itemsize 160 [17608.119920] inode generation 6 size 262144 mode 100600 [17608.121605] item 4 key (258 12 256) itemoff 15871 itemsize 19 [17608.123293] item 5 key (258 108 0) itemoff 15818 itemsize 53 [17608.125212] extent data disk bytenr 13905920 nr 4096 [17608.126851] extent data offset 0 nr 4096 ram 4096 [17608.128342] item 6 key (258 108 4096) itemoff 15765 itemsize 53 [17608.130106] extent data disk bytenr 13631488 nr 262144 [17608.131774] extent data offset 0 nr 4096 ram 262144 [17608.133330] item 7 key (258 108 8192) itemoff 15712 itemsize 53 [17608.135151] extent data disk bytenr 13631488 nr 262144 [17608.136680] extent data offset 8192 nr 8192 ram 262144 [17608.137931] item 8 key (258 108 16384) itemoff 15659 itemsize 53 [17608.139257] extent data disk bytenr 13631488 nr 262144 [17608.140939] extent data offset 204800 nr 16384 ram 262144 [17608.142255] item 9 key (258 108 32768) itemoff 15606 itemsize 53 [17608.143941] extent data disk bytenr 13910016 nr 28672 [17608.145576] extent data offset 0 nr 28672 ram 28672 [17608.147165] item 10 key (258 108 61440) itemoff 15553 itemsize 53 [17608.148964] extent data disk bytenr 13631488 nr 262144 [17608.150563] extent data offset 249856 nr 8192 ram 262144 [17608.152204] item 11 key (258 108 69632) itemoff 15500 itemsize 53 [17608.153974] extent data disk bytenr 13631488 nr 262144 [17608.155588] extent data offset 69632 nr 126976 ram 262144 [17608.157099] item 12 key (258 108 196608) itemoff 15447 itemsize 53 [17608.158698] extent data disk bytenr 13938688 nr 4096 [17608.160123] extent data offset 0 nr 4096 ram 4096 [17608.161300] item 13 key (258 108 200704) itemoff 15394 itemsize 53 [17608.162597] extent data disk bytenr 0 nr 0 [17608.164057] extent data offset 0 nr 28672 ram 28672 [17608.165782] item 14 key (258 108 229376) itemoff 15341 itemsize 53 [17608.167691] extent data disk bytenr 13942784 nr 4096 [17608.169259] extent data offset 0 nr 4096 ram 4096 [17608.170821] item 15 key (258 108 233472) itemoff 15288 itemsize 53 [17608.172687] extent data disk bytenr 13631488 nr 262144 [17608.174310] extent data offset 233472 nr 12288 ram 262144 [17608.175993] item 16 key (258 108 245760) itemoff 15235 itemsize 53 [17608.177821] extent data disk bytenr 13946880 nr 16384 [17608.179440] extent data offset 0 nr 16384 ram 16384 [17608.181043] item 17 key (259 1 0) itemoff 15075 itemsize 160 [17608.182796] inode generation 6 size 262144 mode 100600 [17608.184424] item 18 key (259 12 256) itemoff 15056 itemsize 19 [17608.186085] item 19 key (259 108 0) itemoff 15003 itemsize 53 [17608.187859] extent data disk bytenr 13975552 nr 4096 [17608.189525] extent data offset 0 nr 4096 ram 4096 [17608.191074] item 20 key (259 108 4096) itemoff 14950 itemsize 53 [17608.192816] extent data disk bytenr 13631488 nr 262144 [17608.194464] extent data offset 0 nr 4096 ram 262144 [17608.196206] item 21 key (259 108 8192) itemoff 14897 itemsize 53 [17608.198212] extent data disk bytenr 13631488 nr 262144 [17608.200048] extent data offset 8192 nr 106496 ram 262144 [17608.201684] item 22 key (259 108 114688) itemoff 14844 itemsize 53 [17608.203659] extent data disk bytenr 14217216 nr 49152 [17608.205398] extent data offset 0 nr 49152 ram 49152 [17608.207126] item 23 key (259 108 163840) itemoff 14791 itemsize 53 [17608.209019] extent data disk bytenr 13631488 nr 262144 [17608.210736] extent data offset 163840 nr 98304 ram 262144 [17608.212371] item 24 key (261 1 0) itemoff 14631 itemsize 160 [17608.222944] inode generation 6 size 262144 mode 100600 [17608.224637] item 25 key (261 12 256) itemoff 14612 itemsize 19 [17608.226400] item 26 key (261 108 0) itemoff 14559 itemsize 53 [17608.228169] extent data disk bytenr 14123008 nr 4096 [17608.229794] extent data offset 0 nr 4096 ram 4096 [17608.231292] item 27 key (261 108 4096) itemoff 14506 itemsize 53 [17608.233203] extent data disk bytenr 13631488 nr 262144 [17608.234794] extent data offset 0 nr 4096 ram 262144 [17608.236362] item 28 key (261 108 8192) itemoff 14453 itemsize 53 [17608.238102] extent data disk bytenr 13631488 nr 262144 [17608.239782] extent data offset 8192 nr 16384 ram 262144 [17608.241587] item 29 key (261 108 24576) itemoff 14400 itemsize 53 [17608.243431] extent data disk bytenr 13631488 nr 262144 [17608.245176] extent data offset 212992 nr 36864 ram 262144 [17608.246991] item 30 key (261 108 61440) itemoff 14347 itemsize 53 [17608.248868] extent data disk bytenr 13631488 nr 262144 [17608.250456] extent data offset 61440 nr 114688 ram 262144 [17608.252333] item 31 key (261 108 176128) itemoff 14294 itemsize 53 [17608.254343] extent data disk bytenr 14135296 nr 61440 [17608.256131] extent data offset 0 nr 61440 ram 61440 [17608.257857] item 32 key (261 108 237568) itemoff 14241 itemsize 53 [17608.259864] extent data disk bytenr 13631488 nr 262144 [17608.261484] extent data offset 237568 nr 24576 ram 262144 [17608.263134] item 33 key (262 1 0) itemoff 14081 itemsize 160 [17608.264865] inode generation 6 size 178311 mode 100600 [17608.266625] item 34 key (262 12 256) itemoff 14062 itemsize 19 [17608.268551] item 35 key (262 108 0) itemoff 14009 itemsize 53 [17608.270363] extent data disk bytenr 14196736 nr 4096 [17608.272019] extent data offset 0 nr 4096 ram 4096 [17608.273505] item 36 key (262 108 4096) itemoff 13956 itemsize 53 [17608.275387] extent data disk bytenr 13631488 nr 262144 [17608.277125] extent data offset 0 nr 4096 ram 262144 [17608.278780] item 37 key (262 108 8192) itemoff 13903 itemsize 53 [17608.280507] extent data disk bytenr 13631488 nr 262144 [17608.282163] extent data offset 8192 nr 106496 ram 262144 [17608.283866] item 38 key (262 108 114688) itemoff 13850 itemsize 53 [17608.285727] extent data disk bytenr 14200832 nr 4096 [17608.287264] extent data offset 0 nr 4096 ram 4096 [17608.288719] item 39 key (262 108 118784) itemoff 13797 itemsize 53 [17608.290696] extent data disk bytenr 0 nr 0 [17608.292237] extent data offset 0 nr 49152 ram 49152 [17608.293667] item 40 key (262 108 167936) itemoff 13744 itemsize 53 [17608.295436] extent data disk bytenr 14209024 nr 4096 [17608.297051] extent data offset 0 nr 4096 ram 4096 [17608.298583] item 41 key (262 108 172032) itemoff 13691 itemsize 53 [17608.300573] extent data disk bytenr 13631488 nr 262144 [17608.302323] extent data offset 172032 nr 8192 ram 262144 [17608.304112] item 42 key (263 1 0) itemoff 13531 itemsize 160 [17608.305810] inode generation 6 size 262144 mode 100600 [17608.307444] item 43 key (263 12 256) itemoff 13512 itemsize 19 [17608.309108] item 44 key (263 108 0) itemoff 13459 itemsize 53 [17608.310760] extent data disk bytenr 14266368 nr 4096 [17608.312269] extent data offset 0 nr 4096 ram 4096 [17608.313883] item 45 key (263 108 4096) itemoff 13406 itemsize 53 [17608.315659] extent data disk bytenr 13631488 nr 262144 [17608.317133] extent data offset 0 nr 4096 ram 262144 [17608.318691] item 46 key (263 108 8192) itemoff 13353 itemsize 53 [17608.320417] extent data disk bytenr 13631488 nr 262144 [17608.321851] extent data offset 8192 nr 81920 ram 262144 [17608.323351] item 47 key (263 108 90112) itemoff 13300 itemsize 53 [17608.325222] extent data disk bytenr 14319616 nr 4096 [17608.326968] extent data offset 0 nr 4096 ram 4096 [17608.328583] item 48 key (263 108 94208) itemoff 13247 itemsize 53 [17608.330364] extent data disk bytenr 0 nr 0 [17608.331708] extent data offset 0 nr 45056 ram 45056 [17608.333200] item 49 key (263 108 139264) itemoff 13194 itemsize 53 [17608.334979] extent data disk bytenr 14323712 nr 4096 [17608.336570] extent data offset 0 nr 4096 ram 4096 [17608.338023] item 50 key (263 108 143360) itemoff 13141 itemsize 53 [17608.340091] extent data disk bytenr 13631488 nr 262144 [17608.341746] extent data offset 143360 nr 8192 ram 262144 [17608.343400] item 51 key (263 108 151552) itemoff 13088 itemsize 53 [17608.345228] extent data disk bytenr 14327808 nr 61440 [17608.346924] extent data offset 0 nr 61440 ram 61440 [17608.348626] item 52 key (263 108 212992) itemoff 13035 itemsize 53 [17608.350371] extent data disk bytenr 14270464 nr 49152 [17608.351985] extent data offset 0 nr 49152 ram 49152 [17608.353542] item 53 key (264 1 0) itemoff 12875 itemsize 160 [17608.355125] inode generation 6 size 247974 mode 100600 [17608.356813] item 54 key (264 12 256) itemoff 12856 itemsize 19 [17608.358539] item 55 key (264 108 0) itemoff 12803 itemsize 53 [17608.360340] extent data disk bytenr 14471168 nr 4096 [17608.361935] extent data offset 0 nr 4096 ram 4096 [17608.363426] item 56 key (264 108 4096) itemoff 12750 itemsize 53 [17608.365224] extent data disk bytenr 13631488 nr 262144 [17608.366836] extent data offset 0 nr 4096 ram 262144 [17608.368439] item 57 key (264 108 8192) itemoff 12697 itemsize 53 [17608.370226] extent data disk bytenr 13631488 nr 262144 [17608.371714] extent data offset 8192 nr 8192 ram 262144 [17608.373157] item 58 key (264 108 16384) itemoff 12644 itemsize 53 [17608.374820] extent data disk bytenr 14483456 nr 4096 [17608.376495] extent data offset 0 nr 4096 ram 4096 [17608.378014] item 59 key (264 108 20480) itemoff 12591 itemsize 53 [17608.379882] extent data disk bytenr 0 nr 0 [17608.381425] extent data offset 0 nr 20480 ram 20480 [17608.383188] item 60 key (264 108 40960) itemoff 12538 itemsize 53 [17608.385037] extent data disk bytenr 14487552 nr 4096 [17608.386644] extent data offset 0 nr 4096 ram 4096 [17608.388191] item 61 key (264 108 45056) itemoff 12485 itemsize 53 [17608.390035] extent data disk bytenr 13631488 nr 262144 [17608.391634] extent data offset 45056 nr 20480 ram 262144 [17608.393440] item 62 key (264 108 65536) itemoff 12432 itemsize 53 [17608.395464] extent data disk bytenr 14475264 nr 4096 [17608.397179] extent data offset 0 nr 4096 ram 4096 [17608.398675] item 63 key (264 108 69632) itemoff 12379 itemsize 53 [17608.400412] extent data disk bytenr 0 nr 0 [17608.401843] extent data offset 0 nr 28672 ram 49152 [17608.403409] item 64 key (264 108 98304) itemoff 12326 itemsize 53 [17608.405389] extent data disk bytenr 14491648 nr 20480 [17608.407111] extent data offset 0 nr 8192 ram 20480 [17608.408795] item 65 key (264 108 106496) itemoff 12273 itemsize 53 [17608.410757] extent data disk bytenr 14561280 nr 53248 [17608.412483] extent data offset 0 nr 53248 ram 53248 [17608.414166] item 66 key (264 108 159744) itemoff 12220 itemsize 53 [17608.416164] extent data disk bytenr 13631488 nr 262144 [17608.417897] extent data offset 159744 nr 16384 ram 262144 [17608.419677] item 67 key (264 108 176128) itemoff 12167 itemsize 53 [17608.421598] extent data disk bytenr 13631488 nr 262144 [17608.423313] extent data offset 237568 nr 4096 ram 262144 [17608.425051] item 68 key (264 108 180224) itemoff 12114 itemsize 53 [17608.427009] extent data disk bytenr 13631488 nr 262144 [17608.428787] extent data offset 180224 nr 45056 ram 262144 [17608.430588] item 69 key (264 108 225280) itemoff 12061 itemsize 53 [17608.432457] extent data disk bytenr 14614528 nr 16384 [17608.434122] extent data offset 0 nr 16384 ram 16384 [17608.435663] item 70 key (264 108 241664) itemoff 12008 itemsize 53 [17608.437520] extent data disk bytenr 13631488 nr 262144 [17608.439160] extent data offset 241664 nr 8192 ram 262144 [17608.440869] item 71 key (265 1 0) itemoff 11848 itemsize 160 [17608.442586] inode generation 6 size 193781 mode 100600 [17608.444197] item 72 key (265 12 256) itemoff 11829 itemsize 19 [17608.446043] item 73 key (265 108 0) itemoff 11776 itemsize 53 [17608.447726] extent data disk bytenr 14524416 nr 4096 [17608.449507] extent data offset 0 nr 4096 ram 4096 [17608.451142] item 74 key (265 108 4096) itemoff 11723 itemsize 53 [17608.452883] extent data disk bytenr 13631488 nr 262144 [17608.454517] extent data offset 0 nr 4096 ram 262144 [17608.456136] item 75 key (265 108 8192) itemoff 11670 itemsize 53 [17608.458065] extent data disk bytenr 13631488 nr 262144 [17608.459732] extent data offset 8192 nr 110592 ram 262144 [17608.461260] item 76 key (265 108 118784) itemoff 11617 itemsize 53 [17608.462918] extent data disk bytenr 14528512 nr 28672 [17608.464562] extent data offset 0 nr 28672 ram 28672 [17608.466204] item 77 key (265 108 147456) itemoff 11564 itemsize 53 [17608.468161] extent data disk bytenr 0 nr 0 [17608.469631] extent data offset 8192 nr 4096 ram 12288 [17608.471331] item 78 key (265 108 151552) itemoff 11511 itemsize 53 [17608.473363] extent data disk bytenr 14557184 nr 4096 [17608.474920] extent data offset 0 nr 4096 ram 4096 [17608.476499] item 79 key (265 108 155648) itemoff 11458 itemsize 53 [17608.478306] extent data disk bytenr 13631488 nr 262144 [17608.479921] extent data offset 155648 nr 40960 ram 262144 [17608.481652] item 80 key (266 1 0) itemoff 11298 itemsize 160 [17608.483268] inode generation 6 size 262144 mode 100600 [17608.484849] item 81 key (266 12 256) itemoff 11279 itemsize 19 [17608.486489] item 82 key (266 108 0) itemoff 11226 itemsize 53 [17608.488298] extent data disk bytenr 14712832 nr 4096 [17608.489866] extent data offset 0 nr 4096 ram 4096 [17608.491399] item 83 key (266 108 4096) itemoff 11173 itemsize 53 [17608.493255] extent data disk bytenr 13631488 nr 262144 [17608.494898] extent data offset 0 nr 4096 ram 262144 [17608.496504] item 84 key (266 108 8192) itemoff 11120 itemsize 53 [17608.498434] extent data disk bytenr 13631488 nr 262144 [17608.500033] extent data offset 8192 nr 73728 ram 262144 [17608.501619] item 85 key (266 108 81920) itemoff 11067 itemsize 53 [17608.503455] extent data disk bytenr 14741504 nr 4096 [17608.505086] extent data offset 0 nr 4096 ram 4096 [17608.506648] item 86 key (266 108 86016) itemoff 11014 itemsize 53 [17608.508346] extent data disk bytenr 13631488 nr 262144 [17608.509862] extent data offset 86016 nr 28672 ram 262144 [17608.511645] item 87 key (266 108 114688) itemoff 10961 itemsize 53 [17608.513620] extent data disk bytenr 13631488 nr 262144 [17608.515403] extent data offset 36864 nr 28672 ram 262144 [17608.517234] item 88 key (266 108 143360) itemoff 10908 itemsize 53 [17608.519248] extent data disk bytenr 13631488 nr 262144 [17608.520976] extent data offset 143360 nr 16384 ram 262144 [17608.522635] item 89 key (266 108 159744) itemoff 10855 itemsize 53 [17608.524583] extent data disk bytenr 14745600 nr 12288 [17608.526254] extent data offset 0 nr 12288 ram 12288 [17608.527858] item 90 key (266 108 172032) itemoff 10802 itemsize 53 [17608.538634] extent data disk bytenr 13631488 nr 262144 [17608.540392] extent data offset 172032 nr 36864 ram 262144 [17608.542156] item 91 key (266 108 208896) itemoff 10749 itemsize 53 [17608.544104] extent data disk bytenr 14716928 nr 4096 [17608.545664] extent data offset 0 nr 4096 ram 4096 [17608.547207] item 92 key (266 108 212992) itemoff 10696 itemsize 53 [17608.549038] extent data disk bytenr 0 nr 0 [17608.550440] extent data offset 0 nr 32768 ram 32768 [17608.552056] item 93 key (266 108 245760) itemoff 10643 itemsize 53 [17608.553997] extent data disk bytenr 14721024 nr 4096 [17608.555578] extent data offset 0 nr 4096 ram 4096 [17608.557113] item 94 key (266 108 249856) itemoff 10590 itemsize 53 [17608.558861] extent data disk bytenr 14688256 nr 24576 [17608.560439] extent data offset 12288 nr 12288 ram 24576 [17608.562137] item 95 key (267 1 0) itemoff 10430 itemsize 160 [17608.563756] inode generation 6 size 262144 mode 100600 [17608.565261] item 96 key (267 12 256) itemoff 10411 itemsize 19 [17608.566912] item 97 key (267 108 0) itemoff 10358 itemsize 53 [17608.568715] extent data disk bytenr 14737408 nr 4096 [17608.570283] extent data offset 0 nr 4096 ram 4096 [17608.571810] item 98 key (267 108 4096) itemoff 10305 itemsize 53 [17608.573502] extent data disk bytenr 13631488 nr 262144 [17608.575033] extent data offset 0 nr 4096 ram 262144 [17608.576507] item 99 key (267 108 8192) itemoff 10252 itemsize 53 [17608.578416] extent data disk bytenr 13631488 nr 262144 [17608.581054] extent data offset 8192 nr 167936 ram 262144 [17608.583613] item 100 key (267 108 176128) itemoff 10199 itemsize 53 [17608.585846] extent data disk bytenr 13631488 nr 262144 [17608.587883] extent data offset 188416 nr 4096 ram 262144 [17608.590034] item 101 key (267 108 180224) itemoff 10146 itemsize 53 [17608.592397] extent data disk bytenr 13631488 nr 262144 [17608.594415] extent data offset 180224 nr 81920 ram 262144 [17608.595941] item 102 key (18446744073709551606 128 13631488) itemoff 10142 itemsize 4 [17608.598465] item 103 key (18446744073709551606 128 13635584) itemoff 10082 itemsize 60 [17608.601262] item 104 key (18446744073709551606 128 13697024) itemoff 9974 itemsize 108 [17608.604090] item 105 key (18446744073709551606 128 13717504) itemoff 9946 itemsize 28 [17608.606573] item 106 key (18446744073709551606 128 13774848) itemoff 9930 itemsize 16 [17608.609125] item 107 key (18446744073709551606 128 13803520) itemoff 9842 itemsize 88 [17608.611709] item 108 key (18446744073709551606 128 13905920) itemoff 9786 itemsize 56 [17608.614269] item 109 key (18446744073709551606 128 13975552) itemoff 9782 itemsize 4 [17608.616992] item 110 key (18446744073709551606 128 14123008) itemoff 9778 itemsize 4 [17608.619523] item 111 key (18446744073709551606 128 14135296) itemoff 9710 itemsize 68 [17608.622144] item 112 key (18446744073709551606 128 14209024) itemoff 9706 itemsize 4 [17608.624841] item 113 key (18446744073709551606 128 14217216) itemoff 9538 itemsize 168 [17608.627409] item 114 key (18446744073709551606 128 14471168) itemoff 9530 itemsize 8 [17608.630034] item 115 key (18446744073709551606 128 14479360) itemoff 9510 itemsize 20 [17608.632681] item 116 key (18446744073709551606 128 14499840) itemoff 9498 itemsize 12 [17608.635448] item 117 key (18446744073709551606 128 14524416) itemoff 9394 itemsize 104 [17608.638122] item 118 key (18446744073709551606 128 14700544) itemoff 9382 itemsize 12 [17608.640633] item 119 key (18446744073709551606 128 14712832) itemoff 9370 itemsize 12 [17608.643227] item 120 key (18446744073709551606 128 14737408) itemoff 9350 itemsize 20 [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected [17608.648714] ------------[ cut here ]------------ [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] [17608.652972] Modules linked in: dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_writes dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [last unloaded: scsi_debug] [17608.660683] CPU: 2 PID: 27162 Comm: fsx Tainted: G L 5.7.0-rc3-default+ #1094 [17608.663520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [17608.667102] RIP: 0010:btree_csum_one_bio+0x297/0x2a0 [btrfs] [17608.668919] Code: bc fd ff ff e8 aa e7 f0 c2 31 f6 48 89 3c 24 e8 ff 7b ff ff 48 8b 3c 24 48 c7 c6 78 75 25 c0 48 8b 17 4c 89 e7 e8 84 bc 0b 00 <0f> 0b e9 8f fd ff ff 66 90 0f 1f 44 00 00 48 89 f7 e9 53 fd ff ff [17608.674610] RSP: 0018:ffffafbb81e978b8 EFLAGS: 00010292 [17608.676299] RAX: 0000000000000000 RBX: ffff94287f810f00 RCX: 0000000000000001 [17608.678260] RDX: 0000000000000000 RSI: ffffffff83109fc2 RDI: ffffffff8310a096 [17608.680484] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [17608.682645] R10: 0000000000000000 R11: 0000000000008ca0 R12: ffff94285f298000 [17608.684681] R13: ffff94274100bd30 R14: 0000000000000000 R15: 00000000ffffff8b [17608.686659] FS: 00007f3479617b80(0000) GS:ffff94287ba00000(0000) knlGS:0000000000000000 [17608.689240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17608.691150] CR2: 0000000000466000 CR3: 000000010227f001 CR4: 0000000000160ee0 [17608.693354] Call Trace: [17608.694341] btree_submit_bio_hook+0x74/0xc0 [btrfs] [17608.696162] submit_one_bio+0x2b/0x40 [btrfs] [17608.698391] btree_write_cache_pages+0x373/0x440 [btrfs] [17608.700085] ? lock_acquire+0xa3/0x400 [17608.701257] ? __filemap_fdatawrite_range+0xad/0x110 [17608.702736] do_writepages+0x40/0xe0 [17608.703953] ? do_raw_spin_unlock+0x4b/0xc0 [17608.705290] ? _raw_spin_unlock+0x1f/0x30 [17608.706571] ? wbc_attach_and_unlock_inode+0x194/0x2a0 [17608.708167] __filemap_fdatawrite_range+0xce/0x110 [17608.709679] btrfs_write_marked_extents+0x68/0x160 [btrfs] [17608.711384] btrfs_sync_log+0x1a8/0xd40 [btrfs] [17608.712892] ? btrfs_log_inode+0x370/0xe20 [btrfs] [17608.714459] ? lock_acquire+0xa3/0x400 [17608.715803] ? lockref_put_or_lock+0x9/0x30 [17608.717115] ? dput+0x20/0x490 [17608.718172] ? dput+0x20/0x490 [17608.719271] ? do_raw_spin_unlock+0x4b/0xc0 [17608.720639] ? _raw_spin_unlock+0x1f/0x30 [17608.722002] btrfs_sync_file+0x335/0x490 [btrfs] [17608.723610] __x64_sys_msync+0x19e/0x200 [17608.724924] do_syscall_64+0x50/0x210 [17608.726143] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [17608.727719] RIP: 0033:0x7f34797f2743 [17608.729030] Code: 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 1a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48 [17608.734572] RSP: 002b:00007ffe84f11858 EFLAGS: 00000246 ORIG_RAX: 000000000000001a [17608.736913] RAX: ffffffffffffffda RBX: 00000000000008c8 RCX: 00007f34797f2743 [17608.738902] RDX: 0000000000000004 RSI: 000000000000293f RDI: 00007f3479814000 [17608.740949] RBP: 00000000000278c8 R08: 000000000000001f R09: 00007f3479814900 [17608.742896] R10: fffffffffffffd0c R11: 0000000000000246 R12: 0000000000002077 [17608.744810] R13: 000000000000293f R14: 00007f3479814000 R15: 0000000000000000 [17608.746764] irq event stamp: 0 [17608.747948] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [17608.749754] hardirqs last disabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 [17608.752428] softirqs last enabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 [17608.754935] softirqs last disabled at (0): [<0000000000000000>] 0x0 [17608.756871] ---[ end trace ddba49bd36651725 ]--- [17608.758590] BTRFS: error (device dm-0) in btrfs_sync_log:3089: errno=-5 IO failure ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: write-time leaf corruption detected 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba @ 2020-05-12 14:26 ` Filipe Manana 2020-05-13 3:10 ` Qu Wenruo 2020-05-19 14:26 ` Bug 5.7-rc: write-time leaf corruption detected (fixed) David Sterba 2 siblings, 0 replies; 24+ messages in thread From: Filipe Manana @ 2020-05-12 14:26 UTC (permalink / raw) To: David Sterba; +Cc: linux-btrfs On Tue, May 12, 2020 at 3:19 PM David Sterba <dsterba@suse.com> wrote: > > Happened once in a VM, defaul mkfs/mount options. > > generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 > [17607.625840] BTRFS: device fsid ae51523a-71e7-4730-9097-faeca36779be devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (27036) > [17607.664903] BTRFS info (device vdb): disk space caching is enabled > [17607.667422] BTRFS info (device vdb): has skinny extents > [17607.669552] BTRFS info (device vdb): flagging fs with big metadata feature > [17607.681613] BTRFS info (device vdb): checking UUID tree > [17607.921588] BTRFS: device fsid 16238d8a-1af3-4fa1-b3bc-fd6f04469192 devid 1 transid 5 /dev/mapper/logwrites-test scanned by mkfs.btrfs (27101) > [17607.948085] BTRFS info (device dm-0): turning on sync discard > [17607.952647] BTRFS info (device dm-0): disk space caching is enabled > [17607.955088] BTRFS info (device dm-0): has skinny extents > [17607.956924] BTRFS info (device dm-0): flagging fs with big metadata feature > [17607.966128] BTRFS info (device dm-0): checking UUID tree > [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item > [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 > [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 > [17608.106426] item 0 key (257 1 0) itemoff 16123 itemsize 160 > [17608.108766] inode generation 6 size 262144 mode 100600 > [17608.110947] item 1 key (257 12 256) itemoff 16103 itemsize 20 > [17608.112698] item 2 key (257 108 0) itemoff 16050 itemsize 53 > [17608.114568] extent data disk bytenr 13631488 nr 262144 > [17608.116352] extent data offset 0 nr 262144 ram 262144 > [17608.118082] item 3 key (258 1 0) itemoff 15890 itemsize 160 > [17608.119920] inode generation 6 size 262144 mode 100600 > [17608.121605] item 4 key (258 12 256) itemoff 15871 itemsize 19 > [17608.123293] item 5 key (258 108 0) itemoff 15818 itemsize 53 > [17608.125212] extent data disk bytenr 13905920 nr 4096 > [17608.126851] extent data offset 0 nr 4096 ram 4096 > [17608.128342] item 6 key (258 108 4096) itemoff 15765 itemsize 53 > [17608.130106] extent data disk bytenr 13631488 nr 262144 > [17608.131774] extent data offset 0 nr 4096 ram 262144 > [17608.133330] item 7 key (258 108 8192) itemoff 15712 itemsize 53 > [17608.135151] extent data disk bytenr 13631488 nr 262144 > [17608.136680] extent data offset 8192 nr 8192 ram 262144 > [17608.137931] item 8 key (258 108 16384) itemoff 15659 itemsize 53 > [17608.139257] extent data disk bytenr 13631488 nr 262144 > [17608.140939] extent data offset 204800 nr 16384 ram 262144 > [17608.142255] item 9 key (258 108 32768) itemoff 15606 itemsize 53 > [17608.143941] extent data disk bytenr 13910016 nr 28672 > [17608.145576] extent data offset 0 nr 28672 ram 28672 > [17608.147165] item 10 key (258 108 61440) itemoff 15553 itemsize 53 > [17608.148964] extent data disk bytenr 13631488 nr 262144 > [17608.150563] extent data offset 249856 nr 8192 ram 262144 > [17608.152204] item 11 key (258 108 69632) itemoff 15500 itemsize 53 > [17608.153974] extent data disk bytenr 13631488 nr 262144 > [17608.155588] extent data offset 69632 nr 126976 ram 262144 > [17608.157099] item 12 key (258 108 196608) itemoff 15447 itemsize 53 > [17608.158698] extent data disk bytenr 13938688 nr 4096 > [17608.160123] extent data offset 0 nr 4096 ram 4096 > [17608.161300] item 13 key (258 108 200704) itemoff 15394 itemsize 53 > [17608.162597] extent data disk bytenr 0 nr 0 > [17608.164057] extent data offset 0 nr 28672 ram 28672 > [17608.165782] item 14 key (258 108 229376) itemoff 15341 itemsize 53 > [17608.167691] extent data disk bytenr 13942784 nr 4096 > [17608.169259] extent data offset 0 nr 4096 ram 4096 > [17608.170821] item 15 key (258 108 233472) itemoff 15288 itemsize 53 > [17608.172687] extent data disk bytenr 13631488 nr 262144 > [17608.174310] extent data offset 233472 nr 12288 ram 262144 > [17608.175993] item 16 key (258 108 245760) itemoff 15235 itemsize 53 > [17608.177821] extent data disk bytenr 13946880 nr 16384 > [17608.179440] extent data offset 0 nr 16384 ram 16384 > [17608.181043] item 17 key (259 1 0) itemoff 15075 itemsize 160 > [17608.182796] inode generation 6 size 262144 mode 100600 > [17608.184424] item 18 key (259 12 256) itemoff 15056 itemsize 19 > [17608.186085] item 19 key (259 108 0) itemoff 15003 itemsize 53 > [17608.187859] extent data disk bytenr 13975552 nr 4096 > [17608.189525] extent data offset 0 nr 4096 ram 4096 > [17608.191074] item 20 key (259 108 4096) itemoff 14950 itemsize 53 > [17608.192816] extent data disk bytenr 13631488 nr 262144 > [17608.194464] extent data offset 0 nr 4096 ram 262144 > [17608.196206] item 21 key (259 108 8192) itemoff 14897 itemsize 53 > [17608.198212] extent data disk bytenr 13631488 nr 262144 > [17608.200048] extent data offset 8192 nr 106496 ram 262144 > [17608.201684] item 22 key (259 108 114688) itemoff 14844 itemsize 53 > [17608.203659] extent data disk bytenr 14217216 nr 49152 > [17608.205398] extent data offset 0 nr 49152 ram 49152 > [17608.207126] item 23 key (259 108 163840) itemoff 14791 itemsize 53 > [17608.209019] extent data disk bytenr 13631488 nr 262144 > [17608.210736] extent data offset 163840 nr 98304 ram 262144 > [17608.212371] item 24 key (261 1 0) itemoff 14631 itemsize 160 > [17608.222944] inode generation 6 size 262144 mode 100600 > [17608.224637] item 25 key (261 12 256) itemoff 14612 itemsize 19 > [17608.226400] item 26 key (261 108 0) itemoff 14559 itemsize 53 > [17608.228169] extent data disk bytenr 14123008 nr 4096 > [17608.229794] extent data offset 0 nr 4096 ram 4096 > [17608.231292] item 27 key (261 108 4096) itemoff 14506 itemsize 53 > [17608.233203] extent data disk bytenr 13631488 nr 262144 > [17608.234794] extent data offset 0 nr 4096 ram 262144 > [17608.236362] item 28 key (261 108 8192) itemoff 14453 itemsize 53 > [17608.238102] extent data disk bytenr 13631488 nr 262144 > [17608.239782] extent data offset 8192 nr 16384 ram 262144 > [17608.241587] item 29 key (261 108 24576) itemoff 14400 itemsize 53 > [17608.243431] extent data disk bytenr 13631488 nr 262144 > [17608.245176] extent data offset 212992 nr 36864 ram 262144 > [17608.246991] item 30 key (261 108 61440) itemoff 14347 itemsize 53 > [17608.248868] extent data disk bytenr 13631488 nr 262144 > [17608.250456] extent data offset 61440 nr 114688 ram 262144 > [17608.252333] item 31 key (261 108 176128) itemoff 14294 itemsize 53 > [17608.254343] extent data disk bytenr 14135296 nr 61440 > [17608.256131] extent data offset 0 nr 61440 ram 61440 > [17608.257857] item 32 key (261 108 237568) itemoff 14241 itemsize 53 > [17608.259864] extent data disk bytenr 13631488 nr 262144 > [17608.261484] extent data offset 237568 nr 24576 ram 262144 > [17608.263134] item 33 key (262 1 0) itemoff 14081 itemsize 160 > [17608.264865] inode generation 6 size 178311 mode 100600 > [17608.266625] item 34 key (262 12 256) itemoff 14062 itemsize 19 > [17608.268551] item 35 key (262 108 0) itemoff 14009 itemsize 53 > [17608.270363] extent data disk bytenr 14196736 nr 4096 > [17608.272019] extent data offset 0 nr 4096 ram 4096 > [17608.273505] item 36 key (262 108 4096) itemoff 13956 itemsize 53 > [17608.275387] extent data disk bytenr 13631488 nr 262144 > [17608.277125] extent data offset 0 nr 4096 ram 262144 > [17608.278780] item 37 key (262 108 8192) itemoff 13903 itemsize 53 > [17608.280507] extent data disk bytenr 13631488 nr 262144 > [17608.282163] extent data offset 8192 nr 106496 ram 262144 > [17608.283866] item 38 key (262 108 114688) itemoff 13850 itemsize 53 > [17608.285727] extent data disk bytenr 14200832 nr 4096 > [17608.287264] extent data offset 0 nr 4096 ram 4096 > [17608.288719] item 39 key (262 108 118784) itemoff 13797 itemsize 53 > [17608.290696] extent data disk bytenr 0 nr 0 > [17608.292237] extent data offset 0 nr 49152 ram 49152 > [17608.293667] item 40 key (262 108 167936) itemoff 13744 itemsize 53 > [17608.295436] extent data disk bytenr 14209024 nr 4096 > [17608.297051] extent data offset 0 nr 4096 ram 4096 > [17608.298583] item 41 key (262 108 172032) itemoff 13691 itemsize 53 > [17608.300573] extent data disk bytenr 13631488 nr 262144 > [17608.302323] extent data offset 172032 nr 8192 ram 262144 > [17608.304112] item 42 key (263 1 0) itemoff 13531 itemsize 160 > [17608.305810] inode generation 6 size 262144 mode 100600 > [17608.307444] item 43 key (263 12 256) itemoff 13512 itemsize 19 > [17608.309108] item 44 key (263 108 0) itemoff 13459 itemsize 53 > [17608.310760] extent data disk bytenr 14266368 nr 4096 > [17608.312269] extent data offset 0 nr 4096 ram 4096 > [17608.313883] item 45 key (263 108 4096) itemoff 13406 itemsize 53 > [17608.315659] extent data disk bytenr 13631488 nr 262144 > [17608.317133] extent data offset 0 nr 4096 ram 262144 > [17608.318691] item 46 key (263 108 8192) itemoff 13353 itemsize 53 > [17608.320417] extent data disk bytenr 13631488 nr 262144 > [17608.321851] extent data offset 8192 nr 81920 ram 262144 > [17608.323351] item 47 key (263 108 90112) itemoff 13300 itemsize 53 > [17608.325222] extent data disk bytenr 14319616 nr 4096 > [17608.326968] extent data offset 0 nr 4096 ram 4096 > [17608.328583] item 48 key (263 108 94208) itemoff 13247 itemsize 53 > [17608.330364] extent data disk bytenr 0 nr 0 > [17608.331708] extent data offset 0 nr 45056 ram 45056 > [17608.333200] item 49 key (263 108 139264) itemoff 13194 itemsize 53 > [17608.334979] extent data disk bytenr 14323712 nr 4096 > [17608.336570] extent data offset 0 nr 4096 ram 4096 > [17608.338023] item 50 key (263 108 143360) itemoff 13141 itemsize 53 > [17608.340091] extent data disk bytenr 13631488 nr 262144 > [17608.341746] extent data offset 143360 nr 8192 ram 262144 > [17608.343400] item 51 key (263 108 151552) itemoff 13088 itemsize 53 > [17608.345228] extent data disk bytenr 14327808 nr 61440 > [17608.346924] extent data offset 0 nr 61440 ram 61440 > [17608.348626] item 52 key (263 108 212992) itemoff 13035 itemsize 53 > [17608.350371] extent data disk bytenr 14270464 nr 49152 > [17608.351985] extent data offset 0 nr 49152 ram 49152 > [17608.353542] item 53 key (264 1 0) itemoff 12875 itemsize 160 > [17608.355125] inode generation 6 size 247974 mode 100600 > [17608.356813] item 54 key (264 12 256) itemoff 12856 itemsize 19 > [17608.358539] item 55 key (264 108 0) itemoff 12803 itemsize 53 > [17608.360340] extent data disk bytenr 14471168 nr 4096 > [17608.361935] extent data offset 0 nr 4096 ram 4096 > [17608.363426] item 56 key (264 108 4096) itemoff 12750 itemsize 53 > [17608.365224] extent data disk bytenr 13631488 nr 262144 > [17608.366836] extent data offset 0 nr 4096 ram 262144 > [17608.368439] item 57 key (264 108 8192) itemoff 12697 itemsize 53 > [17608.370226] extent data disk bytenr 13631488 nr 262144 > [17608.371714] extent data offset 8192 nr 8192 ram 262144 > [17608.373157] item 58 key (264 108 16384) itemoff 12644 itemsize 53 > [17608.374820] extent data disk bytenr 14483456 nr 4096 > [17608.376495] extent data offset 0 nr 4096 ram 4096 > [17608.378014] item 59 key (264 108 20480) itemoff 12591 itemsize 53 > [17608.379882] extent data disk bytenr 0 nr 0 > [17608.381425] extent data offset 0 nr 20480 ram 20480 > [17608.383188] item 60 key (264 108 40960) itemoff 12538 itemsize 53 > [17608.385037] extent data disk bytenr 14487552 nr 4096 > [17608.386644] extent data offset 0 nr 4096 ram 4096 > [17608.388191] item 61 key (264 108 45056) itemoff 12485 itemsize 53 > [17608.390035] extent data disk bytenr 13631488 nr 262144 > [17608.391634] extent data offset 45056 nr 20480 ram 262144 > [17608.393440] item 62 key (264 108 65536) itemoff 12432 itemsize 53 > [17608.395464] extent data disk bytenr 14475264 nr 4096 > [17608.397179] extent data offset 0 nr 4096 ram 4096 > [17608.398675] item 63 key (264 108 69632) itemoff 12379 itemsize 53 > [17608.400412] extent data disk bytenr 0 nr 0 > [17608.401843] extent data offset 0 nr 28672 ram 49152 > [17608.403409] item 64 key (264 108 98304) itemoff 12326 itemsize 53 > [17608.405389] extent data disk bytenr 14491648 nr 20480 > [17608.407111] extent data offset 0 nr 8192 ram 20480 > [17608.408795] item 65 key (264 108 106496) itemoff 12273 itemsize 53 > [17608.410757] extent data disk bytenr 14561280 nr 53248 > [17608.412483] extent data offset 0 nr 53248 ram 53248 > [17608.414166] item 66 key (264 108 159744) itemoff 12220 itemsize 53 > [17608.416164] extent data disk bytenr 13631488 nr 262144 > [17608.417897] extent data offset 159744 nr 16384 ram 262144 > [17608.419677] item 67 key (264 108 176128) itemoff 12167 itemsize 53 > [17608.421598] extent data disk bytenr 13631488 nr 262144 > [17608.423313] extent data offset 237568 nr 4096 ram 262144 > [17608.425051] item 68 key (264 108 180224) itemoff 12114 itemsize 53 > [17608.427009] extent data disk bytenr 13631488 nr 262144 > [17608.428787] extent data offset 180224 nr 45056 ram 262144 > [17608.430588] item 69 key (264 108 225280) itemoff 12061 itemsize 53 > [17608.432457] extent data disk bytenr 14614528 nr 16384 > [17608.434122] extent data offset 0 nr 16384 ram 16384 > [17608.435663] item 70 key (264 108 241664) itemoff 12008 itemsize 53 > [17608.437520] extent data disk bytenr 13631488 nr 262144 > [17608.439160] extent data offset 241664 nr 8192 ram 262144 > [17608.440869] item 71 key (265 1 0) itemoff 11848 itemsize 160 > [17608.442586] inode generation 6 size 193781 mode 100600 > [17608.444197] item 72 key (265 12 256) itemoff 11829 itemsize 19 > [17608.446043] item 73 key (265 108 0) itemoff 11776 itemsize 53 > [17608.447726] extent data disk bytenr 14524416 nr 4096 > [17608.449507] extent data offset 0 nr 4096 ram 4096 > [17608.451142] item 74 key (265 108 4096) itemoff 11723 itemsize 53 > [17608.452883] extent data disk bytenr 13631488 nr 262144 > [17608.454517] extent data offset 0 nr 4096 ram 262144 > [17608.456136] item 75 key (265 108 8192) itemoff 11670 itemsize 53 > [17608.458065] extent data disk bytenr 13631488 nr 262144 > [17608.459732] extent data offset 8192 nr 110592 ram 262144 > [17608.461260] item 76 key (265 108 118784) itemoff 11617 itemsize 53 > [17608.462918] extent data disk bytenr 14528512 nr 28672 > [17608.464562] extent data offset 0 nr 28672 ram 28672 > [17608.466204] item 77 key (265 108 147456) itemoff 11564 itemsize 53 > [17608.468161] extent data disk bytenr 0 nr 0 > [17608.469631] extent data offset 8192 nr 4096 ram 12288 > [17608.471331] item 78 key (265 108 151552) itemoff 11511 itemsize 53 > [17608.473363] extent data disk bytenr 14557184 nr 4096 > [17608.474920] extent data offset 0 nr 4096 ram 4096 > [17608.476499] item 79 key (265 108 155648) itemoff 11458 itemsize 53 > [17608.478306] extent data disk bytenr 13631488 nr 262144 > [17608.479921] extent data offset 155648 nr 40960 ram 262144 > [17608.481652] item 80 key (266 1 0) itemoff 11298 itemsize 160 > [17608.483268] inode generation 6 size 262144 mode 100600 > [17608.484849] item 81 key (266 12 256) itemoff 11279 itemsize 19 > [17608.486489] item 82 key (266 108 0) itemoff 11226 itemsize 53 > [17608.488298] extent data disk bytenr 14712832 nr 4096 > [17608.489866] extent data offset 0 nr 4096 ram 4096 > [17608.491399] item 83 key (266 108 4096) itemoff 11173 itemsize 53 > [17608.493255] extent data disk bytenr 13631488 nr 262144 > [17608.494898] extent data offset 0 nr 4096 ram 262144 > [17608.496504] item 84 key (266 108 8192) itemoff 11120 itemsize 53 > [17608.498434] extent data disk bytenr 13631488 nr 262144 > [17608.500033] extent data offset 8192 nr 73728 ram 262144 > [17608.501619] item 85 key (266 108 81920) itemoff 11067 itemsize 53 > [17608.503455] extent data disk bytenr 14741504 nr 4096 > [17608.505086] extent data offset 0 nr 4096 ram 4096 > [17608.506648] item 86 key (266 108 86016) itemoff 11014 itemsize 53 > [17608.508346] extent data disk bytenr 13631488 nr 262144 > [17608.509862] extent data offset 86016 nr 28672 ram 262144 > [17608.511645] item 87 key (266 108 114688) itemoff 10961 itemsize 53 > [17608.513620] extent data disk bytenr 13631488 nr 262144 > [17608.515403] extent data offset 36864 nr 28672 ram 262144 > [17608.517234] item 88 key (266 108 143360) itemoff 10908 itemsize 53 > [17608.519248] extent data disk bytenr 13631488 nr 262144 > [17608.520976] extent data offset 143360 nr 16384 ram 262144 > [17608.522635] item 89 key (266 108 159744) itemoff 10855 itemsize 53 > [17608.524583] extent data disk bytenr 14745600 nr 12288 > [17608.526254] extent data offset 0 nr 12288 ram 12288 > [17608.527858] item 90 key (266 108 172032) itemoff 10802 itemsize 53 > [17608.538634] extent data disk bytenr 13631488 nr 262144 > [17608.540392] extent data offset 172032 nr 36864 ram 262144 > [17608.542156] item 91 key (266 108 208896) itemoff 10749 itemsize 53 > [17608.544104] extent data disk bytenr 14716928 nr 4096 > [17608.545664] extent data offset 0 nr 4096 ram 4096 > [17608.547207] item 92 key (266 108 212992) itemoff 10696 itemsize 53 > [17608.549038] extent data disk bytenr 0 nr 0 > [17608.550440] extent data offset 0 nr 32768 ram 32768 > [17608.552056] item 93 key (266 108 245760) itemoff 10643 itemsize 53 > [17608.553997] extent data disk bytenr 14721024 nr 4096 > [17608.555578] extent data offset 0 nr 4096 ram 4096 > [17608.557113] item 94 key (266 108 249856) itemoff 10590 itemsize 53 > [17608.558861] extent data disk bytenr 14688256 nr 24576 > [17608.560439] extent data offset 12288 nr 12288 ram 24576 > [17608.562137] item 95 key (267 1 0) itemoff 10430 itemsize 160 > [17608.563756] inode generation 6 size 262144 mode 100600 > [17608.565261] item 96 key (267 12 256) itemoff 10411 itemsize 19 > [17608.566912] item 97 key (267 108 0) itemoff 10358 itemsize 53 > [17608.568715] extent data disk bytenr 14737408 nr 4096 > [17608.570283] extent data offset 0 nr 4096 ram 4096 > [17608.571810] item 98 key (267 108 4096) itemoff 10305 itemsize 53 > [17608.573502] extent data disk bytenr 13631488 nr 262144 > [17608.575033] extent data offset 0 nr 4096 ram 262144 > [17608.576507] item 99 key (267 108 8192) itemoff 10252 itemsize 53 > [17608.578416] extent data disk bytenr 13631488 nr 262144 > [17608.581054] extent data offset 8192 nr 167936 ram 262144 > [17608.583613] item 100 key (267 108 176128) itemoff 10199 itemsize 53 > [17608.585846] extent data disk bytenr 13631488 nr 262144 > [17608.587883] extent data offset 188416 nr 4096 ram 262144 > [17608.590034] item 101 key (267 108 180224) itemoff 10146 itemsize 53 > [17608.592397] extent data disk bytenr 13631488 nr 262144 > [17608.594415] extent data offset 180224 nr 81920 ram 262144 > [17608.595941] item 102 key (18446744073709551606 128 13631488) itemoff 10142 itemsize 4 > [17608.598465] item 103 key (18446744073709551606 128 13635584) itemoff 10082 itemsize 60 > [17608.601262] item 104 key (18446744073709551606 128 13697024) itemoff 9974 itemsize 108 > [17608.604090] item 105 key (18446744073709551606 128 13717504) itemoff 9946 itemsize 28 > [17608.606573] item 106 key (18446744073709551606 128 13774848) itemoff 9930 itemsize 16 > [17608.609125] item 107 key (18446744073709551606 128 13803520) itemoff 9842 itemsize 88 > [17608.611709] item 108 key (18446744073709551606 128 13905920) itemoff 9786 itemsize 56 > [17608.614269] item 109 key (18446744073709551606 128 13975552) itemoff 9782 itemsize 4 > [17608.616992] item 110 key (18446744073709551606 128 14123008) itemoff 9778 itemsize 4 > [17608.619523] item 111 key (18446744073709551606 128 14135296) itemoff 9710 itemsize 68 > [17608.622144] item 112 key (18446744073709551606 128 14209024) itemoff 9706 itemsize 4 > [17608.624841] item 113 key (18446744073709551606 128 14217216) itemoff 9538 itemsize 168 > [17608.627409] item 114 key (18446744073709551606 128 14471168) itemoff 9530 itemsize 8 > [17608.630034] item 115 key (18446744073709551606 128 14479360) itemoff 9510 itemsize 20 > [17608.632681] item 116 key (18446744073709551606 128 14499840) itemoff 9498 itemsize 12 > [17608.635448] item 117 key (18446744073709551606 128 14524416) itemoff 9394 itemsize 104 > [17608.638122] item 118 key (18446744073709551606 128 14700544) itemoff 9382 itemsize 12 > [17608.640633] item 119 key (18446744073709551606 128 14712832) itemoff 9370 itemsize 12 > [17608.643227] item 120 key (18446744073709551606 128 14737408) itemoff 9350 itemsize 20 > [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected > [17608.648714] ------------[ cut here ]------------ > [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] > [17608.652972] Modules linked in: dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_writes dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [last unloaded: scsi_debug] > [17608.660683] CPU: 2 PID: 27162 Comm: fsx Tainted: G L 5.7.0-rc3-default+ #1094 > [17608.663520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 > [17608.667102] RIP: 0010:btree_csum_one_bio+0x297/0x2a0 [btrfs] > [17608.668919] Code: bc fd ff ff e8 aa e7 f0 c2 31 f6 48 89 3c 24 e8 ff 7b ff ff 48 8b 3c 24 48 c7 c6 78 75 25 c0 48 8b 17 4c 89 e7 e8 84 bc 0b 00 <0f> 0b e9 8f fd ff ff 66 90 0f 1f 44 00 00 48 89 f7 e9 53 fd ff ff > [17608.674610] RSP: 0018:ffffafbb81e978b8 EFLAGS: 00010292 > [17608.676299] RAX: 0000000000000000 RBX: ffff94287f810f00 RCX: 0000000000000001 > [17608.678260] RDX: 0000000000000000 RSI: ffffffff83109fc2 RDI: ffffffff8310a096 > [17608.680484] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 > [17608.682645] R10: 0000000000000000 R11: 0000000000008ca0 R12: ffff94285f298000 > [17608.684681] R13: ffff94274100bd30 R14: 0000000000000000 R15: 00000000ffffff8b > [17608.686659] FS: 00007f3479617b80(0000) GS:ffff94287ba00000(0000) knlGS:0000000000000000 > [17608.689240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [17608.691150] CR2: 0000000000466000 CR3: 000000010227f001 CR4: 0000000000160ee0 > [17608.693354] Call Trace: > [17608.694341] btree_submit_bio_hook+0x74/0xc0 [btrfs] > [17608.696162] submit_one_bio+0x2b/0x40 [btrfs] > [17608.698391] btree_write_cache_pages+0x373/0x440 [btrfs] > [17608.700085] ? lock_acquire+0xa3/0x400 > [17608.701257] ? __filemap_fdatawrite_range+0xad/0x110 > [17608.702736] do_writepages+0x40/0xe0 > [17608.703953] ? do_raw_spin_unlock+0x4b/0xc0 > [17608.705290] ? _raw_spin_unlock+0x1f/0x30 > [17608.706571] ? wbc_attach_and_unlock_inode+0x194/0x2a0 > [17608.708167] __filemap_fdatawrite_range+0xce/0x110 > [17608.709679] btrfs_write_marked_extents+0x68/0x160 [btrfs] > [17608.711384] btrfs_sync_log+0x1a8/0xd40 [btrfs] > [17608.712892] ? btrfs_log_inode+0x370/0xe20 [btrfs] > [17608.714459] ? lock_acquire+0xa3/0x400 > [17608.715803] ? lockref_put_or_lock+0x9/0x30 > [17608.717115] ? dput+0x20/0x490 > [17608.718172] ? dput+0x20/0x490 > [17608.719271] ? do_raw_spin_unlock+0x4b/0xc0 > [17608.720639] ? _raw_spin_unlock+0x1f/0x30 > [17608.722002] btrfs_sync_file+0x335/0x490 [btrfs] > [17608.723610] __x64_sys_msync+0x19e/0x200 > [17608.724924] do_syscall_64+0x50/0x210 > [17608.726143] entry_SYSCALL_64_after_hwframe+0x49/0xb3 > [17608.727719] RIP: 0033:0x7f34797f2743 > [17608.729030] Code: 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 1a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48 > [17608.734572] RSP: 002b:00007ffe84f11858 EFLAGS: 00000246 ORIG_RAX: 000000000000001a > [17608.736913] RAX: ffffffffffffffda RBX: 00000000000008c8 RCX: 00007f34797f2743 > [17608.738902] RDX: 0000000000000004 RSI: 000000000000293f RDI: 00007f3479814000 > [17608.740949] RBP: 00000000000278c8 R08: 000000000000001f R09: 00007f3479814900 > [17608.742896] R10: fffffffffffffd0c R11: 0000000000000246 R12: 0000000000002077 > [17608.744810] R13: 000000000000293f R14: 00007f3479814000 R15: 0000000000000000 > [17608.746764] irq event stamp: 0 > [17608.747948] hardirqs last enabled at (0): [<0000000000000000>] 0x0 > [17608.749754] hardirqs last disabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > [17608.752428] softirqs last enabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > [17608.754935] softirqs last disabled at (0): [<0000000000000000>] 0x0 > [17608.756871] ---[ end trace ddba49bd36651725 ]--- > [17608.758590] BTRFS: error (device dm-0) in btrfs_sync_log:3089: errno=-5 IO failure This is due to concurrent fsyncs against files with shared extents. Not a new issue, but it's now easier to detect since I added the check in tree-checker for csum items with overlapping ranges. Before that it just happened silently. Besides the transaction abort, due to the tree-checker failure detection, it can also cause a more annoying BUG_ON() during fsync: [ 3566.138145] BTRFS critical (device dm-0): slot 81 key (18446744073709551606 128 13635584) new key (18446744073709551606 128 13635584) [ 3566.140427] BTRFS info (device dm-0): leaf 30949376 gen 7 total ptrs 98 free space 8527 owner 18446744073709551610 [ 3566.142673] BTRFS info (device dm-0): refs 4 lock (w:1 r:0 bw:0 br:0 sw:1 sr:0) lock_owner 13473 current 13473 [ 3566.144562] item 0 key (257 1 0) itemoff 16123 itemsize 160 [ 3566.145656] inode generation 7 size 262144 mode 100600 .... [ 3566.336479] ------------[ cut here ]------------ [ 3566.337054] kernel BUG at fs/btrfs/ctree.c:3153! [ 3566.337654] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC PTI [ 3566.338449] CPU: 1 PID: 13473 Comm: fsx Not tainted 5.6.0-rc7-btrfs-next-58 #1 [ 3566.339359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014 [ 3566.340820] RIP: 0010:btrfs_set_item_key_safe+0x1ea/0x270 [btrfs] [ 3566.341570] Code: 0f b6 47 08 41 ff 77 09 48 89 e9 89 da 48 c7 c6 d8 9d 8b c0 50 41 ff 37 48 8b 7c 24 28 e8 41 d9 0e 00 4c 89 e7 e8 76 53 01 00 <0f> 0b 48 c7 c7 e0 20 92 c0 e8 78 2a 2a f6 41 0f b6 47 08 41 ff 77 [ 3566.343869] RSP: 0018:ffff95e3889179d0 EFLAGS: 00010282 [ 3566.344513] RAX: 0000000000000000 RBX: 0000000000000051 RCX: 0000000000000000 [ 3566.345389] RDX: 0000000000000000 RSI: ffffffffb7763988 RDI: 0000000000000001 [ 3566.346265] RBP: fffffffffffffff6 R08: 0000000000000000 R09: 0000000000000001 [ 3566.347162] R10: 00000000000009ef R11: 0000000000000000 R12: ffff8912a8ba5a08 [ 3566.348037] R13: ffff95e388917a06 R14: ffff89138dcf68c8 R15: ffff95e388917ace [ 3566.348911] FS: 00007fe587084e80(0000) GS:ffff8913baa00000(0000) knlGS:0000000000000000 [ 3566.349902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3566.350636] CR2: 00007fe587091000 CR3: 0000000126dac005 CR4: 00000000003606e0 [ 3566.351514] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3566.352387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 3566.353420] Call Trace: [ 3566.353807] btrfs_del_csums+0x2f4/0x540 [btrfs] [ 3566.354512] copy_items+0x4b5/0x560 [btrfs] [ 3566.355151] btrfs_log_inode+0x910/0xf90 [btrfs] [ 3566.355853] btrfs_log_inode_parent+0x2a0/0xe40 [btrfs] [ 3566.356639] ? dget_parent+0x5/0x370 [ 3566.357183] btrfs_log_dentry_safe+0x4a/0x70 [btrfs] [ 3566.357925] btrfs_sync_file+0x42b/0x4d0 [btrfs] [ 3566.358629] __x64_sys_msync+0x199/0x200 [ 3566.359214] do_syscall_64+0x5c/0x280 [ 3566.359764] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 3566.360504] RIP: 0033:0x7fe586c65760 I've had this on my todo list for a while and a simple idea about how to fix it but haven't tried it yet. I'll see if I try it soon. -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: write-time leaf corruption detected 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba 2020-05-12 14:26 ` Filipe Manana @ 2020-05-13 3:10 ` Qu Wenruo 2020-05-13 3:17 ` Qu Wenruo 2020-05-13 9:25 ` Filipe Manana 2020-05-19 14:26 ` Bug 5.7-rc: write-time leaf corruption detected (fixed) David Sterba 2 siblings, 2 replies; 24+ messages in thread From: Qu Wenruo @ 2020-05-13 3:10 UTC (permalink / raw) To: David Sterba, linux-btrfs, Filipe Manana [-- Attachment #1.1: Type: text/plain, Size: 25947 bytes --] On 2020/5/12 下午10:15, David Sterba wrote: > Happened once in a VM, defaul mkfs/mount options. It looks like it's pretty hard to reproduce. > > generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 > [17607.625840] BTRFS: device fsid ae51523a-71e7-4730-9097-faeca36779be devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (27036) > [17607.664903] BTRFS info (device vdb): disk space caching is enabled > [17607.667422] BTRFS info (device vdb): has skinny extents > [17607.669552] BTRFS info (device vdb): flagging fs with big metadata feature > [17607.681613] BTRFS info (device vdb): checking UUID tree > [17607.921588] BTRFS: device fsid 16238d8a-1af3-4fa1-b3bc-fd6f04469192 devid 1 transid 5 /dev/mapper/logwrites-test scanned by mkfs.btrfs (27101) > [17607.948085] BTRFS info (device dm-0): turning on sync discard > [17607.952647] BTRFS info (device dm-0): disk space caching is enabled > [17607.955088] BTRFS info (device dm-0): has skinny extents > [17607.956924] BTRFS info (device dm-0): flagging fs with big metadata feature > [17607.966128] BTRFS info (device dm-0): checking UUID tree > [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item > [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 > [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 > [17608.106426] item 0 key (257 1 0) itemoff 16123 itemsize 160 Since we have 257 INODE_ITEM here, I guess that's all the log tree have. And surprisingly, the csum range around 1371000 doesn't even get referred in this log tree. Filipe, any idea on how this could happen? Thanks, Qu > [17608.108766] inode generation 6 size 262144 mode 100600 > [17608.110947] item 1 key (257 12 256) itemoff 16103 itemsize 20 > [17608.112698] item 2 key (257 108 0) itemoff 16050 itemsize 53 > [17608.114568] extent data disk bytenr 13631488 nr 262144 > [17608.116352] extent data offset 0 nr 262144 ram 262144 > [17608.118082] item 3 key (258 1 0) itemoff 15890 itemsize 160 > [17608.119920] inode generation 6 size 262144 mode 100600 > [17608.121605] item 4 key (258 12 256) itemoff 15871 itemsize 19 > [17608.123293] item 5 key (258 108 0) itemoff 15818 itemsize 53 > [17608.125212] extent data disk bytenr 13905920 nr 4096 > [17608.126851] extent data offset 0 nr 4096 ram 4096 > [17608.128342] item 6 key (258 108 4096) itemoff 15765 itemsize 53 > [17608.130106] extent data disk bytenr 13631488 nr 262144 > [17608.131774] extent data offset 0 nr 4096 ram 262144 > [17608.133330] item 7 key (258 108 8192) itemoff 15712 itemsize 53 > [17608.135151] extent data disk bytenr 13631488 nr 262144 > [17608.136680] extent data offset 8192 nr 8192 ram 262144 > [17608.137931] item 8 key (258 108 16384) itemoff 15659 itemsize 53 > [17608.139257] extent data disk bytenr 13631488 nr 262144 > [17608.140939] extent data offset 204800 nr 16384 ram 262144 > [17608.142255] item 9 key (258 108 32768) itemoff 15606 itemsize 53 > [17608.143941] extent data disk bytenr 13910016 nr 28672 > [17608.145576] extent data offset 0 nr 28672 ram 28672 > [17608.147165] item 10 key (258 108 61440) itemoff 15553 itemsize 53 > [17608.148964] extent data disk bytenr 13631488 nr 262144 > [17608.150563] extent data offset 249856 nr 8192 ram 262144 > [17608.152204] item 11 key (258 108 69632) itemoff 15500 itemsize 53 > [17608.153974] extent data disk bytenr 13631488 nr 262144 > [17608.155588] extent data offset 69632 nr 126976 ram 262144 > [17608.157099] item 12 key (258 108 196608) itemoff 15447 itemsize 53 > [17608.158698] extent data disk bytenr 13938688 nr 4096 > [17608.160123] extent data offset 0 nr 4096 ram 4096 > [17608.161300] item 13 key (258 108 200704) itemoff 15394 itemsize 53 > [17608.162597] extent data disk bytenr 0 nr 0 > [17608.164057] extent data offset 0 nr 28672 ram 28672 > [17608.165782] item 14 key (258 108 229376) itemoff 15341 itemsize 53 > [17608.167691] extent data disk bytenr 13942784 nr 4096 > [17608.169259] extent data offset 0 nr 4096 ram 4096 > [17608.170821] item 15 key (258 108 233472) itemoff 15288 itemsize 53 > [17608.172687] extent data disk bytenr 13631488 nr 262144 > [17608.174310] extent data offset 233472 nr 12288 ram 262144 > [17608.175993] item 16 key (258 108 245760) itemoff 15235 itemsize 53 > [17608.177821] extent data disk bytenr 13946880 nr 16384 > [17608.179440] extent data offset 0 nr 16384 ram 16384 > [17608.181043] item 17 key (259 1 0) itemoff 15075 itemsize 160 > [17608.182796] inode generation 6 size 262144 mode 100600 > [17608.184424] item 18 key (259 12 256) itemoff 15056 itemsize 19 > [17608.186085] item 19 key (259 108 0) itemoff 15003 itemsize 53 > [17608.187859] extent data disk bytenr 13975552 nr 4096 > [17608.189525] extent data offset 0 nr 4096 ram 4096 > [17608.191074] item 20 key (259 108 4096) itemoff 14950 itemsize 53 > [17608.192816] extent data disk bytenr 13631488 nr 262144 > [17608.194464] extent data offset 0 nr 4096 ram 262144 > [17608.196206] item 21 key (259 108 8192) itemoff 14897 itemsize 53 > [17608.198212] extent data disk bytenr 13631488 nr 262144 > [17608.200048] extent data offset 8192 nr 106496 ram 262144 > [17608.201684] item 22 key (259 108 114688) itemoff 14844 itemsize 53 > [17608.203659] extent data disk bytenr 14217216 nr 49152 > [17608.205398] extent data offset 0 nr 49152 ram 49152 > [17608.207126] item 23 key (259 108 163840) itemoff 14791 itemsize 53 > [17608.209019] extent data disk bytenr 13631488 nr 262144 > [17608.210736] extent data offset 163840 nr 98304 ram 262144 > [17608.212371] item 24 key (261 1 0) itemoff 14631 itemsize 160 > [17608.222944] inode generation 6 size 262144 mode 100600 > [17608.224637] item 25 key (261 12 256) itemoff 14612 itemsize 19 > [17608.226400] item 26 key (261 108 0) itemoff 14559 itemsize 53 > [17608.228169] extent data disk bytenr 14123008 nr 4096 > [17608.229794] extent data offset 0 nr 4096 ram 4096 > [17608.231292] item 27 key (261 108 4096) itemoff 14506 itemsize 53 > [17608.233203] extent data disk bytenr 13631488 nr 262144 > [17608.234794] extent data offset 0 nr 4096 ram 262144 > [17608.236362] item 28 key (261 108 8192) itemoff 14453 itemsize 53 > [17608.238102] extent data disk bytenr 13631488 nr 262144 > [17608.239782] extent data offset 8192 nr 16384 ram 262144 > [17608.241587] item 29 key (261 108 24576) itemoff 14400 itemsize 53 > [17608.243431] extent data disk bytenr 13631488 nr 262144 > [17608.245176] extent data offset 212992 nr 36864 ram 262144 > [17608.246991] item 30 key (261 108 61440) itemoff 14347 itemsize 53 > [17608.248868] extent data disk bytenr 13631488 nr 262144 > [17608.250456] extent data offset 61440 nr 114688 ram 262144 > [17608.252333] item 31 key (261 108 176128) itemoff 14294 itemsize 53 > [17608.254343] extent data disk bytenr 14135296 nr 61440 > [17608.256131] extent data offset 0 nr 61440 ram 61440 > [17608.257857] item 32 key (261 108 237568) itemoff 14241 itemsize 53 > [17608.259864] extent data disk bytenr 13631488 nr 262144 > [17608.261484] extent data offset 237568 nr 24576 ram 262144 > [17608.263134] item 33 key (262 1 0) itemoff 14081 itemsize 160 > [17608.264865] inode generation 6 size 178311 mode 100600 > [17608.266625] item 34 key (262 12 256) itemoff 14062 itemsize 19 > [17608.268551] item 35 key (262 108 0) itemoff 14009 itemsize 53 > [17608.270363] extent data disk bytenr 14196736 nr 4096 > [17608.272019] extent data offset 0 nr 4096 ram 4096 > [17608.273505] item 36 key (262 108 4096) itemoff 13956 itemsize 53 > [17608.275387] extent data disk bytenr 13631488 nr 262144 > [17608.277125] extent data offset 0 nr 4096 ram 262144 > [17608.278780] item 37 key (262 108 8192) itemoff 13903 itemsize 53 > [17608.280507] extent data disk bytenr 13631488 nr 262144 > [17608.282163] extent data offset 8192 nr 106496 ram 262144 > [17608.283866] item 38 key (262 108 114688) itemoff 13850 itemsize 53 > [17608.285727] extent data disk bytenr 14200832 nr 4096 > [17608.287264] extent data offset 0 nr 4096 ram 4096 > [17608.288719] item 39 key (262 108 118784) itemoff 13797 itemsize 53 > [17608.290696] extent data disk bytenr 0 nr 0 > [17608.292237] extent data offset 0 nr 49152 ram 49152 > [17608.293667] item 40 key (262 108 167936) itemoff 13744 itemsize 53 > [17608.295436] extent data disk bytenr 14209024 nr 4096 > [17608.297051] extent data offset 0 nr 4096 ram 4096 > [17608.298583] item 41 key (262 108 172032) itemoff 13691 itemsize 53 > [17608.300573] extent data disk bytenr 13631488 nr 262144 > [17608.302323] extent data offset 172032 nr 8192 ram 262144 > [17608.304112] item 42 key (263 1 0) itemoff 13531 itemsize 160 > [17608.305810] inode generation 6 size 262144 mode 100600 > [17608.307444] item 43 key (263 12 256) itemoff 13512 itemsize 19 > [17608.309108] item 44 key (263 108 0) itemoff 13459 itemsize 53 > [17608.310760] extent data disk bytenr 14266368 nr 4096 > [17608.312269] extent data offset 0 nr 4096 ram 4096 > [17608.313883] item 45 key (263 108 4096) itemoff 13406 itemsize 53 > [17608.315659] extent data disk bytenr 13631488 nr 262144 > [17608.317133] extent data offset 0 nr 4096 ram 262144 > [17608.318691] item 46 key (263 108 8192) itemoff 13353 itemsize 53 > [17608.320417] extent data disk bytenr 13631488 nr 262144 > [17608.321851] extent data offset 8192 nr 81920 ram 262144 > [17608.323351] item 47 key (263 108 90112) itemoff 13300 itemsize 53 > [17608.325222] extent data disk bytenr 14319616 nr 4096 > [17608.326968] extent data offset 0 nr 4096 ram 4096 > [17608.328583] item 48 key (263 108 94208) itemoff 13247 itemsize 53 > [17608.330364] extent data disk bytenr 0 nr 0 > [17608.331708] extent data offset 0 nr 45056 ram 45056 > [17608.333200] item 49 key (263 108 139264) itemoff 13194 itemsize 53 > [17608.334979] extent data disk bytenr 14323712 nr 4096 > [17608.336570] extent data offset 0 nr 4096 ram 4096 > [17608.338023] item 50 key (263 108 143360) itemoff 13141 itemsize 53 > [17608.340091] extent data disk bytenr 13631488 nr 262144 > [17608.341746] extent data offset 143360 nr 8192 ram 262144 > [17608.343400] item 51 key (263 108 151552) itemoff 13088 itemsize 53 > [17608.345228] extent data disk bytenr 14327808 nr 61440 > [17608.346924] extent data offset 0 nr 61440 ram 61440 > [17608.348626] item 52 key (263 108 212992) itemoff 13035 itemsize 53 > [17608.350371] extent data disk bytenr 14270464 nr 49152 > [17608.351985] extent data offset 0 nr 49152 ram 49152 > [17608.353542] item 53 key (264 1 0) itemoff 12875 itemsize 160 > [17608.355125] inode generation 6 size 247974 mode 100600 > [17608.356813] item 54 key (264 12 256) itemoff 12856 itemsize 19 > [17608.358539] item 55 key (264 108 0) itemoff 12803 itemsize 53 > [17608.360340] extent data disk bytenr 14471168 nr 4096 > [17608.361935] extent data offset 0 nr 4096 ram 4096 > [17608.363426] item 56 key (264 108 4096) itemoff 12750 itemsize 53 > [17608.365224] extent data disk bytenr 13631488 nr 262144 > [17608.366836] extent data offset 0 nr 4096 ram 262144 > [17608.368439] item 57 key (264 108 8192) itemoff 12697 itemsize 53 > [17608.370226] extent data disk bytenr 13631488 nr 262144 > [17608.371714] extent data offset 8192 nr 8192 ram 262144 > [17608.373157] item 58 key (264 108 16384) itemoff 12644 itemsize 53 > [17608.374820] extent data disk bytenr 14483456 nr 4096 > [17608.376495] extent data offset 0 nr 4096 ram 4096 > [17608.378014] item 59 key (264 108 20480) itemoff 12591 itemsize 53 > [17608.379882] extent data disk bytenr 0 nr 0 > [17608.381425] extent data offset 0 nr 20480 ram 20480 > [17608.383188] item 60 key (264 108 40960) itemoff 12538 itemsize 53 > [17608.385037] extent data disk bytenr 14487552 nr 4096 > [17608.386644] extent data offset 0 nr 4096 ram 4096 > [17608.388191] item 61 key (264 108 45056) itemoff 12485 itemsize 53 > [17608.390035] extent data disk bytenr 13631488 nr 262144 > [17608.391634] extent data offset 45056 nr 20480 ram 262144 > [17608.393440] item 62 key (264 108 65536) itemoff 12432 itemsize 53 > [17608.395464] extent data disk bytenr 14475264 nr 4096 > [17608.397179] extent data offset 0 nr 4096 ram 4096 > [17608.398675] item 63 key (264 108 69632) itemoff 12379 itemsize 53 > [17608.400412] extent data disk bytenr 0 nr 0 > [17608.401843] extent data offset 0 nr 28672 ram 49152 > [17608.403409] item 64 key (264 108 98304) itemoff 12326 itemsize 53 > [17608.405389] extent data disk bytenr 14491648 nr 20480 > [17608.407111] extent data offset 0 nr 8192 ram 20480 > [17608.408795] item 65 key (264 108 106496) itemoff 12273 itemsize 53 > [17608.410757] extent data disk bytenr 14561280 nr 53248 > [17608.412483] extent data offset 0 nr 53248 ram 53248 > [17608.414166] item 66 key (264 108 159744) itemoff 12220 itemsize 53 > [17608.416164] extent data disk bytenr 13631488 nr 262144 > [17608.417897] extent data offset 159744 nr 16384 ram 262144 > [17608.419677] item 67 key (264 108 176128) itemoff 12167 itemsize 53 > [17608.421598] extent data disk bytenr 13631488 nr 262144 > [17608.423313] extent data offset 237568 nr 4096 ram 262144 > [17608.425051] item 68 key (264 108 180224) itemoff 12114 itemsize 53 > [17608.427009] extent data disk bytenr 13631488 nr 262144 > [17608.428787] extent data offset 180224 nr 45056 ram 262144 > [17608.430588] item 69 key (264 108 225280) itemoff 12061 itemsize 53 > [17608.432457] extent data disk bytenr 14614528 nr 16384 > [17608.434122] extent data offset 0 nr 16384 ram 16384 > [17608.435663] item 70 key (264 108 241664) itemoff 12008 itemsize 53 > [17608.437520] extent data disk bytenr 13631488 nr 262144 > [17608.439160] extent data offset 241664 nr 8192 ram 262144 > [17608.440869] item 71 key (265 1 0) itemoff 11848 itemsize 160 > [17608.442586] inode generation 6 size 193781 mode 100600 > [17608.444197] item 72 key (265 12 256) itemoff 11829 itemsize 19 > [17608.446043] item 73 key (265 108 0) itemoff 11776 itemsize 53 > [17608.447726] extent data disk bytenr 14524416 nr 4096 > [17608.449507] extent data offset 0 nr 4096 ram 4096 > [17608.451142] item 74 key (265 108 4096) itemoff 11723 itemsize 53 > [17608.452883] extent data disk bytenr 13631488 nr 262144 > [17608.454517] extent data offset 0 nr 4096 ram 262144 > [17608.456136] item 75 key (265 108 8192) itemoff 11670 itemsize 53 > [17608.458065] extent data disk bytenr 13631488 nr 262144 > [17608.459732] extent data offset 8192 nr 110592 ram 262144 > [17608.461260] item 76 key (265 108 118784) itemoff 11617 itemsize 53 > [17608.462918] extent data disk bytenr 14528512 nr 28672 > [17608.464562] extent data offset 0 nr 28672 ram 28672 > [17608.466204] item 77 key (265 108 147456) itemoff 11564 itemsize 53 > [17608.468161] extent data disk bytenr 0 nr 0 > [17608.469631] extent data offset 8192 nr 4096 ram 12288 > [17608.471331] item 78 key (265 108 151552) itemoff 11511 itemsize 53 > [17608.473363] extent data disk bytenr 14557184 nr 4096 > [17608.474920] extent data offset 0 nr 4096 ram 4096 > [17608.476499] item 79 key (265 108 155648) itemoff 11458 itemsize 53 > [17608.478306] extent data disk bytenr 13631488 nr 262144 > [17608.479921] extent data offset 155648 nr 40960 ram 262144 > [17608.481652] item 80 key (266 1 0) itemoff 11298 itemsize 160 > [17608.483268] inode generation 6 size 262144 mode 100600 > [17608.484849] item 81 key (266 12 256) itemoff 11279 itemsize 19 > [17608.486489] item 82 key (266 108 0) itemoff 11226 itemsize 53 > [17608.488298] extent data disk bytenr 14712832 nr 4096 > [17608.489866] extent data offset 0 nr 4096 ram 4096 > [17608.491399] item 83 key (266 108 4096) itemoff 11173 itemsize 53 > [17608.493255] extent data disk bytenr 13631488 nr 262144 > [17608.494898] extent data offset 0 nr 4096 ram 262144 > [17608.496504] item 84 key (266 108 8192) itemoff 11120 itemsize 53 > [17608.498434] extent data disk bytenr 13631488 nr 262144 > [17608.500033] extent data offset 8192 nr 73728 ram 262144 > [17608.501619] item 85 key (266 108 81920) itemoff 11067 itemsize 53 > [17608.503455] extent data disk bytenr 14741504 nr 4096 > [17608.505086] extent data offset 0 nr 4096 ram 4096 > [17608.506648] item 86 key (266 108 86016) itemoff 11014 itemsize 53 > [17608.508346] extent data disk bytenr 13631488 nr 262144 > [17608.509862] extent data offset 86016 nr 28672 ram 262144 > [17608.511645] item 87 key (266 108 114688) itemoff 10961 itemsize 53 > [17608.513620] extent data disk bytenr 13631488 nr 262144 > [17608.515403] extent data offset 36864 nr 28672 ram 262144 > [17608.517234] item 88 key (266 108 143360) itemoff 10908 itemsize 53 > [17608.519248] extent data disk bytenr 13631488 nr 262144 > [17608.520976] extent data offset 143360 nr 16384 ram 262144 > [17608.522635] item 89 key (266 108 159744) itemoff 10855 itemsize 53 > [17608.524583] extent data disk bytenr 14745600 nr 12288 > [17608.526254] extent data offset 0 nr 12288 ram 12288 > [17608.527858] item 90 key (266 108 172032) itemoff 10802 itemsize 53 > [17608.538634] extent data disk bytenr 13631488 nr 262144 > [17608.540392] extent data offset 172032 nr 36864 ram 262144 > [17608.542156] item 91 key (266 108 208896) itemoff 10749 itemsize 53 > [17608.544104] extent data disk bytenr 14716928 nr 4096 > [17608.545664] extent data offset 0 nr 4096 ram 4096 > [17608.547207] item 92 key (266 108 212992) itemoff 10696 itemsize 53 > [17608.549038] extent data disk bytenr 0 nr 0 > [17608.550440] extent data offset 0 nr 32768 ram 32768 > [17608.552056] item 93 key (266 108 245760) itemoff 10643 itemsize 53 > [17608.553997] extent data disk bytenr 14721024 nr 4096 > [17608.555578] extent data offset 0 nr 4096 ram 4096 > [17608.557113] item 94 key (266 108 249856) itemoff 10590 itemsize 53 > [17608.558861] extent data disk bytenr 14688256 nr 24576 > [17608.560439] extent data offset 12288 nr 12288 ram 24576 > [17608.562137] item 95 key (267 1 0) itemoff 10430 itemsize 160 > [17608.563756] inode generation 6 size 262144 mode 100600 > [17608.565261] item 96 key (267 12 256) itemoff 10411 itemsize 19 > [17608.566912] item 97 key (267 108 0) itemoff 10358 itemsize 53 > [17608.568715] extent data disk bytenr 14737408 nr 4096 > [17608.570283] extent data offset 0 nr 4096 ram 4096 > [17608.571810] item 98 key (267 108 4096) itemoff 10305 itemsize 53 > [17608.573502] extent data disk bytenr 13631488 nr 262144 > [17608.575033] extent data offset 0 nr 4096 ram 262144 > [17608.576507] item 99 key (267 108 8192) itemoff 10252 itemsize 53 > [17608.578416] extent data disk bytenr 13631488 nr 262144 > [17608.581054] extent data offset 8192 nr 167936 ram 262144 > [17608.583613] item 100 key (267 108 176128) itemoff 10199 itemsize 53 > [17608.585846] extent data disk bytenr 13631488 nr 262144 > [17608.587883] extent data offset 188416 nr 4096 ram 262144 > [17608.590034] item 101 key (267 108 180224) itemoff 10146 itemsize 53 > [17608.592397] extent data disk bytenr 13631488 nr 262144 > [17608.594415] extent data offset 180224 nr 81920 ram 262144 > [17608.595941] item 102 key (18446744073709551606 128 13631488) itemoff 10142 itemsize 4 > [17608.598465] item 103 key (18446744073709551606 128 13635584) itemoff 10082 itemsize 60 > [17608.601262] item 104 key (18446744073709551606 128 13697024) itemoff 9974 itemsize 108 > [17608.604090] item 105 key (18446744073709551606 128 13717504) itemoff 9946 itemsize 28 > [17608.606573] item 106 key (18446744073709551606 128 13774848) itemoff 9930 itemsize 16 > [17608.609125] item 107 key (18446744073709551606 128 13803520) itemoff 9842 itemsize 88 > [17608.611709] item 108 key (18446744073709551606 128 13905920) itemoff 9786 itemsize 56 > [17608.614269] item 109 key (18446744073709551606 128 13975552) itemoff 9782 itemsize 4 > [17608.616992] item 110 key (18446744073709551606 128 14123008) itemoff 9778 itemsize 4 > [17608.619523] item 111 key (18446744073709551606 128 14135296) itemoff 9710 itemsize 68 > [17608.622144] item 112 key (18446744073709551606 128 14209024) itemoff 9706 itemsize 4 > [17608.624841] item 113 key (18446744073709551606 128 14217216) itemoff 9538 itemsize 168 > [17608.627409] item 114 key (18446744073709551606 128 14471168) itemoff 9530 itemsize 8 > [17608.630034] item 115 key (18446744073709551606 128 14479360) itemoff 9510 itemsize 20 > [17608.632681] item 116 key (18446744073709551606 128 14499840) itemoff 9498 itemsize 12 > [17608.635448] item 117 key (18446744073709551606 128 14524416) itemoff 9394 itemsize 104 > [17608.638122] item 118 key (18446744073709551606 128 14700544) itemoff 9382 itemsize 12 > [17608.640633] item 119 key (18446744073709551606 128 14712832) itemoff 9370 itemsize 12 > [17608.643227] item 120 key (18446744073709551606 128 14737408) itemoff 9350 itemsize 20 > [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected > [17608.648714] ------------[ cut here ]------------ > [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] > [17608.652972] Modules linked in: dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_writes dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [last unloaded: scsi_debug] > [17608.660683] CPU: 2 PID: 27162 Comm: fsx Tainted: G L 5.7.0-rc3-default+ #1094 > [17608.663520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 > [17608.667102] RIP: 0010:btree_csum_one_bio+0x297/0x2a0 [btrfs] > [17608.668919] Code: bc fd ff ff e8 aa e7 f0 c2 31 f6 48 89 3c 24 e8 ff 7b ff ff 48 8b 3c 24 48 c7 c6 78 75 25 c0 48 8b 17 4c 89 e7 e8 84 bc 0b 00 <0f> 0b e9 8f fd ff ff 66 90 0f 1f 44 00 00 48 89 f7 e9 53 fd ff ff > [17608.674610] RSP: 0018:ffffafbb81e978b8 EFLAGS: 00010292 > [17608.676299] RAX: 0000000000000000 RBX: ffff94287f810f00 RCX: 0000000000000001 > [17608.678260] RDX: 0000000000000000 RSI: ffffffff83109fc2 RDI: ffffffff8310a096 > [17608.680484] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 > [17608.682645] R10: 0000000000000000 R11: 0000000000008ca0 R12: ffff94285f298000 > [17608.684681] R13: ffff94274100bd30 R14: 0000000000000000 R15: 00000000ffffff8b > [17608.686659] FS: 00007f3479617b80(0000) GS:ffff94287ba00000(0000) knlGS:0000000000000000 > [17608.689240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [17608.691150] CR2: 0000000000466000 CR3: 000000010227f001 CR4: 0000000000160ee0 > [17608.693354] Call Trace: > [17608.694341] btree_submit_bio_hook+0x74/0xc0 [btrfs] > [17608.696162] submit_one_bio+0x2b/0x40 [btrfs] > [17608.698391] btree_write_cache_pages+0x373/0x440 [btrfs] > [17608.700085] ? lock_acquire+0xa3/0x400 > [17608.701257] ? __filemap_fdatawrite_range+0xad/0x110 > [17608.702736] do_writepages+0x40/0xe0 > [17608.703953] ? do_raw_spin_unlock+0x4b/0xc0 > [17608.705290] ? _raw_spin_unlock+0x1f/0x30 > [17608.706571] ? wbc_attach_and_unlock_inode+0x194/0x2a0 > [17608.708167] __filemap_fdatawrite_range+0xce/0x110 > [17608.709679] btrfs_write_marked_extents+0x68/0x160 [btrfs] > [17608.711384] btrfs_sync_log+0x1a8/0xd40 [btrfs] > [17608.712892] ? btrfs_log_inode+0x370/0xe20 [btrfs] > [17608.714459] ? lock_acquire+0xa3/0x400 > [17608.715803] ? lockref_put_or_lock+0x9/0x30 > [17608.717115] ? dput+0x20/0x490 > [17608.718172] ? dput+0x20/0x490 > [17608.719271] ? do_raw_spin_unlock+0x4b/0xc0 > [17608.720639] ? _raw_spin_unlock+0x1f/0x30 > [17608.722002] btrfs_sync_file+0x335/0x490 [btrfs] > [17608.723610] __x64_sys_msync+0x19e/0x200 > [17608.724924] do_syscall_64+0x50/0x210 > [17608.726143] entry_SYSCALL_64_after_hwframe+0x49/0xb3 > [17608.727719] RIP: 0033:0x7f34797f2743 > [17608.729030] Code: 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 1a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48 > [17608.734572] RSP: 002b:00007ffe84f11858 EFLAGS: 00000246 ORIG_RAX: 000000000000001a > [17608.736913] RAX: ffffffffffffffda RBX: 00000000000008c8 RCX: 00007f34797f2743 > [17608.738902] RDX: 0000000000000004 RSI: 000000000000293f RDI: 00007f3479814000 > [17608.740949] RBP: 00000000000278c8 R08: 000000000000001f R09: 00007f3479814900 > [17608.742896] R10: fffffffffffffd0c R11: 0000000000000246 R12: 0000000000002077 > [17608.744810] R13: 000000000000293f R14: 00007f3479814000 R15: 0000000000000000 > [17608.746764] irq event stamp: 0 > [17608.747948] hardirqs last enabled at (0): [<0000000000000000>] 0x0 > [17608.749754] hardirqs last disabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > [17608.752428] softirqs last enabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > [17608.754935] softirqs last disabled at (0): [<0000000000000000>] 0x0 > [17608.756871] ---[ end trace ddba49bd36651725 ]--- > [17608.758590] BTRFS: error (device dm-0) in btrfs_sync_log:3089: errno=-5 IO failure > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: write-time leaf corruption detected 2020-05-13 3:10 ` Qu Wenruo @ 2020-05-13 3:17 ` Qu Wenruo 2020-05-13 9:25 ` Filipe Manana 1 sibling, 0 replies; 24+ messages in thread From: Qu Wenruo @ 2020-05-13 3:17 UTC (permalink / raw) To: David Sterba, linux-btrfs, Filipe Manana [-- Attachment #1.1: Type: text/plain, Size: 26636 bytes --] On 2020/5/13 上午11:10, Qu Wenruo wrote: > > > On 2020/5/12 下午10:15, David Sterba wrote: >> Happened once in a VM, defaul mkfs/mount options. > > It looks like it's pretty hard to reproduce. > >> >> generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 >> [17607.625840] BTRFS: device fsid ae51523a-71e7-4730-9097-faeca36779be devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (27036) >> [17607.664903] BTRFS info (device vdb): disk space caching is enabled >> [17607.667422] BTRFS info (device vdb): has skinny extents >> [17607.669552] BTRFS info (device vdb): flagging fs with big metadata feature >> [17607.681613] BTRFS info (device vdb): checking UUID tree >> [17607.921588] BTRFS: device fsid 16238d8a-1af3-4fa1-b3bc-fd6f04469192 devid 1 transid 5 /dev/mapper/logwrites-test scanned by mkfs.btrfs (27101) >> [17607.948085] BTRFS info (device dm-0): turning on sync discard >> [17607.952647] BTRFS info (device dm-0): disk space caching is enabled >> [17607.955088] BTRFS info (device dm-0): has skinny extents >> [17607.956924] BTRFS info (device dm-0): flagging fs with big metadata feature >> [17607.966128] BTRFS info (device dm-0): checking UUID tree >> [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item >> [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 >> [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 >> [17608.106426] item 0 key (257 1 0) itemoff 16123 itemsize 160 > > Since we have 257 INODE_ITEM here, I guess that's all the log tree have. > > And surprisingly, the csum range around 1371000 doesn't even get > referred in this log tree. > > Filipe, any idea on how this could happen? Oh, my bad again. The extent [13631488, 13631488 + 262144) covers the offending csum range. So it's indeed some overlapping csum problem. It's a pitty that kernel dump tree doesn't contain the file extent type. Thanks, Qu > > Thanks, > Qu >> [17608.108766] inode generation 6 size 262144 mode 100600 >> [17608.110947] item 1 key (257 12 256) itemoff 16103 itemsize 20 >> [17608.112698] item 2 key (257 108 0) itemoff 16050 itemsize 53 >> [17608.114568] extent data disk bytenr 13631488 nr 262144 >> [17608.116352] extent data offset 0 nr 262144 ram 262144 >> [17608.118082] item 3 key (258 1 0) itemoff 15890 itemsize 160 >> [17608.119920] inode generation 6 size 262144 mode 100600 >> [17608.121605] item 4 key (258 12 256) itemoff 15871 itemsize 19 >> [17608.123293] item 5 key (258 108 0) itemoff 15818 itemsize 53 >> [17608.125212] extent data disk bytenr 13905920 nr 4096 >> [17608.126851] extent data offset 0 nr 4096 ram 4096 >> [17608.128342] item 6 key (258 108 4096) itemoff 15765 itemsize 53 >> [17608.130106] extent data disk bytenr 13631488 nr 262144 >> [17608.131774] extent data offset 0 nr 4096 ram 262144 >> [17608.133330] item 7 key (258 108 8192) itemoff 15712 itemsize 53 >> [17608.135151] extent data disk bytenr 13631488 nr 262144 >> [17608.136680] extent data offset 8192 nr 8192 ram 262144 >> [17608.137931] item 8 key (258 108 16384) itemoff 15659 itemsize 53 >> [17608.139257] extent data disk bytenr 13631488 nr 262144 >> [17608.140939] extent data offset 204800 nr 16384 ram 262144 >> [17608.142255] item 9 key (258 108 32768) itemoff 15606 itemsize 53 >> [17608.143941] extent data disk bytenr 13910016 nr 28672 >> [17608.145576] extent data offset 0 nr 28672 ram 28672 >> [17608.147165] item 10 key (258 108 61440) itemoff 15553 itemsize 53 >> [17608.148964] extent data disk bytenr 13631488 nr 262144 >> [17608.150563] extent data offset 249856 nr 8192 ram 262144 >> [17608.152204] item 11 key (258 108 69632) itemoff 15500 itemsize 53 >> [17608.153974] extent data disk bytenr 13631488 nr 262144 >> [17608.155588] extent data offset 69632 nr 126976 ram 262144 >> [17608.157099] item 12 key (258 108 196608) itemoff 15447 itemsize 53 >> [17608.158698] extent data disk bytenr 13938688 nr 4096 >> [17608.160123] extent data offset 0 nr 4096 ram 4096 >> [17608.161300] item 13 key (258 108 200704) itemoff 15394 itemsize 53 >> [17608.162597] extent data disk bytenr 0 nr 0 >> [17608.164057] extent data offset 0 nr 28672 ram 28672 >> [17608.165782] item 14 key (258 108 229376) itemoff 15341 itemsize 53 >> [17608.167691] extent data disk bytenr 13942784 nr 4096 >> [17608.169259] extent data offset 0 nr 4096 ram 4096 >> [17608.170821] item 15 key (258 108 233472) itemoff 15288 itemsize 53 >> [17608.172687] extent data disk bytenr 13631488 nr 262144 >> [17608.174310] extent data offset 233472 nr 12288 ram 262144 >> [17608.175993] item 16 key (258 108 245760) itemoff 15235 itemsize 53 >> [17608.177821] extent data disk bytenr 13946880 nr 16384 >> [17608.179440] extent data offset 0 nr 16384 ram 16384 >> [17608.181043] item 17 key (259 1 0) itemoff 15075 itemsize 160 >> [17608.182796] inode generation 6 size 262144 mode 100600 >> [17608.184424] item 18 key (259 12 256) itemoff 15056 itemsize 19 >> [17608.186085] item 19 key (259 108 0) itemoff 15003 itemsize 53 >> [17608.187859] extent data disk bytenr 13975552 nr 4096 >> [17608.189525] extent data offset 0 nr 4096 ram 4096 >> [17608.191074] item 20 key (259 108 4096) itemoff 14950 itemsize 53 >> [17608.192816] extent data disk bytenr 13631488 nr 262144 >> [17608.194464] extent data offset 0 nr 4096 ram 262144 >> [17608.196206] item 21 key (259 108 8192) itemoff 14897 itemsize 53 >> [17608.198212] extent data disk bytenr 13631488 nr 262144 >> [17608.200048] extent data offset 8192 nr 106496 ram 262144 >> [17608.201684] item 22 key (259 108 114688) itemoff 14844 itemsize 53 >> [17608.203659] extent data disk bytenr 14217216 nr 49152 >> [17608.205398] extent data offset 0 nr 49152 ram 49152 >> [17608.207126] item 23 key (259 108 163840) itemoff 14791 itemsize 53 >> [17608.209019] extent data disk bytenr 13631488 nr 262144 >> [17608.210736] extent data offset 163840 nr 98304 ram 262144 >> [17608.212371] item 24 key (261 1 0) itemoff 14631 itemsize 160 >> [17608.222944] inode generation 6 size 262144 mode 100600 >> [17608.224637] item 25 key (261 12 256) itemoff 14612 itemsize 19 >> [17608.226400] item 26 key (261 108 0) itemoff 14559 itemsize 53 >> [17608.228169] extent data disk bytenr 14123008 nr 4096 >> [17608.229794] extent data offset 0 nr 4096 ram 4096 >> [17608.231292] item 27 key (261 108 4096) itemoff 14506 itemsize 53 >> [17608.233203] extent data disk bytenr 13631488 nr 262144 >> [17608.234794] extent data offset 0 nr 4096 ram 262144 >> [17608.236362] item 28 key (261 108 8192) itemoff 14453 itemsize 53 >> [17608.238102] extent data disk bytenr 13631488 nr 262144 >> [17608.239782] extent data offset 8192 nr 16384 ram 262144 >> [17608.241587] item 29 key (261 108 24576) itemoff 14400 itemsize 53 >> [17608.243431] extent data disk bytenr 13631488 nr 262144 >> [17608.245176] extent data offset 212992 nr 36864 ram 262144 >> [17608.246991] item 30 key (261 108 61440) itemoff 14347 itemsize 53 >> [17608.248868] extent data disk bytenr 13631488 nr 262144 >> [17608.250456] extent data offset 61440 nr 114688 ram 262144 >> [17608.252333] item 31 key (261 108 176128) itemoff 14294 itemsize 53 >> [17608.254343] extent data disk bytenr 14135296 nr 61440 >> [17608.256131] extent data offset 0 nr 61440 ram 61440 >> [17608.257857] item 32 key (261 108 237568) itemoff 14241 itemsize 53 >> [17608.259864] extent data disk bytenr 13631488 nr 262144 >> [17608.261484] extent data offset 237568 nr 24576 ram 262144 >> [17608.263134] item 33 key (262 1 0) itemoff 14081 itemsize 160 >> [17608.264865] inode generation 6 size 178311 mode 100600 >> [17608.266625] item 34 key (262 12 256) itemoff 14062 itemsize 19 >> [17608.268551] item 35 key (262 108 0) itemoff 14009 itemsize 53 >> [17608.270363] extent data disk bytenr 14196736 nr 4096 >> [17608.272019] extent data offset 0 nr 4096 ram 4096 >> [17608.273505] item 36 key (262 108 4096) itemoff 13956 itemsize 53 >> [17608.275387] extent data disk bytenr 13631488 nr 262144 >> [17608.277125] extent data offset 0 nr 4096 ram 262144 >> [17608.278780] item 37 key (262 108 8192) itemoff 13903 itemsize 53 >> [17608.280507] extent data disk bytenr 13631488 nr 262144 >> [17608.282163] extent data offset 8192 nr 106496 ram 262144 >> [17608.283866] item 38 key (262 108 114688) itemoff 13850 itemsize 53 >> [17608.285727] extent data disk bytenr 14200832 nr 4096 >> [17608.287264] extent data offset 0 nr 4096 ram 4096 >> [17608.288719] item 39 key (262 108 118784) itemoff 13797 itemsize 53 >> [17608.290696] extent data disk bytenr 0 nr 0 >> [17608.292237] extent data offset 0 nr 49152 ram 49152 >> [17608.293667] item 40 key (262 108 167936) itemoff 13744 itemsize 53 >> [17608.295436] extent data disk bytenr 14209024 nr 4096 >> [17608.297051] extent data offset 0 nr 4096 ram 4096 >> [17608.298583] item 41 key (262 108 172032) itemoff 13691 itemsize 53 >> [17608.300573] extent data disk bytenr 13631488 nr 262144 >> [17608.302323] extent data offset 172032 nr 8192 ram 262144 >> [17608.304112] item 42 key (263 1 0) itemoff 13531 itemsize 160 >> [17608.305810] inode generation 6 size 262144 mode 100600 >> [17608.307444] item 43 key (263 12 256) itemoff 13512 itemsize 19 >> [17608.309108] item 44 key (263 108 0) itemoff 13459 itemsize 53 >> [17608.310760] extent data disk bytenr 14266368 nr 4096 >> [17608.312269] extent data offset 0 nr 4096 ram 4096 >> [17608.313883] item 45 key (263 108 4096) itemoff 13406 itemsize 53 >> [17608.315659] extent data disk bytenr 13631488 nr 262144 >> [17608.317133] extent data offset 0 nr 4096 ram 262144 >> [17608.318691] item 46 key (263 108 8192) itemoff 13353 itemsize 53 >> [17608.320417] extent data disk bytenr 13631488 nr 262144 >> [17608.321851] extent data offset 8192 nr 81920 ram 262144 >> [17608.323351] item 47 key (263 108 90112) itemoff 13300 itemsize 53 >> [17608.325222] extent data disk bytenr 14319616 nr 4096 >> [17608.326968] extent data offset 0 nr 4096 ram 4096 >> [17608.328583] item 48 key (263 108 94208) itemoff 13247 itemsize 53 >> [17608.330364] extent data disk bytenr 0 nr 0 >> [17608.331708] extent data offset 0 nr 45056 ram 45056 >> [17608.333200] item 49 key (263 108 139264) itemoff 13194 itemsize 53 >> [17608.334979] extent data disk bytenr 14323712 nr 4096 >> [17608.336570] extent data offset 0 nr 4096 ram 4096 >> [17608.338023] item 50 key (263 108 143360) itemoff 13141 itemsize 53 >> [17608.340091] extent data disk bytenr 13631488 nr 262144 >> [17608.341746] extent data offset 143360 nr 8192 ram 262144 >> [17608.343400] item 51 key (263 108 151552) itemoff 13088 itemsize 53 >> [17608.345228] extent data disk bytenr 14327808 nr 61440 >> [17608.346924] extent data offset 0 nr 61440 ram 61440 >> [17608.348626] item 52 key (263 108 212992) itemoff 13035 itemsize 53 >> [17608.350371] extent data disk bytenr 14270464 nr 49152 >> [17608.351985] extent data offset 0 nr 49152 ram 49152 >> [17608.353542] item 53 key (264 1 0) itemoff 12875 itemsize 160 >> [17608.355125] inode generation 6 size 247974 mode 100600 >> [17608.356813] item 54 key (264 12 256) itemoff 12856 itemsize 19 >> [17608.358539] item 55 key (264 108 0) itemoff 12803 itemsize 53 >> [17608.360340] extent data disk bytenr 14471168 nr 4096 >> [17608.361935] extent data offset 0 nr 4096 ram 4096 >> [17608.363426] item 56 key (264 108 4096) itemoff 12750 itemsize 53 >> [17608.365224] extent data disk bytenr 13631488 nr 262144 >> [17608.366836] extent data offset 0 nr 4096 ram 262144 >> [17608.368439] item 57 key (264 108 8192) itemoff 12697 itemsize 53 >> [17608.370226] extent data disk bytenr 13631488 nr 262144 >> [17608.371714] extent data offset 8192 nr 8192 ram 262144 >> [17608.373157] item 58 key (264 108 16384) itemoff 12644 itemsize 53 >> [17608.374820] extent data disk bytenr 14483456 nr 4096 >> [17608.376495] extent data offset 0 nr 4096 ram 4096 >> [17608.378014] item 59 key (264 108 20480) itemoff 12591 itemsize 53 >> [17608.379882] extent data disk bytenr 0 nr 0 >> [17608.381425] extent data offset 0 nr 20480 ram 20480 >> [17608.383188] item 60 key (264 108 40960) itemoff 12538 itemsize 53 >> [17608.385037] extent data disk bytenr 14487552 nr 4096 >> [17608.386644] extent data offset 0 nr 4096 ram 4096 >> [17608.388191] item 61 key (264 108 45056) itemoff 12485 itemsize 53 >> [17608.390035] extent data disk bytenr 13631488 nr 262144 >> [17608.391634] extent data offset 45056 nr 20480 ram 262144 >> [17608.393440] item 62 key (264 108 65536) itemoff 12432 itemsize 53 >> [17608.395464] extent data disk bytenr 14475264 nr 4096 >> [17608.397179] extent data offset 0 nr 4096 ram 4096 >> [17608.398675] item 63 key (264 108 69632) itemoff 12379 itemsize 53 >> [17608.400412] extent data disk bytenr 0 nr 0 >> [17608.401843] extent data offset 0 nr 28672 ram 49152 >> [17608.403409] item 64 key (264 108 98304) itemoff 12326 itemsize 53 >> [17608.405389] extent data disk bytenr 14491648 nr 20480 >> [17608.407111] extent data offset 0 nr 8192 ram 20480 >> [17608.408795] item 65 key (264 108 106496) itemoff 12273 itemsize 53 >> [17608.410757] extent data disk bytenr 14561280 nr 53248 >> [17608.412483] extent data offset 0 nr 53248 ram 53248 >> [17608.414166] item 66 key (264 108 159744) itemoff 12220 itemsize 53 >> [17608.416164] extent data disk bytenr 13631488 nr 262144 >> [17608.417897] extent data offset 159744 nr 16384 ram 262144 >> [17608.419677] item 67 key (264 108 176128) itemoff 12167 itemsize 53 >> [17608.421598] extent data disk bytenr 13631488 nr 262144 >> [17608.423313] extent data offset 237568 nr 4096 ram 262144 >> [17608.425051] item 68 key (264 108 180224) itemoff 12114 itemsize 53 >> [17608.427009] extent data disk bytenr 13631488 nr 262144 >> [17608.428787] extent data offset 180224 nr 45056 ram 262144 >> [17608.430588] item 69 key (264 108 225280) itemoff 12061 itemsize 53 >> [17608.432457] extent data disk bytenr 14614528 nr 16384 >> [17608.434122] extent data offset 0 nr 16384 ram 16384 >> [17608.435663] item 70 key (264 108 241664) itemoff 12008 itemsize 53 >> [17608.437520] extent data disk bytenr 13631488 nr 262144 >> [17608.439160] extent data offset 241664 nr 8192 ram 262144 >> [17608.440869] item 71 key (265 1 0) itemoff 11848 itemsize 160 >> [17608.442586] inode generation 6 size 193781 mode 100600 >> [17608.444197] item 72 key (265 12 256) itemoff 11829 itemsize 19 >> [17608.446043] item 73 key (265 108 0) itemoff 11776 itemsize 53 >> [17608.447726] extent data disk bytenr 14524416 nr 4096 >> [17608.449507] extent data offset 0 nr 4096 ram 4096 >> [17608.451142] item 74 key (265 108 4096) itemoff 11723 itemsize 53 >> [17608.452883] extent data disk bytenr 13631488 nr 262144 >> [17608.454517] extent data offset 0 nr 4096 ram 262144 >> [17608.456136] item 75 key (265 108 8192) itemoff 11670 itemsize 53 >> [17608.458065] extent data disk bytenr 13631488 nr 262144 >> [17608.459732] extent data offset 8192 nr 110592 ram 262144 >> [17608.461260] item 76 key (265 108 118784) itemoff 11617 itemsize 53 >> [17608.462918] extent data disk bytenr 14528512 nr 28672 >> [17608.464562] extent data offset 0 nr 28672 ram 28672 >> [17608.466204] item 77 key (265 108 147456) itemoff 11564 itemsize 53 >> [17608.468161] extent data disk bytenr 0 nr 0 >> [17608.469631] extent data offset 8192 nr 4096 ram 12288 >> [17608.471331] item 78 key (265 108 151552) itemoff 11511 itemsize 53 >> [17608.473363] extent data disk bytenr 14557184 nr 4096 >> [17608.474920] extent data offset 0 nr 4096 ram 4096 >> [17608.476499] item 79 key (265 108 155648) itemoff 11458 itemsize 53 >> [17608.478306] extent data disk bytenr 13631488 nr 262144 >> [17608.479921] extent data offset 155648 nr 40960 ram 262144 >> [17608.481652] item 80 key (266 1 0) itemoff 11298 itemsize 160 >> [17608.483268] inode generation 6 size 262144 mode 100600 >> [17608.484849] item 81 key (266 12 256) itemoff 11279 itemsize 19 >> [17608.486489] item 82 key (266 108 0) itemoff 11226 itemsize 53 >> [17608.488298] extent data disk bytenr 14712832 nr 4096 >> [17608.489866] extent data offset 0 nr 4096 ram 4096 >> [17608.491399] item 83 key (266 108 4096) itemoff 11173 itemsize 53 >> [17608.493255] extent data disk bytenr 13631488 nr 262144 >> [17608.494898] extent data offset 0 nr 4096 ram 262144 >> [17608.496504] item 84 key (266 108 8192) itemoff 11120 itemsize 53 >> [17608.498434] extent data disk bytenr 13631488 nr 262144 >> [17608.500033] extent data offset 8192 nr 73728 ram 262144 >> [17608.501619] item 85 key (266 108 81920) itemoff 11067 itemsize 53 >> [17608.503455] extent data disk bytenr 14741504 nr 4096 >> [17608.505086] extent data offset 0 nr 4096 ram 4096 >> [17608.506648] item 86 key (266 108 86016) itemoff 11014 itemsize 53 >> [17608.508346] extent data disk bytenr 13631488 nr 262144 >> [17608.509862] extent data offset 86016 nr 28672 ram 262144 >> [17608.511645] item 87 key (266 108 114688) itemoff 10961 itemsize 53 >> [17608.513620] extent data disk bytenr 13631488 nr 262144 >> [17608.515403] extent data offset 36864 nr 28672 ram 262144 >> [17608.517234] item 88 key (266 108 143360) itemoff 10908 itemsize 53 >> [17608.519248] extent data disk bytenr 13631488 nr 262144 >> [17608.520976] extent data offset 143360 nr 16384 ram 262144 >> [17608.522635] item 89 key (266 108 159744) itemoff 10855 itemsize 53 >> [17608.524583] extent data disk bytenr 14745600 nr 12288 >> [17608.526254] extent data offset 0 nr 12288 ram 12288 >> [17608.527858] item 90 key (266 108 172032) itemoff 10802 itemsize 53 >> [17608.538634] extent data disk bytenr 13631488 nr 262144 >> [17608.540392] extent data offset 172032 nr 36864 ram 262144 >> [17608.542156] item 91 key (266 108 208896) itemoff 10749 itemsize 53 >> [17608.544104] extent data disk bytenr 14716928 nr 4096 >> [17608.545664] extent data offset 0 nr 4096 ram 4096 >> [17608.547207] item 92 key (266 108 212992) itemoff 10696 itemsize 53 >> [17608.549038] extent data disk bytenr 0 nr 0 >> [17608.550440] extent data offset 0 nr 32768 ram 32768 >> [17608.552056] item 93 key (266 108 245760) itemoff 10643 itemsize 53 >> [17608.553997] extent data disk bytenr 14721024 nr 4096 >> [17608.555578] extent data offset 0 nr 4096 ram 4096 >> [17608.557113] item 94 key (266 108 249856) itemoff 10590 itemsize 53 >> [17608.558861] extent data disk bytenr 14688256 nr 24576 >> [17608.560439] extent data offset 12288 nr 12288 ram 24576 >> [17608.562137] item 95 key (267 1 0) itemoff 10430 itemsize 160 >> [17608.563756] inode generation 6 size 262144 mode 100600 >> [17608.565261] item 96 key (267 12 256) itemoff 10411 itemsize 19 >> [17608.566912] item 97 key (267 108 0) itemoff 10358 itemsize 53 >> [17608.568715] extent data disk bytenr 14737408 nr 4096 >> [17608.570283] extent data offset 0 nr 4096 ram 4096 >> [17608.571810] item 98 key (267 108 4096) itemoff 10305 itemsize 53 >> [17608.573502] extent data disk bytenr 13631488 nr 262144 >> [17608.575033] extent data offset 0 nr 4096 ram 262144 >> [17608.576507] item 99 key (267 108 8192) itemoff 10252 itemsize 53 >> [17608.578416] extent data disk bytenr 13631488 nr 262144 >> [17608.581054] extent data offset 8192 nr 167936 ram 262144 >> [17608.583613] item 100 key (267 108 176128) itemoff 10199 itemsize 53 >> [17608.585846] extent data disk bytenr 13631488 nr 262144 >> [17608.587883] extent data offset 188416 nr 4096 ram 262144 >> [17608.590034] item 101 key (267 108 180224) itemoff 10146 itemsize 53 >> [17608.592397] extent data disk bytenr 13631488 nr 262144 >> [17608.594415] extent data offset 180224 nr 81920 ram 262144 >> [17608.595941] item 102 key (18446744073709551606 128 13631488) itemoff 10142 itemsize 4 >> [17608.598465] item 103 key (18446744073709551606 128 13635584) itemoff 10082 itemsize 60 >> [17608.601262] item 104 key (18446744073709551606 128 13697024) itemoff 9974 itemsize 108 >> [17608.604090] item 105 key (18446744073709551606 128 13717504) itemoff 9946 itemsize 28 >> [17608.606573] item 106 key (18446744073709551606 128 13774848) itemoff 9930 itemsize 16 >> [17608.609125] item 107 key (18446744073709551606 128 13803520) itemoff 9842 itemsize 88 >> [17608.611709] item 108 key (18446744073709551606 128 13905920) itemoff 9786 itemsize 56 >> [17608.614269] item 109 key (18446744073709551606 128 13975552) itemoff 9782 itemsize 4 >> [17608.616992] item 110 key (18446744073709551606 128 14123008) itemoff 9778 itemsize 4 >> [17608.619523] item 111 key (18446744073709551606 128 14135296) itemoff 9710 itemsize 68 >> [17608.622144] item 112 key (18446744073709551606 128 14209024) itemoff 9706 itemsize 4 >> [17608.624841] item 113 key (18446744073709551606 128 14217216) itemoff 9538 itemsize 168 >> [17608.627409] item 114 key (18446744073709551606 128 14471168) itemoff 9530 itemsize 8 >> [17608.630034] item 115 key (18446744073709551606 128 14479360) itemoff 9510 itemsize 20 >> [17608.632681] item 116 key (18446744073709551606 128 14499840) itemoff 9498 itemsize 12 >> [17608.635448] item 117 key (18446744073709551606 128 14524416) itemoff 9394 itemsize 104 >> [17608.638122] item 118 key (18446744073709551606 128 14700544) itemoff 9382 itemsize 12 >> [17608.640633] item 119 key (18446744073709551606 128 14712832) itemoff 9370 itemsize 12 >> [17608.643227] item 120 key (18446744073709551606 128 14737408) itemoff 9350 itemsize 20 >> [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected >> [17608.648714] ------------[ cut here ]------------ >> [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] >> [17608.652972] Modules linked in: dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_writes dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [last unloaded: scsi_debug] >> [17608.660683] CPU: 2 PID: 27162 Comm: fsx Tainted: G L 5.7.0-rc3-default+ #1094 >> [17608.663520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 >> [17608.667102] RIP: 0010:btree_csum_one_bio+0x297/0x2a0 [btrfs] >> [17608.668919] Code: bc fd ff ff e8 aa e7 f0 c2 31 f6 48 89 3c 24 e8 ff 7b ff ff 48 8b 3c 24 48 c7 c6 78 75 25 c0 48 8b 17 4c 89 e7 e8 84 bc 0b 00 <0f> 0b e9 8f fd ff ff 66 90 0f 1f 44 00 00 48 89 f7 e9 53 fd ff ff >> [17608.674610] RSP: 0018:ffffafbb81e978b8 EFLAGS: 00010292 >> [17608.676299] RAX: 0000000000000000 RBX: ffff94287f810f00 RCX: 0000000000000001 >> [17608.678260] RDX: 0000000000000000 RSI: ffffffff83109fc2 RDI: ffffffff8310a096 >> [17608.680484] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 >> [17608.682645] R10: 0000000000000000 R11: 0000000000008ca0 R12: ffff94285f298000 >> [17608.684681] R13: ffff94274100bd30 R14: 0000000000000000 R15: 00000000ffffff8b >> [17608.686659] FS: 00007f3479617b80(0000) GS:ffff94287ba00000(0000) knlGS:0000000000000000 >> [17608.689240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [17608.691150] CR2: 0000000000466000 CR3: 000000010227f001 CR4: 0000000000160ee0 >> [17608.693354] Call Trace: >> [17608.694341] btree_submit_bio_hook+0x74/0xc0 [btrfs] >> [17608.696162] submit_one_bio+0x2b/0x40 [btrfs] >> [17608.698391] btree_write_cache_pages+0x373/0x440 [btrfs] >> [17608.700085] ? lock_acquire+0xa3/0x400 >> [17608.701257] ? __filemap_fdatawrite_range+0xad/0x110 >> [17608.702736] do_writepages+0x40/0xe0 >> [17608.703953] ? do_raw_spin_unlock+0x4b/0xc0 >> [17608.705290] ? _raw_spin_unlock+0x1f/0x30 >> [17608.706571] ? wbc_attach_and_unlock_inode+0x194/0x2a0 >> [17608.708167] __filemap_fdatawrite_range+0xce/0x110 >> [17608.709679] btrfs_write_marked_extents+0x68/0x160 [btrfs] >> [17608.711384] btrfs_sync_log+0x1a8/0xd40 [btrfs] >> [17608.712892] ? btrfs_log_inode+0x370/0xe20 [btrfs] >> [17608.714459] ? lock_acquire+0xa3/0x400 >> [17608.715803] ? lockref_put_or_lock+0x9/0x30 >> [17608.717115] ? dput+0x20/0x490 >> [17608.718172] ? dput+0x20/0x490 >> [17608.719271] ? do_raw_spin_unlock+0x4b/0xc0 >> [17608.720639] ? _raw_spin_unlock+0x1f/0x30 >> [17608.722002] btrfs_sync_file+0x335/0x490 [btrfs] >> [17608.723610] __x64_sys_msync+0x19e/0x200 >> [17608.724924] do_syscall_64+0x50/0x210 >> [17608.726143] entry_SYSCALL_64_after_hwframe+0x49/0xb3 >> [17608.727719] RIP: 0033:0x7f34797f2743 >> [17608.729030] Code: 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 1a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48 >> [17608.734572] RSP: 002b:00007ffe84f11858 EFLAGS: 00000246 ORIG_RAX: 000000000000001a >> [17608.736913] RAX: ffffffffffffffda RBX: 00000000000008c8 RCX: 00007f34797f2743 >> [17608.738902] RDX: 0000000000000004 RSI: 000000000000293f RDI: 00007f3479814000 >> [17608.740949] RBP: 00000000000278c8 R08: 000000000000001f R09: 00007f3479814900 >> [17608.742896] R10: fffffffffffffd0c R11: 0000000000000246 R12: 0000000000002077 >> [17608.744810] R13: 000000000000293f R14: 00007f3479814000 R15: 0000000000000000 >> [17608.746764] irq event stamp: 0 >> [17608.747948] hardirqs last enabled at (0): [<0000000000000000>] 0x0 >> [17608.749754] hardirqs last disabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 >> [17608.752428] softirqs last enabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 >> [17608.754935] softirqs last disabled at (0): [<0000000000000000>] 0x0 >> [17608.756871] ---[ end trace ddba49bd36651725 ]--- >> [17608.758590] BTRFS: error (device dm-0) in btrfs_sync_log:3089: errno=-5 IO failure >> > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: write-time leaf corruption detected 2020-05-13 3:10 ` Qu Wenruo 2020-05-13 3:17 ` Qu Wenruo @ 2020-05-13 9:25 ` Filipe Manana 1 sibling, 0 replies; 24+ messages in thread From: Filipe Manana @ 2020-05-13 9:25 UTC (permalink / raw) To: Qu Wenruo; +Cc: David Sterba, linux-btrfs, Filipe Manana On Wed, May 13, 2020 at 4:12 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > > > > On 2020/5/12 下午10:15, David Sterba wrote: > > Happened once in a VM, defaul mkfs/mount options. > > It looks like it's pretty hard to reproduce. > > > > > generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 > > [17607.625840] BTRFS: device fsid ae51523a-71e7-4730-9097-faeca36779be devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (27036) > > [17607.664903] BTRFS info (device vdb): disk space caching is enabled > > [17607.667422] BTRFS info (device vdb): has skinny extents > > [17607.669552] BTRFS info (device vdb): flagging fs with big metadata feature > > [17607.681613] BTRFS info (device vdb): checking UUID tree > > [17607.921588] BTRFS: device fsid 16238d8a-1af3-4fa1-b3bc-fd6f04469192 devid 1 transid 5 /dev/mapper/logwrites-test scanned by mkfs.btrfs (27101) > > [17607.948085] BTRFS info (device dm-0): turning on sync discard > > [17607.952647] BTRFS info (device dm-0): disk space caching is enabled > > [17607.955088] BTRFS info (device dm-0): has skinny extents > > [17607.956924] BTRFS info (device dm-0): flagging fs with big metadata feature > > [17607.966128] BTRFS info (device dm-0): checking UUID tree > > [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item > > [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 > > [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 > > [17608.106426] item 0 key (257 1 0) itemoff 16123 itemsize 160 > > Since we have 257 INODE_ITEM here, I guess that's all the log tree have. > > And surprisingly, the csum range around 1371000 doesn't even get > referred in this log tree. > > Filipe, any idea on how this could happen? Yes, and I replied yesterday: https://lore.kernel.org/linux-btrfs/CAL3q7H4EwU9ZwKxXuZmSpVVPmUMvaCX3peizROX4Obr0xug0pw@mail.gmail.com/ > > Thanks, > Qu > > [17608.108766] inode generation 6 size 262144 mode 100600 > > [17608.110947] item 1 key (257 12 256) itemoff 16103 itemsize 20 > > [17608.112698] item 2 key (257 108 0) itemoff 16050 itemsize 53 > > [17608.114568] extent data disk bytenr 13631488 nr 262144 > > [17608.116352] extent data offset 0 nr 262144 ram 262144 > > [17608.118082] item 3 key (258 1 0) itemoff 15890 itemsize 160 > > [17608.119920] inode generation 6 size 262144 mode 100600 > > [17608.121605] item 4 key (258 12 256) itemoff 15871 itemsize 19 > > [17608.123293] item 5 key (258 108 0) itemoff 15818 itemsize 53 > > [17608.125212] extent data disk bytenr 13905920 nr 4096 > > [17608.126851] extent data offset 0 nr 4096 ram 4096 > > [17608.128342] item 6 key (258 108 4096) itemoff 15765 itemsize 53 > > [17608.130106] extent data disk bytenr 13631488 nr 262144 > > [17608.131774] extent data offset 0 nr 4096 ram 262144 > > [17608.133330] item 7 key (258 108 8192) itemoff 15712 itemsize 53 > > [17608.135151] extent data disk bytenr 13631488 nr 262144 > > [17608.136680] extent data offset 8192 nr 8192 ram 262144 > > [17608.137931] item 8 key (258 108 16384) itemoff 15659 itemsize 53 > > [17608.139257] extent data disk bytenr 13631488 nr 262144 > > [17608.140939] extent data offset 204800 nr 16384 ram 262144 > > [17608.142255] item 9 key (258 108 32768) itemoff 15606 itemsize 53 > > [17608.143941] extent data disk bytenr 13910016 nr 28672 > > [17608.145576] extent data offset 0 nr 28672 ram 28672 > > [17608.147165] item 10 key (258 108 61440) itemoff 15553 itemsize 53 > > [17608.148964] extent data disk bytenr 13631488 nr 262144 > > [17608.150563] extent data offset 249856 nr 8192 ram 262144 > > [17608.152204] item 11 key (258 108 69632) itemoff 15500 itemsize 53 > > [17608.153974] extent data disk bytenr 13631488 nr 262144 > > [17608.155588] extent data offset 69632 nr 126976 ram 262144 > > [17608.157099] item 12 key (258 108 196608) itemoff 15447 itemsize 53 > > [17608.158698] extent data disk bytenr 13938688 nr 4096 > > [17608.160123] extent data offset 0 nr 4096 ram 4096 > > [17608.161300] item 13 key (258 108 200704) itemoff 15394 itemsize 53 > > [17608.162597] extent data disk bytenr 0 nr 0 > > [17608.164057] extent data offset 0 nr 28672 ram 28672 > > [17608.165782] item 14 key (258 108 229376) itemoff 15341 itemsize 53 > > [17608.167691] extent data disk bytenr 13942784 nr 4096 > > [17608.169259] extent data offset 0 nr 4096 ram 4096 > > [17608.170821] item 15 key (258 108 233472) itemoff 15288 itemsize 53 > > [17608.172687] extent data disk bytenr 13631488 nr 262144 > > [17608.174310] extent data offset 233472 nr 12288 ram 262144 > > [17608.175993] item 16 key (258 108 245760) itemoff 15235 itemsize 53 > > [17608.177821] extent data disk bytenr 13946880 nr 16384 > > [17608.179440] extent data offset 0 nr 16384 ram 16384 > > [17608.181043] item 17 key (259 1 0) itemoff 15075 itemsize 160 > > [17608.182796] inode generation 6 size 262144 mode 100600 > > [17608.184424] item 18 key (259 12 256) itemoff 15056 itemsize 19 > > [17608.186085] item 19 key (259 108 0) itemoff 15003 itemsize 53 > > [17608.187859] extent data disk bytenr 13975552 nr 4096 > > [17608.189525] extent data offset 0 nr 4096 ram 4096 > > [17608.191074] item 20 key (259 108 4096) itemoff 14950 itemsize 53 > > [17608.192816] extent data disk bytenr 13631488 nr 262144 > > [17608.194464] extent data offset 0 nr 4096 ram 262144 > > [17608.196206] item 21 key (259 108 8192) itemoff 14897 itemsize 53 > > [17608.198212] extent data disk bytenr 13631488 nr 262144 > > [17608.200048] extent data offset 8192 nr 106496 ram 262144 > > [17608.201684] item 22 key (259 108 114688) itemoff 14844 itemsize 53 > > [17608.203659] extent data disk bytenr 14217216 nr 49152 > > [17608.205398] extent data offset 0 nr 49152 ram 49152 > > [17608.207126] item 23 key (259 108 163840) itemoff 14791 itemsize 53 > > [17608.209019] extent data disk bytenr 13631488 nr 262144 > > [17608.210736] extent data offset 163840 nr 98304 ram 262144 > > [17608.212371] item 24 key (261 1 0) itemoff 14631 itemsize 160 > > [17608.222944] inode generation 6 size 262144 mode 100600 > > [17608.224637] item 25 key (261 12 256) itemoff 14612 itemsize 19 > > [17608.226400] item 26 key (261 108 0) itemoff 14559 itemsize 53 > > [17608.228169] extent data disk bytenr 14123008 nr 4096 > > [17608.229794] extent data offset 0 nr 4096 ram 4096 > > [17608.231292] item 27 key (261 108 4096) itemoff 14506 itemsize 53 > > [17608.233203] extent data disk bytenr 13631488 nr 262144 > > [17608.234794] extent data offset 0 nr 4096 ram 262144 > > [17608.236362] item 28 key (261 108 8192) itemoff 14453 itemsize 53 > > [17608.238102] extent data disk bytenr 13631488 nr 262144 > > [17608.239782] extent data offset 8192 nr 16384 ram 262144 > > [17608.241587] item 29 key (261 108 24576) itemoff 14400 itemsize 53 > > [17608.243431] extent data disk bytenr 13631488 nr 262144 > > [17608.245176] extent data offset 212992 nr 36864 ram 262144 > > [17608.246991] item 30 key (261 108 61440) itemoff 14347 itemsize 53 > > [17608.248868] extent data disk bytenr 13631488 nr 262144 > > [17608.250456] extent data offset 61440 nr 114688 ram 262144 > > [17608.252333] item 31 key (261 108 176128) itemoff 14294 itemsize 53 > > [17608.254343] extent data disk bytenr 14135296 nr 61440 > > [17608.256131] extent data offset 0 nr 61440 ram 61440 > > [17608.257857] item 32 key (261 108 237568) itemoff 14241 itemsize 53 > > [17608.259864] extent data disk bytenr 13631488 nr 262144 > > [17608.261484] extent data offset 237568 nr 24576 ram 262144 > > [17608.263134] item 33 key (262 1 0) itemoff 14081 itemsize 160 > > [17608.264865] inode generation 6 size 178311 mode 100600 > > [17608.266625] item 34 key (262 12 256) itemoff 14062 itemsize 19 > > [17608.268551] item 35 key (262 108 0) itemoff 14009 itemsize 53 > > [17608.270363] extent data disk bytenr 14196736 nr 4096 > > [17608.272019] extent data offset 0 nr 4096 ram 4096 > > [17608.273505] item 36 key (262 108 4096) itemoff 13956 itemsize 53 > > [17608.275387] extent data disk bytenr 13631488 nr 262144 > > [17608.277125] extent data offset 0 nr 4096 ram 262144 > > [17608.278780] item 37 key (262 108 8192) itemoff 13903 itemsize 53 > > [17608.280507] extent data disk bytenr 13631488 nr 262144 > > [17608.282163] extent data offset 8192 nr 106496 ram 262144 > > [17608.283866] item 38 key (262 108 114688) itemoff 13850 itemsize 53 > > [17608.285727] extent data disk bytenr 14200832 nr 4096 > > [17608.287264] extent data offset 0 nr 4096 ram 4096 > > [17608.288719] item 39 key (262 108 118784) itemoff 13797 itemsize 53 > > [17608.290696] extent data disk bytenr 0 nr 0 > > [17608.292237] extent data offset 0 nr 49152 ram 49152 > > [17608.293667] item 40 key (262 108 167936) itemoff 13744 itemsize 53 > > [17608.295436] extent data disk bytenr 14209024 nr 4096 > > [17608.297051] extent data offset 0 nr 4096 ram 4096 > > [17608.298583] item 41 key (262 108 172032) itemoff 13691 itemsize 53 > > [17608.300573] extent data disk bytenr 13631488 nr 262144 > > [17608.302323] extent data offset 172032 nr 8192 ram 262144 > > [17608.304112] item 42 key (263 1 0) itemoff 13531 itemsize 160 > > [17608.305810] inode generation 6 size 262144 mode 100600 > > [17608.307444] item 43 key (263 12 256) itemoff 13512 itemsize 19 > > [17608.309108] item 44 key (263 108 0) itemoff 13459 itemsize 53 > > [17608.310760] extent data disk bytenr 14266368 nr 4096 > > [17608.312269] extent data offset 0 nr 4096 ram 4096 > > [17608.313883] item 45 key (263 108 4096) itemoff 13406 itemsize 53 > > [17608.315659] extent data disk bytenr 13631488 nr 262144 > > [17608.317133] extent data offset 0 nr 4096 ram 262144 > > [17608.318691] item 46 key (263 108 8192) itemoff 13353 itemsize 53 > > [17608.320417] extent data disk bytenr 13631488 nr 262144 > > [17608.321851] extent data offset 8192 nr 81920 ram 262144 > > [17608.323351] item 47 key (263 108 90112) itemoff 13300 itemsize 53 > > [17608.325222] extent data disk bytenr 14319616 nr 4096 > > [17608.326968] extent data offset 0 nr 4096 ram 4096 > > [17608.328583] item 48 key (263 108 94208) itemoff 13247 itemsize 53 > > [17608.330364] extent data disk bytenr 0 nr 0 > > [17608.331708] extent data offset 0 nr 45056 ram 45056 > > [17608.333200] item 49 key (263 108 139264) itemoff 13194 itemsize 53 > > [17608.334979] extent data disk bytenr 14323712 nr 4096 > > [17608.336570] extent data offset 0 nr 4096 ram 4096 > > [17608.338023] item 50 key (263 108 143360) itemoff 13141 itemsize 53 > > [17608.340091] extent data disk bytenr 13631488 nr 262144 > > [17608.341746] extent data offset 143360 nr 8192 ram 262144 > > [17608.343400] item 51 key (263 108 151552) itemoff 13088 itemsize 53 > > [17608.345228] extent data disk bytenr 14327808 nr 61440 > > [17608.346924] extent data offset 0 nr 61440 ram 61440 > > [17608.348626] item 52 key (263 108 212992) itemoff 13035 itemsize 53 > > [17608.350371] extent data disk bytenr 14270464 nr 49152 > > [17608.351985] extent data offset 0 nr 49152 ram 49152 > > [17608.353542] item 53 key (264 1 0) itemoff 12875 itemsize 160 > > [17608.355125] inode generation 6 size 247974 mode 100600 > > [17608.356813] item 54 key (264 12 256) itemoff 12856 itemsize 19 > > [17608.358539] item 55 key (264 108 0) itemoff 12803 itemsize 53 > > [17608.360340] extent data disk bytenr 14471168 nr 4096 > > [17608.361935] extent data offset 0 nr 4096 ram 4096 > > [17608.363426] item 56 key (264 108 4096) itemoff 12750 itemsize 53 > > [17608.365224] extent data disk bytenr 13631488 nr 262144 > > [17608.366836] extent data offset 0 nr 4096 ram 262144 > > [17608.368439] item 57 key (264 108 8192) itemoff 12697 itemsize 53 > > [17608.370226] extent data disk bytenr 13631488 nr 262144 > > [17608.371714] extent data offset 8192 nr 8192 ram 262144 > > [17608.373157] item 58 key (264 108 16384) itemoff 12644 itemsize 53 > > [17608.374820] extent data disk bytenr 14483456 nr 4096 > > [17608.376495] extent data offset 0 nr 4096 ram 4096 > > [17608.378014] item 59 key (264 108 20480) itemoff 12591 itemsize 53 > > [17608.379882] extent data disk bytenr 0 nr 0 > > [17608.381425] extent data offset 0 nr 20480 ram 20480 > > [17608.383188] item 60 key (264 108 40960) itemoff 12538 itemsize 53 > > [17608.385037] extent data disk bytenr 14487552 nr 4096 > > [17608.386644] extent data offset 0 nr 4096 ram 4096 > > [17608.388191] item 61 key (264 108 45056) itemoff 12485 itemsize 53 > > [17608.390035] extent data disk bytenr 13631488 nr 262144 > > [17608.391634] extent data offset 45056 nr 20480 ram 262144 > > [17608.393440] item 62 key (264 108 65536) itemoff 12432 itemsize 53 > > [17608.395464] extent data disk bytenr 14475264 nr 4096 > > [17608.397179] extent data offset 0 nr 4096 ram 4096 > > [17608.398675] item 63 key (264 108 69632) itemoff 12379 itemsize 53 > > [17608.400412] extent data disk bytenr 0 nr 0 > > [17608.401843] extent data offset 0 nr 28672 ram 49152 > > [17608.403409] item 64 key (264 108 98304) itemoff 12326 itemsize 53 > > [17608.405389] extent data disk bytenr 14491648 nr 20480 > > [17608.407111] extent data offset 0 nr 8192 ram 20480 > > [17608.408795] item 65 key (264 108 106496) itemoff 12273 itemsize 53 > > [17608.410757] extent data disk bytenr 14561280 nr 53248 > > [17608.412483] extent data offset 0 nr 53248 ram 53248 > > [17608.414166] item 66 key (264 108 159744) itemoff 12220 itemsize 53 > > [17608.416164] extent data disk bytenr 13631488 nr 262144 > > [17608.417897] extent data offset 159744 nr 16384 ram 262144 > > [17608.419677] item 67 key (264 108 176128) itemoff 12167 itemsize 53 > > [17608.421598] extent data disk bytenr 13631488 nr 262144 > > [17608.423313] extent data offset 237568 nr 4096 ram 262144 > > [17608.425051] item 68 key (264 108 180224) itemoff 12114 itemsize 53 > > [17608.427009] extent data disk bytenr 13631488 nr 262144 > > [17608.428787] extent data offset 180224 nr 45056 ram 262144 > > [17608.430588] item 69 key (264 108 225280) itemoff 12061 itemsize 53 > > [17608.432457] extent data disk bytenr 14614528 nr 16384 > > [17608.434122] extent data offset 0 nr 16384 ram 16384 > > [17608.435663] item 70 key (264 108 241664) itemoff 12008 itemsize 53 > > [17608.437520] extent data disk bytenr 13631488 nr 262144 > > [17608.439160] extent data offset 241664 nr 8192 ram 262144 > > [17608.440869] item 71 key (265 1 0) itemoff 11848 itemsize 160 > > [17608.442586] inode generation 6 size 193781 mode 100600 > > [17608.444197] item 72 key (265 12 256) itemoff 11829 itemsize 19 > > [17608.446043] item 73 key (265 108 0) itemoff 11776 itemsize 53 > > [17608.447726] extent data disk bytenr 14524416 nr 4096 > > [17608.449507] extent data offset 0 nr 4096 ram 4096 > > [17608.451142] item 74 key (265 108 4096) itemoff 11723 itemsize 53 > > [17608.452883] extent data disk bytenr 13631488 nr 262144 > > [17608.454517] extent data offset 0 nr 4096 ram 262144 > > [17608.456136] item 75 key (265 108 8192) itemoff 11670 itemsize 53 > > [17608.458065] extent data disk bytenr 13631488 nr 262144 > > [17608.459732] extent data offset 8192 nr 110592 ram 262144 > > [17608.461260] item 76 key (265 108 118784) itemoff 11617 itemsize 53 > > [17608.462918] extent data disk bytenr 14528512 nr 28672 > > [17608.464562] extent data offset 0 nr 28672 ram 28672 > > [17608.466204] item 77 key (265 108 147456) itemoff 11564 itemsize 53 > > [17608.468161] extent data disk bytenr 0 nr 0 > > [17608.469631] extent data offset 8192 nr 4096 ram 12288 > > [17608.471331] item 78 key (265 108 151552) itemoff 11511 itemsize 53 > > [17608.473363] extent data disk bytenr 14557184 nr 4096 > > [17608.474920] extent data offset 0 nr 4096 ram 4096 > > [17608.476499] item 79 key (265 108 155648) itemoff 11458 itemsize 53 > > [17608.478306] extent data disk bytenr 13631488 nr 262144 > > [17608.479921] extent data offset 155648 nr 40960 ram 262144 > > [17608.481652] item 80 key (266 1 0) itemoff 11298 itemsize 160 > > [17608.483268] inode generation 6 size 262144 mode 100600 > > [17608.484849] item 81 key (266 12 256) itemoff 11279 itemsize 19 > > [17608.486489] item 82 key (266 108 0) itemoff 11226 itemsize 53 > > [17608.488298] extent data disk bytenr 14712832 nr 4096 > > [17608.489866] extent data offset 0 nr 4096 ram 4096 > > [17608.491399] item 83 key (266 108 4096) itemoff 11173 itemsize 53 > > [17608.493255] extent data disk bytenr 13631488 nr 262144 > > [17608.494898] extent data offset 0 nr 4096 ram 262144 > > [17608.496504] item 84 key (266 108 8192) itemoff 11120 itemsize 53 > > [17608.498434] extent data disk bytenr 13631488 nr 262144 > > [17608.500033] extent data offset 8192 nr 73728 ram 262144 > > [17608.501619] item 85 key (266 108 81920) itemoff 11067 itemsize 53 > > [17608.503455] extent data disk bytenr 14741504 nr 4096 > > [17608.505086] extent data offset 0 nr 4096 ram 4096 > > [17608.506648] item 86 key (266 108 86016) itemoff 11014 itemsize 53 > > [17608.508346] extent data disk bytenr 13631488 nr 262144 > > [17608.509862] extent data offset 86016 nr 28672 ram 262144 > > [17608.511645] item 87 key (266 108 114688) itemoff 10961 itemsize 53 > > [17608.513620] extent data disk bytenr 13631488 nr 262144 > > [17608.515403] extent data offset 36864 nr 28672 ram 262144 > > [17608.517234] item 88 key (266 108 143360) itemoff 10908 itemsize 53 > > [17608.519248] extent data disk bytenr 13631488 nr 262144 > > [17608.520976] extent data offset 143360 nr 16384 ram 262144 > > [17608.522635] item 89 key (266 108 159744) itemoff 10855 itemsize 53 > > [17608.524583] extent data disk bytenr 14745600 nr 12288 > > [17608.526254] extent data offset 0 nr 12288 ram 12288 > > [17608.527858] item 90 key (266 108 172032) itemoff 10802 itemsize 53 > > [17608.538634] extent data disk bytenr 13631488 nr 262144 > > [17608.540392] extent data offset 172032 nr 36864 ram 262144 > > [17608.542156] item 91 key (266 108 208896) itemoff 10749 itemsize 53 > > [17608.544104] extent data disk bytenr 14716928 nr 4096 > > [17608.545664] extent data offset 0 nr 4096 ram 4096 > > [17608.547207] item 92 key (266 108 212992) itemoff 10696 itemsize 53 > > [17608.549038] extent data disk bytenr 0 nr 0 > > [17608.550440] extent data offset 0 nr 32768 ram 32768 > > [17608.552056] item 93 key (266 108 245760) itemoff 10643 itemsize 53 > > [17608.553997] extent data disk bytenr 14721024 nr 4096 > > [17608.555578] extent data offset 0 nr 4096 ram 4096 > > [17608.557113] item 94 key (266 108 249856) itemoff 10590 itemsize 53 > > [17608.558861] extent data disk bytenr 14688256 nr 24576 > > [17608.560439] extent data offset 12288 nr 12288 ram 24576 > > [17608.562137] item 95 key (267 1 0) itemoff 10430 itemsize 160 > > [17608.563756] inode generation 6 size 262144 mode 100600 > > [17608.565261] item 96 key (267 12 256) itemoff 10411 itemsize 19 > > [17608.566912] item 97 key (267 108 0) itemoff 10358 itemsize 53 > > [17608.568715] extent data disk bytenr 14737408 nr 4096 > > [17608.570283] extent data offset 0 nr 4096 ram 4096 > > [17608.571810] item 98 key (267 108 4096) itemoff 10305 itemsize 53 > > [17608.573502] extent data disk bytenr 13631488 nr 262144 > > [17608.575033] extent data offset 0 nr 4096 ram 262144 > > [17608.576507] item 99 key (267 108 8192) itemoff 10252 itemsize 53 > > [17608.578416] extent data disk bytenr 13631488 nr 262144 > > [17608.581054] extent data offset 8192 nr 167936 ram 262144 > > [17608.583613] item 100 key (267 108 176128) itemoff 10199 itemsize 53 > > [17608.585846] extent data disk bytenr 13631488 nr 262144 > > [17608.587883] extent data offset 188416 nr 4096 ram 262144 > > [17608.590034] item 101 key (267 108 180224) itemoff 10146 itemsize 53 > > [17608.592397] extent data disk bytenr 13631488 nr 262144 > > [17608.594415] extent data offset 180224 nr 81920 ram 262144 > > [17608.595941] item 102 key (18446744073709551606 128 13631488) itemoff 10142 itemsize 4 > > [17608.598465] item 103 key (18446744073709551606 128 13635584) itemoff 10082 itemsize 60 > > [17608.601262] item 104 key (18446744073709551606 128 13697024) itemoff 9974 itemsize 108 > > [17608.604090] item 105 key (18446744073709551606 128 13717504) itemoff 9946 itemsize 28 > > [17608.606573] item 106 key (18446744073709551606 128 13774848) itemoff 9930 itemsize 16 > > [17608.609125] item 107 key (18446744073709551606 128 13803520) itemoff 9842 itemsize 88 > > [17608.611709] item 108 key (18446744073709551606 128 13905920) itemoff 9786 itemsize 56 > > [17608.614269] item 109 key (18446744073709551606 128 13975552) itemoff 9782 itemsize 4 > > [17608.616992] item 110 key (18446744073709551606 128 14123008) itemoff 9778 itemsize 4 > > [17608.619523] item 111 key (18446744073709551606 128 14135296) itemoff 9710 itemsize 68 > > [17608.622144] item 112 key (18446744073709551606 128 14209024) itemoff 9706 itemsize 4 > > [17608.624841] item 113 key (18446744073709551606 128 14217216) itemoff 9538 itemsize 168 > > [17608.627409] item 114 key (18446744073709551606 128 14471168) itemoff 9530 itemsize 8 > > [17608.630034] item 115 key (18446744073709551606 128 14479360) itemoff 9510 itemsize 20 > > [17608.632681] item 116 key (18446744073709551606 128 14499840) itemoff 9498 itemsize 12 > > [17608.635448] item 117 key (18446744073709551606 128 14524416) itemoff 9394 itemsize 104 > > [17608.638122] item 118 key (18446744073709551606 128 14700544) itemoff 9382 itemsize 12 > > [17608.640633] item 119 key (18446744073709551606 128 14712832) itemoff 9370 itemsize 12 > > [17608.643227] item 120 key (18446744073709551606 128 14737408) itemoff 9350 itemsize 20 > > [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected > > [17608.648714] ------------[ cut here ]------------ > > [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] > > [17608.652972] Modules linked in: dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_writes dm_flakey dm_mod dax btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop [last unloaded: scsi_debug] > > [17608.660683] CPU: 2 PID: 27162 Comm: fsx Tainted: G L 5.7.0-rc3-default+ #1094 > > [17608.663520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 > > [17608.667102] RIP: 0010:btree_csum_one_bio+0x297/0x2a0 [btrfs] > > [17608.668919] Code: bc fd ff ff e8 aa e7 f0 c2 31 f6 48 89 3c 24 e8 ff 7b ff ff 48 8b 3c 24 48 c7 c6 78 75 25 c0 48 8b 17 4c 89 e7 e8 84 bc 0b 00 <0f> 0b e9 8f fd ff ff 66 90 0f 1f 44 00 00 48 89 f7 e9 53 fd ff ff > > [17608.674610] RSP: 0018:ffffafbb81e978b8 EFLAGS: 00010292 > > [17608.676299] RAX: 0000000000000000 RBX: ffff94287f810f00 RCX: 0000000000000001 > > [17608.678260] RDX: 0000000000000000 RSI: ffffffff83109fc2 RDI: ffffffff8310a096 > > [17608.680484] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 > > [17608.682645] R10: 0000000000000000 R11: 0000000000008ca0 R12: ffff94285f298000 > > [17608.684681] R13: ffff94274100bd30 R14: 0000000000000000 R15: 00000000ffffff8b > > [17608.686659] FS: 00007f3479617b80(0000) GS:ffff94287ba00000(0000) knlGS:0000000000000000 > > [17608.689240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [17608.691150] CR2: 0000000000466000 CR3: 000000010227f001 CR4: 0000000000160ee0 > > [17608.693354] Call Trace: > > [17608.694341] btree_submit_bio_hook+0x74/0xc0 [btrfs] > > [17608.696162] submit_one_bio+0x2b/0x40 [btrfs] > > [17608.698391] btree_write_cache_pages+0x373/0x440 [btrfs] > > [17608.700085] ? lock_acquire+0xa3/0x400 > > [17608.701257] ? __filemap_fdatawrite_range+0xad/0x110 > > [17608.702736] do_writepages+0x40/0xe0 > > [17608.703953] ? do_raw_spin_unlock+0x4b/0xc0 > > [17608.705290] ? _raw_spin_unlock+0x1f/0x30 > > [17608.706571] ? wbc_attach_and_unlock_inode+0x194/0x2a0 > > [17608.708167] __filemap_fdatawrite_range+0xce/0x110 > > [17608.709679] btrfs_write_marked_extents+0x68/0x160 [btrfs] > > [17608.711384] btrfs_sync_log+0x1a8/0xd40 [btrfs] > > [17608.712892] ? btrfs_log_inode+0x370/0xe20 [btrfs] > > [17608.714459] ? lock_acquire+0xa3/0x400 > > [17608.715803] ? lockref_put_or_lock+0x9/0x30 > > [17608.717115] ? dput+0x20/0x490 > > [17608.718172] ? dput+0x20/0x490 > > [17608.719271] ? do_raw_spin_unlock+0x4b/0xc0 > > [17608.720639] ? _raw_spin_unlock+0x1f/0x30 > > [17608.722002] btrfs_sync_file+0x335/0x490 [btrfs] > > [17608.723610] __x64_sys_msync+0x19e/0x200 > > [17608.724924] do_syscall_64+0x50/0x210 > > [17608.726143] entry_SYSCALL_64_after_hwframe+0x49/0xb3 > > [17608.727719] RIP: 0033:0x7f34797f2743 > > [17608.729030] Code: 64 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 1a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 48 83 ec 28 89 54 24 1c 48 > > [17608.734572] RSP: 002b:00007ffe84f11858 EFLAGS: 00000246 ORIG_RAX: 000000000000001a > > [17608.736913] RAX: ffffffffffffffda RBX: 00000000000008c8 RCX: 00007f34797f2743 > > [17608.738902] RDX: 0000000000000004 RSI: 000000000000293f RDI: 00007f3479814000 > > [17608.740949] RBP: 00000000000278c8 R08: 000000000000001f R09: 00007f3479814900 > > [17608.742896] R10: fffffffffffffd0c R11: 0000000000000246 R12: 0000000000002077 > > [17608.744810] R13: 000000000000293f R14: 00007f3479814000 R15: 0000000000000000 > > [17608.746764] irq event stamp: 0 > > [17608.747948] hardirqs last enabled at (0): [<0000000000000000>] 0x0 > > [17608.749754] hardirqs last disabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > > [17608.752428] softirqs last enabled at (0): [<ffffffff83081b0b>] copy_process+0x67b/0x1b00 > > [17608.754935] softirqs last disabled at (0): [<0000000000000000>] 0x0 > > [17608.756871] ---[ end trace ddba49bd36651725 ]--- > > [17608.758590] BTRFS: error (device dm-0) in btrfs_sync_log:3089: errno=-5 IO failure > > > -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: write-time leaf corruption detected (fixed) 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba 2020-05-12 14:26 ` Filipe Manana 2020-05-13 3:10 ` Qu Wenruo @ 2020-05-19 14:26 ` David Sterba 2 siblings, 0 replies; 24+ messages in thread From: David Sterba @ 2020-05-19 14:26 UTC (permalink / raw) To: David Sterba; +Cc: linux-btrfs On Tue, May 12, 2020 at 04:15:38PM +0200, David Sterba wrote: > generic/457 [05:40:52][17606.984360] run fstests generic/457 at 2020-05-04 05:40:52 > [17608.092098] BTRFS critical (device dm-0): corrupt leaf: root=18446744073709551610 block=30949376 slot=104, csum end range (13807616) goes beyond the start range (13717504) of the next csum item > [17608.098129] BTRFS info (device dm-0): leaf 30949376 gen 6 total ptrs 121 free space 6325 owner 18446744073709551610 > [17608.102727] BTRFS info (device dm-0): refs 2 lock (w:0 r:0 bw:0 br:0 sw:0 sr:0) lock_owner 0 current 27162 > [17608.645834] BTRFS error (device dm-0): block=30949376 write time tree block corruption detected > [17608.650272] WARNING: CPU: 2 PID: 27162 at fs/btrfs/disk-io.c:537 btree_csum_one_bio+0x297/0x2a0 [btrfs] Fixed in misc-next by Filipe's patch "btrfs: fix corrupt log due to concurrent fsync of inodes with shared extents". ^ permalink raw reply [flat|nested] 24+ messages in thread
* Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex 2020-05-12 14:12 Pending bugs for 5.7 David Sterba 2020-05-12 14:14 ` Bug 5.7-rc: root leak, eb leak David Sterba 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba @ 2020-05-12 14:15 ` David Sterba 2020-05-12 23:28 ` David Sterba 2020-05-13 19:46 ` [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex Anand Jain 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, fs_reclaim David Sterba 3 siblings, 2 replies; 24+ messages in thread From: David Sterba @ 2020-05-12 14:15 UTC (permalink / raw) To: linux-btrfs lockdep warning chunk_mutex vs device_list_mutex reversed during device initialization. Reported once. btrfs/161 [12:40:04][ 5172.154390] run fstests btrfs/161 at 2020-05-04 12:40:04 [ 5172.813371] BTRFS info (device vda): disk space caching is enabled [ 5172.816196] BTRFS info (device vda): has skinny extents [ 5173.611139] BTRFS: device fsid 812499c4-c1e2-4bb6-9234-efd07fac3aa1 devid 1 transid 5 /dev/vdb scanned by mkfs.btrfs (30700) [ 5173.662731] BTRFS info (device vdb): disk space caching is enabled [ 5173.664889] BTRFS info (device vdb): has skinny extents [ 5173.666818] BTRFS info (device vdb): flagging fs with big metadata feature [ 5173.682918] BTRFS info (device vdb): checking UUID tree [ 5173.944065] BTRFS info (device vdb): disk space caching is enabled [ 5173.946692] BTRFS info (device vdb): has skinny extents [ 5174.036719] BTRFS info (device vdb): relocating block group 567279616 flags system|dup [ 5174.075460] BTRFS info (device vdb): relocating block group 22020096 flags system|dup [ 5174.118369] BTRFS info (device vdb): disk added /dev/vdc [ 5174.250287] BTRFS info (device vdc): disk space caching is enabled [ 5174.252235] BTRFS info (device vdc): has skinny extents [ 5174.259812] [ 5174.260641] ====================================================== [ 5174.262652] WARNING: possible circular locking dependency detected [ 5174.264662] 5.7.0-rc3-default+ #1094 Not tainted [ 5174.266245] ------------------------------------------------------ [ 5174.268215] mount/30761 is trying to acquire lock: [ 5174.269838] ffff8d950e4164e8 (&fs_devs->device_list_mutex){+.+.}-{3:3}, at: clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.272880] [ 5174.272880] but task is already holding lock: [ 5174.275081] ffff8d952ae80980 (&fs_info->chunk_mutex){+.+.}-{3:3}, at: btrfs_read_chunk_tree+0x5a/0x2a0 [btrfs] [ 5174.278232] [ 5174.278232] which lock already depends on the new lock. [ 5174.278232] [ 5174.281372] [ 5174.281372] the existing dependency chain (in reverse order) is: [ 5174.283784] [ 5174.283784] -> #1 (&fs_info->chunk_mutex){+.+.}-{3:3}: [ 5174.286134] __lock_acquire+0x581/0xae0 [ 5174.287563] lock_acquire+0xa3/0x400 [ 5174.289033] __mutex_lock+0xa0/0xaf0 [ 5174.290488] btrfs_init_new_device+0x316/0x12f0 [btrfs] [ 5174.292209] btrfs_ioctl+0xc3c/0x2590 [btrfs] [ 5174.293673] ksys_ioctl+0x68/0xa0 [ 5174.294883] __x64_sys_ioctl+0x16/0x20 [ 5174.296231] do_syscall_64+0x50/0x210 [ 5174.297548] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 5174.299278] [ 5174.299278] -> #0 (&fs_devs->device_list_mutex){+.+.}-{3:3}: [ 5174.301760] check_prev_add+0x98/0xa20 [ 5174.303219] validate_chain+0xa6c/0x29e0 [ 5174.304770] __lock_acquire+0x581/0xae0 [ 5174.306274] lock_acquire+0xa3/0x400 [ 5174.307716] __mutex_lock+0xa0/0xaf0 [ 5174.309145] clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.310757] read_one_dev+0xc4/0x500 [btrfs] [ 5174.312293] btrfs_read_chunk_tree+0x202/0x2a0 [btrfs] [ 5174.313946] open_ctree+0x7a3/0x10db [btrfs] [ 5174.315411] btrfs_mount_root.cold+0xe/0xcc [btrfs] [ 5174.317122] legacy_get_tree+0x2d/0x60 [ 5174.318543] vfs_get_tree+0x1d/0xb0 [ 5174.319844] fc_mount+0xe/0x40 [ 5174.321122] vfs_kern_mount.part.0+0x71/0x90 [ 5174.322688] btrfs_mount+0x147/0x3e0 [btrfs] [ 5174.324250] legacy_get_tree+0x2d/0x60 [ 5174.325644] vfs_get_tree+0x1d/0xb0 [ 5174.326978] do_mount+0x7d5/0xa40 [ 5174.328294] __x64_sys_mount+0x8e/0xd0 [ 5174.329829] do_syscall_64+0x50/0x210 [ 5174.331260] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 5174.333102] [ 5174.333102] other info that might help us debug this: [ 5174.333102] [ 5174.335988] Possible unsafe locking scenario: [ 5174.335988] [ 5174.338051] CPU0 CPU1 [ 5174.339490] ---- ---- [ 5174.340810] lock(&fs_info->chunk_mutex); [ 5174.342203] lock(&fs_devs->device_list_mutex); [ 5174.344228] lock(&fs_info->chunk_mutex); [ 5174.346161] lock(&fs_devs->device_list_mutex); [ 5174.347666] [ 5174.347666] *** DEADLOCK *** [ 5174.347666] [ 5174.350257] 3 locks held by mount/30761: [ 5174.351558] #0: ffff8d95358310e8 (&type->s_umount_key#24/1){+.+.}-{3:3}, at: alloc_super.isra.0+0xb3/0x3f0 [ 5174.354620] #1: ffffffffc0451c30 (uuid_mutex){+.+.}-{3:3}, at: btrfs_read_chunk_tree+0x50/0x2a0 [btrfs] [ 5174.357770] #2: ffff8d952ae80980 (&fs_info->chunk_mutex){+.+.}-{3:3}, at: btrfs_read_chunk_tree+0x5a/0x2a0 [btrfs] [ 5174.360835] stack backtrace: [ 5174.371458] CPU: 0 PID: 30761 Comm: mount Not tainted 5.7.0-rc3-default+ #1094 [ 5174.373918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 5174.377133] Call Trace: [ 5174.378125] dump_stack+0x71/0xa0 [ 5174.379316] check_noncircular+0x16f/0x190 [ 5174.380673] ? lock_acquire+0xa3/0x400 [ 5174.381938] check_prev_add+0x98/0xa20 [ 5174.383161] validate_chain+0xa6c/0x29e0 [ 5174.384481] ? btrfs_mount+0x146/0x3e0 [btrfs] [ 5174.385999] ? ftrace_caller_op_ptr+0xe/0xe [ 5174.387457] ? btrfs_mount+0x146/0x3e0 [btrfs] [ 5174.388965] __lock_acquire+0x581/0xae0 [ 5174.390325] lock_acquire+0xa3/0x400 [ 5174.391606] ? clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.393146] __mutex_lock+0xa0/0xaf0 [ 5174.394342] ? clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.395906] ? rcu_read_lock_sched_held+0x5d/0x90 [ 5174.397424] ? module_assert_mutex_or_preempt+0x14/0x40 [ 5174.399028] ? __module_address+0x28/0xf0 [ 5174.400422] ? clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.401999] ? static_obj+0x4f/0x60 [ 5174.403208] ? lockdep_init_map_waits+0x4d/0x200 [ 5174.404749] ? clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.406356] clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.407863] read_one_dev+0xc4/0x500 [btrfs] [ 5174.409270] btrfs_read_chunk_tree+0x202/0x2a0 [btrfs] [ 5174.410846] open_ctree+0x7a3/0x10db [btrfs] [ 5174.412333] btrfs_mount_root.cold+0xe/0xcc [btrfs] [ 5174.413945] ? vfs_parse_fs_string+0x84/0xb0 [ 5174.415263] ? rcu_read_lock_sched_held+0x5d/0x90 [ 5174.416728] ? kfree+0x211/0x3a0 [ 5174.417963] legacy_get_tree+0x2d/0x60 [ 5174.419382] vfs_get_tree+0x1d/0xb0 [ 5174.420621] fc_mount+0xe/0x40 [ 5174.421747] vfs_kern_mount.part.0+0x71/0x90 [ 5174.423156] btrfs_mount+0x147/0x3e0 [btrfs] [ 5174.424552] ? vfs_parse_fs_string+0x84/0xb0 [ 5174.425943] ? rcu_read_lock_sched_held+0x5d/0x90 [ 5174.427416] ? kfree+0x211/0x3a0 [ 5174.428567] ? legacy_get_tree+0x2d/0x60 [ 5174.429828] legacy_get_tree+0x2d/0x60 [ 5174.431071] vfs_get_tree+0x1d/0xb0 [ 5174.432391] do_mount+0x7d5/0xa40 [ 5174.433673] ? _copy_from_user+0x6a/0xa0 [ 5174.435045] ? memdup_user+0x4e/0x90 [ 5174.436279] __x64_sys_mount+0x8e/0xd0 [ 5174.437581] do_syscall_64+0x50/0x210 [ 5174.438815] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 5174.440371] RIP: 0033:0x7f9084b35eba [ 5174.446779] RSP: 002b:00007ffe01ce4178 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [ 5174.449335] RAX: ffffffffffffffda RBX: 00007ffe01ce42f0 RCX: 00007f9084b35eba [ 5174.451352] RDX: 00005559ef16dd70 RSI: 00005559ef166bb0 RDI: 00005559ef166b90 [ 5174.453507] RBP: 00005559ef166960 R08: 0000000000000000 R09: 00005559ef166010 [ 5174.455623] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 5174.457621] R13: 00005559ef166b90 R14: 00005559ef16dd70 R15: 00005559ef166960 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex David Sterba @ 2020-05-12 23:28 ` David Sterba 2020-05-12 19:25 ` Anand Jain 2020-05-13 19:46 ` [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex Anand Jain 1 sibling, 1 reply; 24+ messages in thread From: David Sterba @ 2020-05-12 23:28 UTC (permalink / raw) To: David Sterba; +Cc: linux-btrfs On Tue, May 12, 2020 at 04:15:46PM +0200, David Sterba wrote: > [ 5174.283784] -> #1 (&fs_info->chunk_mutex){+.+.}-{3:3}: > [ 5174.286134] __lock_acquire+0x581/0xae0 > [ 5174.287563] lock_acquire+0xa3/0x400 > [ 5174.289033] __mutex_lock+0xa0/0xaf0 > [ 5174.290488] btrfs_init_new_device+0x316/0x12f0 [btrfs] > [ 5174.292209] btrfs_ioctl+0xc3c/0x2590 [btrfs] ioctl called > [ 5174.293673] ksys_ioctl+0x68/0xa0 > [ 5174.294883] __x64_sys_ioctl+0x16/0x20 > [ 5174.296231] do_syscall_64+0x50/0x210 > [ 5174.297548] entry_SYSCALL_64_after_hwframe+0x49/0xb3 > [ 5174.299278] > [ 5174.299278] -> #0 (&fs_devs->device_list_mutex){+.+.}-{3:3}: > [ 5174.301760] check_prev_add+0x98/0xa20 > [ 5174.303219] validate_chain+0xa6c/0x29e0 > [ 5174.304770] __lock_acquire+0x581/0xae0 > [ 5174.306274] lock_acquire+0xa3/0x400 > [ 5174.307716] __mutex_lock+0xa0/0xaf0 > [ 5174.309145] clone_fs_devices+0x3f/0x170 [btrfs] > [ 5174.310757] read_one_dev+0xc4/0x500 [btrfs] > [ 5174.312293] btrfs_read_chunk_tree+0x202/0x2a0 [btrfs] > [ 5174.313946] open_ctree+0x7a3/0x10db [btrfs] ... while the filesystem is being set up. This is actually possible because this is with enabled seeding, so the mounted filesystem accesses the seeding filesystem's structures when cloning the devices. Should be fixed by lifting the device_list_mutex from clone_fs_devices to some of it's callers. In btrfs_read_chunk_tree it's between the uuid mutex and chunk mutex, in btrfs_init_new_device lock device_list_mutex before "if (seeding_dev)". > [ 5174.315411] btrfs_mount_root.cold+0xe/0xcc [btrfs] > [ 5174.317122] legacy_get_tree+0x2d/0x60 > [ 5174.318543] vfs_get_tree+0x1d/0xb0 > [ 5174.319844] fc_mount+0xe/0x40 > [ 5174.321122] vfs_kern_mount.part.0+0x71/0x90 > [ 5174.322688] btrfs_mount+0x147/0x3e0 [btrfs] > [ 5174.324250] legacy_get_tree+0x2d/0x60 > [ 5174.325644] vfs_get_tree+0x1d/0xb0 > [ 5174.326978] do_mount+0x7d5/0xa40 > [ 5174.328294] __x64_sys_mount+0x8e/0xd0 > [ 5174.329829] do_syscall_64+0x50/0x210 > [ 5174.331260] entry_SYSCALL_64_after_hwframe+0x49/0xb3 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex 2020-05-12 23:28 ` David Sterba @ 2020-05-12 19:25 ` Anand Jain 0 siblings, 0 replies; 24+ messages in thread From: Anand Jain @ 2020-05-12 19:25 UTC (permalink / raw) To: dsterba, David Sterba, linux-btrfs On 13/5/2020 7:28 am, David Sterba wrote: > On Tue, May 12, 2020 at 04:15:46PM +0200, David Sterba wrote: >> [ 5174.283784] -> #1 (&fs_info->chunk_mutex){+.+.}-{3:3}: >> [ 5174.286134] __lock_acquire+0x581/0xae0 >> [ 5174.287563] lock_acquire+0xa3/0x400 >> [ 5174.289033] __mutex_lock+0xa0/0xaf0 >> [ 5174.290488] btrfs_init_new_device+0x316/0x12f0 [btrfs] >> [ 5174.292209] btrfs_ioctl+0xc3c/0x2590 [btrfs] > > ioctl called > >> [ 5174.293673] ksys_ioctl+0x68/0xa0 >> [ 5174.294883] __x64_sys_ioctl+0x16/0x20 >> [ 5174.296231] do_syscall_64+0x50/0x210 >> [ 5174.297548] entry_SYSCALL_64_after_hwframe+0x49/0xb3 >> [ 5174.299278] >> [ 5174.299278] -> #0 (&fs_devs->device_list_mutex){+.+.}-{3:3}: >> [ 5174.301760] check_prev_add+0x98/0xa20 >> [ 5174.303219] validate_chain+0xa6c/0x29e0 >> [ 5174.304770] __lock_acquire+0x581/0xae0 >> [ 5174.306274] lock_acquire+0xa3/0x400 >> [ 5174.307716] __mutex_lock+0xa0/0xaf0 >> [ 5174.309145] clone_fs_devices+0x3f/0x170 [btrfs] >> [ 5174.310757] read_one_dev+0xc4/0x500 [btrfs] >> [ 5174.312293] btrfs_read_chunk_tree+0x202/0x2a0 [btrfs] >> [ 5174.313946] open_ctree+0x7a3/0x10db [btrfs] > > ... while the filesystem is being set up. This is actually possible > because this is with enabled seeding, so the mounted filesystem accesses > the seeding filesystem's structures when cloning the devices. > > Should be fixed by lifting the device_list_mutex from clone_fs_devices > to some of it's callers. In btrfs_read_chunk_tree it's between the uuid > mutex and chunk mutex, in btrfs_init_new_device lock device_list_mutex > before "if (seeding_dev)". > Two strange things as of now, why we see this only now and mount thread is still running but we have the device add ioctl thread. >> [ 5174.315411] btrfs_mount_root.cold+0xe/0xcc [btrfs] >> [ 5174.317122] legacy_get_tree+0x2d/0x60 >> [ 5174.318543] vfs_get_tree+0x1d/0xb0 >> [ 5174.319844] fc_mount+0xe/0x40 >> [ 5174.321122] vfs_kern_mount.part.0+0x71/0x90 >> [ 5174.322688] btrfs_mount+0x147/0x3e0 [btrfs] >> [ 5174.324250] legacy_get_tree+0x2d/0x60 >> [ 5174.325644] vfs_get_tree+0x1d/0xb0 >> [ 5174.326978] do_mount+0x7d5/0xa40 >> [ 5174.328294] __x64_sys_mount+0x8e/0xd0 >> [ 5174.329829] do_syscall_64+0x50/0x210 >> [ 5174.331260] entry_SYSCALL_64_after_hwframe+0x49/0xb3 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex David Sterba 2020-05-12 23:28 ` David Sterba @ 2020-05-13 19:46 ` Anand Jain 2020-05-15 17:40 ` David Sterba 1 sibling, 1 reply; 24+ messages in thread From: Anand Jain @ 2020-05-13 19:46 UTC (permalink / raw) To: linux-btrfs; +Cc: dsterba ABBA locking order lockdep warning reported during fstests btrfs/161 as below. [ 5174.262652] WARNING: possible circular locking dependency detected [ 5174.264662] 5.7.0-rc3-default+ #1094 Not tainted [ 5174.266245] ------------------------------------------------------ [ 5174.268215] mount/30761 is trying to acquire lock: [ 5174.269838] ffff8d950e4164e8 (&fs_devs->device_list_mutex){+.+.}-{3:3}, at: clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.272880] [ 5174.272880] but task is already holding lock: [ 5174.275081] ffff8d952ae80980 (&fs_info->chunk_mutex){+.+.}-{3:3}, at: btrfs_read_chunk_tree+0x5a/0x2a0 [btrfs] [ 5174.278232] [ 5174.278232] which lock already depends on the new lock. [ 5174.278232] [ 5174.281372] [ 5174.281372] the existing dependency chain (in reverse order) is: [ 5174.283784] [ 5174.283784] -> #1 (&fs_info->chunk_mutex){+.+.}-{3:3}: [ 5174.286134] __lock_acquire+0x581/0xae0 [ 5174.287563] lock_acquire+0xa3/0x400 [ 5174.289033] __mutex_lock+0xa0/0xaf0 [ 5174.290488] btrfs_init_new_device+0x316/0x12f0 [btrfs] [ 5174.292209] btrfs_ioctl+0xc3c/0x2590 [btrfs] [ 5174.293673] ksys_ioctl+0x68/0xa0 [ 5174.294883] __x64_sys_ioctl+0x16/0x20 [ 5174.296231] do_syscall_64+0x50/0x210 [ 5174.297548] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 5174.299278] [ 5174.299278] -> #0 (&fs_devs->device_list_mutex){+.+.}-{3:3}: [ 5174.301760] check_prev_add+0x98/0xa20 [ 5174.303219] validate_chain+0xa6c/0x29e0 [ 5174.304770] __lock_acquire+0x581/0xae0 [ 5174.306274] lock_acquire+0xa3/0x400 [ 5174.307716] __mutex_lock+0xa0/0xaf0 [ 5174.309145] clone_fs_devices+0x3f/0x170 [btrfs] [ 5174.310757] read_one_dev+0xc4/0x500 [btrfs] [ 5174.312293] btrfs_read_chunk_tree+0x202/0x2a0 [btrfs] [ 5174.313946] open_ctree+0x7a3/0x10db [btrfs] [ 5174.315411] btrfs_mount_root.cold+0xe/0xcc [btrfs] [ 5174.317122] legacy_get_tree+0x2d/0x60 [ 5174.318543] vfs_get_tree+0x1d/0xb0 [ 5174.319844] fc_mount+0xe/0x40 [ 5174.321122] vfs_kern_mount.part.0+0x71/0x90 [ 5174.322688] btrfs_mount+0x147/0x3e0 [btrfs] [ 5174.324250] legacy_get_tree+0x2d/0x60 [ 5174.325644] vfs_get_tree+0x1d/0xb0 [ 5174.326978] do_mount+0x7d5/0xa40 [ 5174.328294] __x64_sys_mount+0x8e/0xd0 [ 5174.329829] do_syscall_64+0x50/0x210 [ 5174.331260] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 5174.333102] [ 5174.333102] other info that might help us debug this: [ 5174.333102] [ 5174.335988] Possible unsafe locking scenario: [ 5174.335988] [ 5174.338051] CPU0 CPU1 [ 5174.339490] ---- ---- [ 5174.340810] lock(&fs_info->chunk_mutex); [ 5174.342203] lock(&fs_devs->device_list_mutex); [ 5174.344228] lock(&fs_info->chunk_mutex); [ 5174.346161] lock(&fs_devs->device_list_mutex); [ 5174.347666] [ 5174.347666] *** DEADLOCK *** The test case btrfs/161 creates seed device and adds sprout device to it using ioctl. The ioctl thread which adds the sprout device to the mounted seed device calls btrfs_prepare_sprout() and holds the lock in the following order. mutex_lock(&fs_devices->device_list_mutex); mutex_lock(&fs_info->chunk_mutex); The add thread still in the CPU#1 however as the test case would the mount the sprout device this time is on CPU#0. As it is mounting the sprout device btrfs_read_chunk_tree() it calls clone_fs_devices() establishing the lock order. mutex_lock(&fs_info->chunk_mutex); mutex_lock(&fs_devices->device_list_mutex); But the address of these two fs_devices and fs_info are different so this ABBA warning won't be true in real. However as per our design the chunk_mutex and device_list_mutex must follow the locking order of the former thread so there is something to fix. volume.c: 279 * Lock nesting 280 * ============ 281 * 282 * uuid_mutex 283 * device_list_mutex 284 * chunk_mutex To fix the idea is to move out the lock from clone_fs_devices() to its call chain up at btrfs_read_chunk_tree(). btrfs_read_chunk_tree() + mutex_lock(&orig->device_list_mutex); mutex_lock(&fs_info->chunk_mutex); read_one_dev(leaf, dev_item); (single parent fn) open_seed_devices(fs_info, fs_uuid); (single parent fn) clone_fs_devices(fs_devices); (also called by btrfs_prepare_sprout only) - mutex_lock(&orig->device_list_mutex); As clone_fs_devices() is also called by btrfs_prepare_sprout() add the required device_list_mutex there. btrfs_prepare_sprout() + mutex_lock(&fs_devices->device_list_mutex); clone_fs_devices() + mutex_unlock(&fs_devices->device_list_mutex); Next, in btrfs_read_chunk_tree() there are several helper functions which are now under device_list_mutex which are fine to be unless there is already device_list_mutex lock some other incompatible lock. So each of those helper functions were checked as below, and they don't hold any locks. open_seed_devices() ok find_fsid() ok clone_fs_devices() ok with fix open_fs_devices() ok free_fs_devices() ok btrfs_find_device() ok add_missing_dev() ok Signed-off-by: Anand Jain <anand.jain@oracle.com> --- Testing: fstests groups volume and seed ran fine. A full list of tests just started. fs/btrfs/volumes.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 60ab41c12e50..ebc8565d0f73 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -984,7 +984,6 @@ static struct btrfs_fs_devices *clone_fs_devices(struct btrfs_fs_devices *orig) if (IS_ERR(fs_devices)) return fs_devices; - mutex_lock(&orig->device_list_mutex); fs_devices->total_devices = orig->total_devices; list_for_each_entry(orig_dev, &orig->devices, dev_list) { @@ -1016,10 +1015,8 @@ static struct btrfs_fs_devices *clone_fs_devices(struct btrfs_fs_devices *orig) device->fs_devices = fs_devices; fs_devices->num_devices++; } - mutex_unlock(&orig->device_list_mutex); return fs_devices; error: - mutex_unlock(&orig->device_list_mutex); free_fs_devices(fs_devices); return ERR_PTR(ret); } @@ -2363,11 +2360,14 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info) if (IS_ERR(seed_devices)) return PTR_ERR(seed_devices); + mutex_lock(&fs_devices->device_list_mutex); old_devices = clone_fs_devices(fs_devices); if (IS_ERR(old_devices)) { + mutex_unlock(&fs_devices->device_list_mutex); kfree(seed_devices); return PTR_ERR(old_devices); } + mutex_unlock(&fs_devices->device_list_mutex); list_add(&old_devices->fs_list, &fs_uuids); @@ -7049,6 +7049,7 @@ int btrfs_read_chunk_tree(struct btrfs_fs_info *fs_info) * otherwise we don't need it. */ mutex_lock(&uuid_mutex); + mutex_lock(&fs_info->fs_devices->device_list_mutex); mutex_lock(&fs_info->chunk_mutex); /* @@ -7117,6 +7118,7 @@ int btrfs_read_chunk_tree(struct btrfs_fs_info *fs_info) ret = 0; error: mutex_unlock(&fs_info->chunk_mutex); + mutex_unlock(&fs_info->fs_devices->device_list_mutex); mutex_unlock(&uuid_mutex); btrfs_free_path(path); -- 2.25.1 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex 2020-05-13 19:46 ` [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex Anand Jain @ 2020-05-15 17:40 ` David Sterba 2020-05-16 3:43 ` Anand Jain 0 siblings, 1 reply; 24+ messages in thread From: David Sterba @ 2020-05-15 17:40 UTC (permalink / raw) To: Anand Jain; +Cc: linux-btrfs, dsterba On Thu, May 14, 2020 at 03:46:59AM +0800, Anand Jain wrote: > A full list of tests just started. > > fs/btrfs/volumes.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) > > diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c > index 60ab41c12e50..ebc8565d0f73 100644 > --- a/fs/btrfs/volumes.c > +++ b/fs/btrfs/volumes.c > @@ -984,7 +984,6 @@ static struct btrfs_fs_devices *clone_fs_devices(struct btrfs_fs_devices *orig) > if (IS_ERR(fs_devices)) > return fs_devices; > So now here's the device_list_mutex taken by a caller but inside clone_fs_devices there's fs_devices = alloc_fs_devices(orig->fsid, NULL); just before this line and it does a GFP_KERNEL allocation. This could deadlock through the allocator trying to flush data and then superblock write locking the device_list_mutex again. > - mutex_lock(&orig->device_list_mutex); > fs_devices->total_devices = orig->total_devices; > > list_for_each_entry(orig_dev, &orig->devices, dev_list) { ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex 2020-05-15 17:40 ` David Sterba @ 2020-05-16 3:43 ` Anand Jain 2020-05-18 11:07 ` Anand Jain 0 siblings, 1 reply; 24+ messages in thread From: Anand Jain @ 2020-05-16 3:43 UTC (permalink / raw) To: dsterba, linux-btrfs, dsterba On 16/5/20 1:40 am, David Sterba wrote: > On Thu, May 14, 2020 at 03:46:59AM +0800, Anand Jain wrote: >> A full list of tests just started. >> >> fs/btrfs/volumes.c | 8 +++++--- >> 1 file changed, 5 insertions(+), 3 deletions(-) >> >> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c >> index 60ab41c12e50..ebc8565d0f73 100644 >> --- a/fs/btrfs/volumes.c >> +++ b/fs/btrfs/volumes.c >> @@ -984,7 +984,6 @@ static struct btrfs_fs_devices *clone_fs_devices(struct btrfs_fs_devices *orig) >> if (IS_ERR(fs_devices)) >> return fs_devices; >> > > So now here's the device_list_mutex taken by a caller but inside > clone_fs_devices there's > > fs_devices = alloc_fs_devices(orig->fsid, NULL); > > just before this line and it does a GFP_KERNEL allocation. Oh right the allocations. Its not just about the other locks as I thought before. There are two ways to fix. Use GFP_NOFS I am not yet sure if it not possible. There were some previous work on the GFP flags. I need to review them. or, Move the allocation outside the locks. Looking into both of these choices. Thanks, Anand > This could > deadlock through the allocator trying to flush data and then superblock > write locking the device_list_mutex again. > >> - mutex_lock(&orig->device_list_mutex); >> fs_devices->total_devices = orig->total_devices; >> >> list_for_each_entry(orig_dev, &orig->devices, dev_list) { ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex 2020-05-16 3:43 ` Anand Jain @ 2020-05-18 11:07 ` Anand Jain 2020-05-18 15:28 ` David Sterba 0 siblings, 1 reply; 24+ messages in thread From: Anand Jain @ 2020-05-18 11:07 UTC (permalink / raw) To: dsterba, linux-btrfs, dsterba On 16/5/20 11:43 am, Anand Jain wrote: > On 16/5/20 1:40 am, David Sterba wrote: >> On Thu, May 14, 2020 at 03:46:59AM +0800, Anand Jain wrote: >>> A full list of tests just started. >>> >>> fs/btrfs/volumes.c | 8 +++++--- >>> 1 file changed, 5 insertions(+), 3 deletions(-) >>> >>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c >>> index 60ab41c12e50..ebc8565d0f73 100644 >>> --- a/fs/btrfs/volumes.c >>> +++ b/fs/btrfs/volumes.c >>> @@ -984,7 +984,6 @@ static struct btrfs_fs_devices >>> *clone_fs_devices(struct btrfs_fs_devices *orig) >>> if (IS_ERR(fs_devices)) >>> return fs_devices; >> >> So now here's the device_list_mutex taken by a caller but inside >> clone_fs_devices there's >> >> fs_devices = alloc_fs_devices(orig->fsid, NULL); >> >> just before this line and it does a GFP_KERNEL allocation. > > Oh right the allocations. Its not just about the other locks > as I thought before. > > There are two ways to fix. > Use GFP_NOFS > I am not yet sure if it not possible. There were some previous > work on the GFP flags. I need to review them. or, > Move the allocation outside the locks. > > Looking into both of these choices. > Nack. On this patch. In general GFP_KERNEL is preferred over GFP_NOFS. For example. -------- 6165572c btrfs: use GFP_KERNEL in btrfs_init_dev_replace_tgtdev cc8385b5 btrfs: preallocate radix tree node for readahead 78f2c9e6 btrfs: device add and remove: use GFP_KERNEL -------- And there are quite a lot of GFP_KERNEL allocation along the path leading to clone_fs_devices(). Thanks, Anand > Thanks, Anand > >> This could >> deadlock through the allocator trying to flush data and then superblock >> write locking the device_list_mutex again. >> >>> - mutex_lock(&orig->device_list_mutex); >>> fs_devices->total_devices = orig->total_devices; >>> list_for_each_entry(orig_dev, &orig->devices, dev_list) { > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex 2020-05-18 11:07 ` Anand Jain @ 2020-05-18 15:28 ` David Sterba 0 siblings, 0 replies; 24+ messages in thread From: David Sterba @ 2020-05-18 15:28 UTC (permalink / raw) To: Anand Jain; +Cc: dsterba, linux-btrfs, dsterba On Mon, May 18, 2020 at 07:07:21PM +0800, Anand Jain wrote: > > > On 16/5/20 11:43 am, Anand Jain wrote: > > On 16/5/20 1:40 am, David Sterba wrote: > >> On Thu, May 14, 2020 at 03:46:59AM +0800, Anand Jain wrote: > >>> A full list of tests just started. > >>> > >>> fs/btrfs/volumes.c | 8 +++++--- > >>> 1 file changed, 5 insertions(+), 3 deletions(-) > >>> > >>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c > >>> index 60ab41c12e50..ebc8565d0f73 100644 > >>> --- a/fs/btrfs/volumes.c > >>> +++ b/fs/btrfs/volumes.c > >>> @@ -984,7 +984,6 @@ static struct btrfs_fs_devices > >>> *clone_fs_devices(struct btrfs_fs_devices *orig) > >>> if (IS_ERR(fs_devices)) > >>> return fs_devices; > >> > >> So now here's the device_list_mutex taken by a caller but inside > >> clone_fs_devices there's > >> > >> fs_devices = alloc_fs_devices(orig->fsid, NULL); > >> > >> just before this line and it does a GFP_KERNEL allocation. > > > > Oh right the allocations. Its not just about the other locks > > as I thought before. > > > > There are two ways to fix. > > Use GFP_NOFS > > I am not yet sure if it not possible. There were some previous > > work on the GFP flags. I need to review them. or, > > Move the allocation outside the locks. > > > > Looking into both of these choices. > > > > > Nack. On this patch. > > In general GFP_KERNEL is preferred over GFP_NOFS. For example. > > -------- > 6165572c btrfs: use GFP_KERNEL in btrfs_init_dev_replace_tgtdev > cc8385b5 btrfs: preallocate radix tree node for readahead > 78f2c9e6 btrfs: device add and remove: use GFP_KERNEL > -------- > > And there are quite a lot of GFP_KERNEL allocation along the > path leading to clone_fs_devices(). For that we have the scoped NOFS, using the memalloc_nofs_save/memalloc_nofs_restore. So if this is set before/after the device_list_mutex is taken when calling the cloning then it's safe with any potential GFP_KERNEL allocations on the way. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Bug 5.7-rc: lockdep warning, fs_reclaim 2020-05-12 14:12 Pending bugs for 5.7 David Sterba ` (2 preceding siblings ...) 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex David Sterba @ 2020-05-12 14:15 ` David Sterba 3 siblings, 0 replies; 24+ messages in thread From: David Sterba @ 2020-05-12 14:15 UTC (permalink / raw) To: linux-btrfs lockdep warning on chunk_mutex/fs_reclaim/delayed_node->mutex btrfs/073 [16:46:49][ 4389.528850] run fstests btrfs/073 at 2020-05-11 16:46:49 [ 4394.208553] BTRFS info (device vdb): disk space caching is enabled [ 4394.208698] BTRFS info (device vdb): scrub: finished on devid 4 with status: 0 [ 4394.212495] BTRFS info (device vdb): scrub: finished on devid 5 with status: 0 [ 4394.276890] BTRFS info (device vdb): scrub: finished on devid 6 with status: 0 [ 4394.408713] BTRFS info (device vdb): scrub: finished on devid 1 with status: 0 [ 4394.408814] BTRFS info (device vdb): use zlib compression, level 3 [ 4394.413630] BTRFS info (device vdb): disk space caching is enabled [ 4394.452197] BTRFS info (device vdb): scrub: started on devid 5 [ 4394.452587] BTRFS info (device vdb): scrub: started on devid 4 [ 4394.462946] BTRFS info (device vdb): scrub: finished on devid 5 with status: 0 [ 4394.473072] BTRFS info (device vdb): scrub: started on devid 3 [ 4394.483089] BTRFS info (device vdb): scrub: started on devid 1 [ 4394.490128] BTRFS info (device vdb): scrub: started on devid 6 [ 4394.494793] BTRFS info (device vdb): scrub: started on devid 2 [ 4394.497311] BTRFS info (device vdb): scrub: finished on devid 4 with status: 0 [ 4394.530402] [ 4394.532133] ====================================================== [ 4394.534771] WARNING: possible circular locking dependency detected [ 4394.536949] 5.7.0-rc5-default+ #1100 Not tainted [ 4394.538809] ------------------------------------------------------ [ 4394.541508] fsstress/15173 is trying to acquire lock: [ 4394.542204] BTRFS info (device vdb): scrub: finished on devid 2 with status: 0 [ 4394.544236] ffff9a5c7153c980 (&fs_info->chunk_mutex){+.+.}-{3:3}, at: btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.544274] [ 4394.544274] but task is already holding lock: [ 4394.553183] ffff9a5c74857458 (&delayed_node->mutex){+.+.}-{3:3}, at: __btrfs_commit_inode_delayed_items+0x58/0x510 [btrfs] [ 4394.557542] [ 4394.557542] which lock already depends on the new lock. [ 4394.557542] [ 4394.561420] [ 4394.561420] the existing dependency chain (in reverse order) is: [ 4394.565023] [ 4394.565023] -> #2 (&delayed_node->mutex){+.+.}-{3:3}: [ 4394.567738] __lock_acquire+0x581/0xae0 [ 4394.568965] lock_acquire+0xa3/0x400 [ 4394.570307] __mutex_lock+0xa0/0xaf0 [ 4394.571983] __btrfs_release_delayed_node.part.0+0x3f/0x310 [btrfs] [ 4394.574411] btrfs_evict_inode+0x3b7/0x550 [btrfs] [ 4394.576443] evict+0xd6/0x1c0 [ 4394.578185] dispose_list+0x48/0x70 [ 4394.579583] prune_icache_sb+0x54/0x80 [ 4394.581163] super_cache_scan+0x121/0x1a0 [ 4394.583106] do_shrink_slab+0x175/0x420 [ 4394.585205] shrink_slab+0xb1/0x2e0 [ 4394.587248] shrink_node+0x162/0x5c0 [ 4394.588947] balance_pgdat+0x324/0x750 [ 4394.590764] kswapd+0x236/0x530 [ 4394.592227] kthread+0x130/0x150 [ 4394.593454] ret_from_fork+0x24/0x30 [ 4394.594737] [ 4394.594737] -> #1 (fs_reclaim){+.+.}-{0:0}: [ 4394.597285] __lock_acquire+0x581/0xae0 [ 4394.598822] lock_acquire+0xa3/0x400 [ 4394.600814] fs_reclaim_acquire.part.0+0x25/0x30 [ 4394.602349] __kmalloc_track_caller+0x4a/0x370 [ 4394.603671] kstrdup+0x2e/0x60 [ 4394.604706] __kernfs_new_node.constprop.0+0x41/0x1c0 [ 4394.605874] kernfs_new_node+0x25/0x50 [ 4394.606856] kernfs_create_link+0x34/0xa0 [ 4394.607871] sysfs_do_create_link_sd+0x5e/0xd0 [ 4394.609063] btrfs_sysfs_add_devices_dir+0x65/0x100 [btrfs] [ 4394.610879] btrfs_init_new_device+0x44b/0x1300 [btrfs] [ 4394.612519] btrfs_ioctl+0xc3c/0x2590 [btrfs] [ 4394.614139] ksys_ioctl+0x68/0xa0 [ 4394.615490] __x64_sys_ioctl+0x16/0x20 [ 4394.616988] do_syscall_64+0x50/0x210 [ 4394.618235] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 4394.619541] [ 4394.619541] -> #0 (&fs_info->chunk_mutex){+.+.}-{3:3}: [ 4394.621414] check_prev_add+0x98/0xa20 [ 4394.622859] validate_chain+0xa6c/0x29e0 [ 4394.623992] __lock_acquire+0x581/0xae0 [ 4394.625165] lock_acquire+0xa3/0x400 [ 4394.626628] __mutex_lock+0xa0/0xaf0 [ 4394.627990] btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.629365] find_free_extent+0xb47/0xfb0 [btrfs] [ 4394.630642] btrfs_reserve_extent+0x9b/0x180 [btrfs] [ 4394.631977] btrfs_alloc_tree_block+0xc1/0x350 [btrfs] [ 4394.633273] alloc_tree_block_no_bg_flush+0x4a/0x60 [btrfs] [ 4394.635117] __btrfs_cow_block+0x143/0x7a0 [btrfs] [ 4394.636408] btrfs_cow_block+0x15f/0x310 [btrfs] [ 4394.638120] btrfs_search_slot+0x937/0xf70 [btrfs] [ 4394.639746] btrfs_insert_empty_items+0x64/0xc0 [btrfs] [ 4394.641636] __btrfs_commit_inode_delayed_items+0xaf/0x510 [btrfs] [ 4394.643733] __btrfs_run_delayed_items+0x8e/0x140 [btrfs] [ 4394.645344] btrfs_commit_transaction+0x312/0xae0 [btrfs] [ 4394.646804] btrfs_sync_file+0x345/0x490 [btrfs] [ 4394.648369] do_fsync+0x38/0x70 [ 4394.649732] __x64_sys_fsync+0x10/0x20 [ 4394.651214] do_syscall_64+0x50/0x210 [ 4394.652691] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 4394.654307] [ 4394.654307] other info that might help us debug this: [ 4394.654307] [ 4394.666340] Chain exists of: [ 4394.666340] &fs_info->chunk_mutex --> fs_reclaim --> &delayed_node->mutex [ 4394.666340] [ 4394.669883] Possible unsafe locking scenario: [ 4394.669883] [ 4394.671702] CPU0 CPU1 [ 4394.672939] ---- ---- [ 4394.674445] lock(&delayed_node->mutex); [ 4394.675745] lock(fs_reclaim); [ 4394.677590] lock(&delayed_node->mutex); [ 4394.679685] lock(&fs_info->chunk_mutex); [ 4394.681219] [ 4394.681219] *** DEADLOCK *** [ 4394.681219] [ 4394.683928] 2 locks held by fsstress/15173: [ 4394.685460] #0: ffff9a5c72851698 (sb_internal#2){.+.+}-{0:0}, at: start_transaction+0x409/0x5e0 [btrfs] [ 4394.688402] #1: ffff9a5c74857458 (&delayed_node->mutex){+.+.}-{3:3}, at: __btrfs_commit_inode_delayed_items+0x58/0x510 [btrfs] [ 4394.691958] [ 4394.691958] stack backtrace: [ 4394.693535] CPU: 3 PID: 15173 Comm: fsstress Not tainted 5.7.0-rc5-default+ #1100 [ 4394.695708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014 [ 4394.698798] Call Trace: [ 4394.699882] dump_stack+0x71/0xa0 [ 4394.701212] check_noncircular+0x16f/0x190 [ 4394.702598] check_prev_add+0x98/0xa20 [ 4394.703937] validate_chain+0xa6c/0x29e0 [ 4394.705332] ? btrfs_sync_file+0x344/0x490 [btrfs] [ 4394.706947] ? btrfs_sync_file+0x344/0x490 [btrfs] [ 4394.708510] __lock_acquire+0x581/0xae0 [ 4394.709719] lock_acquire+0xa3/0x400 [ 4394.710743] ? btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.711960] __mutex_lock+0xa0/0xaf0 [ 4394.712969] ? btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.714301] ? lock_acquire+0xa3/0x400 [ 4394.715359] ? btrfs_chunk_alloc+0x60/0x3e0 [btrfs] [ 4394.716540] ? btrfs_chunk_alloc+0x115/0x3e0 [btrfs] [ 4394.718071] ? btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.719301] ? sched_clock+0x5/0x10 [ 4394.720295] ? sched_clock_cpu+0x15/0x130 [ 4394.721589] ? btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.723112] btrfs_chunk_alloc+0x137/0x3e0 [btrfs] [ 4394.724618] find_free_extent+0xb47/0xfb0 [btrfs] [ 4394.726111] ? kvm_sched_clock_read+0x14/0x30 [ 4394.727509] ? sched_clock+0x5/0x10 [ 4394.728768] btrfs_reserve_extent+0x9b/0x180 [btrfs] [ 4394.730391] btrfs_alloc_tree_block+0xc1/0x350 [btrfs] [ 4394.732123] alloc_tree_block_no_bg_flush+0x4a/0x60 [btrfs] [ 4394.733863] __btrfs_cow_block+0x143/0x7a0 [btrfs] [ 4394.735463] btrfs_cow_block+0x15f/0x310 [btrfs] [ 4394.736755] btrfs_search_slot+0x937/0xf70 [btrfs] [ 4394.738465] btrfs_insert_empty_items+0x64/0xc0 [btrfs] [ 4394.740230] __btrfs_commit_inode_delayed_items+0xaf/0x510 [btrfs] [ 4394.742450] __btrfs_run_delayed_items+0x8e/0x140 [btrfs] [ 4394.743916] btrfs_commit_transaction+0x312/0xae0 [btrfs] [ 4394.745409] btrfs_sync_file+0x345/0x490 [btrfs] [ 4394.746958] do_fsync+0x38/0x70 [ 4394.747900] __x64_sys_fsync+0x10/0x20 [ 4394.749010] do_syscall_64+0x50/0x210 [ 4394.750312] entry_SYSCALL_64_after_hwframe+0x49/0xb3 [ 4394.751628] RIP: 0033:0x7f139f754683 [ 4394.758709] RSP: 002b:00007ffc208bf158 EFLAGS: 00000246 ORIG_RAX: 000000000000004a [ 4394.761394] RAX: ffffffffffffffda RBX: 0000000000000032 RCX: 00007f139f754683 [ 4394.763330] RDX: 00007ffc208bf0c0 RSI: 00007ffc208bf0c0 RDI: 0000000000000003 [ 4394.764955] RBP: 0000000000000003 R08: 0000000000000001 R09: 00007ffc208bf16c [ 4394.766611] R10: 000000000040108e R11: 0000000000000246 R12: 0000000000000017 [ 4394.768113] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2020-05-19 14:26 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-12 14:12 Pending bugs for 5.7 David Sterba 2020-05-12 14:14 ` Bug 5.7-rc: root leak, eb leak David Sterba 2020-05-12 23:03 ` David Sterba 2020-05-13 11:54 ` Johannes Thumshirn 2020-05-13 11:57 ` Qu Wenruo 2020-05-13 12:06 ` Johannes Thumshirn 2020-05-13 12:11 ` Qu Wenruo 2020-05-13 12:17 ` Johannes Thumshirn 2020-05-13 12:29 ` Johannes Thumshirn 2020-05-12 14:15 ` Bug 5.7-rc: write-time leaf corruption detected David Sterba 2020-05-12 14:26 ` Filipe Manana 2020-05-13 3:10 ` Qu Wenruo 2020-05-13 3:17 ` Qu Wenruo 2020-05-13 9:25 ` Filipe Manana 2020-05-19 14:26 ` Bug 5.7-rc: write-time leaf corruption detected (fixed) David Sterba 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, chunk_mutex/device_list_mutex David Sterba 2020-05-12 23:28 ` David Sterba 2020-05-12 19:25 ` Anand Jain 2020-05-13 19:46 ` [PATCH] btrfs: fix lockdep warning chunk_mutex vs device_list_mutex Anand Jain 2020-05-15 17:40 ` David Sterba 2020-05-16 3:43 ` Anand Jain 2020-05-18 11:07 ` Anand Jain 2020-05-18 15:28 ` David Sterba 2020-05-12 14:15 ` Bug 5.7-rc: lockdep warning, fs_reclaim David Sterba
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.