* btrfs check (not lowmem) and OOM-like hangs (4.17.6)
@ 2018-07-17 20:32 Marc MERLIN
2018-07-17 20:59 ` Marc MERLIN
0 siblings, 1 reply; 479+ messages in thread
From: Marc MERLIN @ 2018-07-17 20:32 UTC (permalink / raw)
To: Su Yue; +Cc: Su Yue, quwenruo.btrfs, linux-btrfs
On Tue, Jul 17, 2018 at 10:50:32AM -0700, Marc MERLIN wrote:
> I got the following on 4.17.6 while running btrfs check --repair on an
> unmounted filesystem (not the lowmem version)
>
> I understand that btrfs check is userland only, although it seems that
> it caused these FS hangs on a different filesystem (the trace of course
> does not provide info on which FS)
>
> Any idea what happened here?
> I'm going to wait a few hours without running btrfs check to see if it
> happens again and then if running btrfs check will re-create this issue,
> but other suggestions (if any), are welcome:
Hi Qu, I know we were talking about this last week and then, btrfs check
just worked for me so I wasn't able to reproduce.
Now I'm able to reproduce again.
I tried again, it's definitely triggered by btrfs check --repair
I tried to capture what happens, and memory didn't dip to 0, but the system
got very slow and things started failing.
btrfs was never killed though while ssh was.
Is there a chance that maybe btrfs is in some kernel OOM exclude list?
Here is what I got when the system was not doing well (it took minutes to run):
total used free shared buffers cached
Mem: 32643788 32070952 572836 0 102160 4378772
-/+ buffers/cache: 27590020 5053768
Swap: 15616764 973596 14643168
gargamel:~# cat /proc/meminfo
MemTotal: 32643788 kB
MemFree: 2726276 kB
MemAvailable: 2502200 kB
Buffers: 12360 kB
Cached: 1676388 kB
SwapCached: 11048580 kB
Active: 16443004 kB
Inactive: 12010456 kB
Active(anon): 16287780 kB
Inactive(anon): 11651692 kB
Active(file): 155224 kB
Inactive(file): 358764 kB
Unevictable: 5776 kB
Mlocked: 5776 kB
SwapTotal: 15616764 kB
SwapFree: 294592 kB
Dirty: 3032 kB
Writeback: 76064 kB
AnonPages: 15723272 kB
Mapped: 612124 kB
Shmem: 1171032 kB
Slab: 399824 kB
SReclaimable: 84568 kB
SUnreclaim: 315256 kB
KernelStack: 20576 kB
PageTables: 94268 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 31938656 kB
Committed_AS: 37909452 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 98304 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 16384 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 355604 kB
DirectMap2M: 32897024 kB
and console:
[ 9184.345329] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9184.366258] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9184.385323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9184.408803] zmtrigger.pl D 0 9981 9804 0x20020080
[ 9184.425249] Call Trace:
[ 9184.432580] ? __schedule+0x53e/0x59b
[ 9184.443551] schedule+0x7f/0x98
[ 9184.452960] io_schedule+0x16/0x38
[ 9184.463154] wait_on_page_bit_common+0x10c/0x199
[ 9184.476996] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9184.493339] shmem_getpage_gfp+0x2dd/0x975
[ 9184.506558] shmem_fault+0x188/0x1c3
[ 9184.518199] ? filemap_map_pages+0x6f/0x295
[ 9184.531680] __do_fault+0x1d/0x6e
[ 9184.542505] __handle_mm_fault+0x675/0xa61
[ 9184.555653] ? list_move+0x21/0x3a
[ 9184.566737] handle_mm_fault+0x11c/0x16b
[ 9184.579355] __do_page_fault+0x324/0x41c
[ 9184.591996] ? page_fault+0x8/0x30
[ 9184.603059] page_fault+0x1e/0x30
[ 9184.613846] RIP: 0023:0xf7d2d022
[ 9184.624366] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9184.640868] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9184.663095] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9184.685308] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9184.707524] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9184.729757] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9184.751988] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9184.775106] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9184.795072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9184.819423] /usr/sbin/apach D 0 11868 11311 0x20020080
[ 9184.836748] Call Trace:
[ 9184.844926] ? __schedule+0x53e/0x59b
[ 9184.856811] schedule+0x7f/0x98
[ 9184.867075] io_schedule+0x16/0x38
[ 9184.878114] wait_on_page_bit_common+0x10c/0x199
[ 9184.892807] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9184.909036] shmem_getpage_gfp+0x2dd/0x975
[ 9184.922157] shmem_fault+0x188/0x1c3
[ 9184.933667] ? filemap_map_pages+0x6f/0x295
[ 9184.947504] __do_fault+0x1d/0x6e
[ 9184.958234] __handle_mm_fault+0x675/0xa61
[ 9184.971581] handle_mm_fault+0x11c/0x16b
[ 9184.984114] __do_page_fault+0x324/0x41c
[ 9184.996584] ? page_fault+0x8/0x30
[ 9185.007462] page_fault+0x1e/0x30
[ 9185.018085] RIP: 0023:0xf7aaa89e
[ 9185.028414] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9185.044686] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8
[ 9185.066697] RDX: 00000000f3664820 RSI: 0000000056e8ad20 RDI: 00000000f7aaee40
[ 9185.088720] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 9185.110713] R10: 0000000000000000 R11: 0000000000200282 R12: 0000000000000000
[ 9185.132677] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9305.182796] INFO: task atop:6459 blocked for more than 120 seconds.
[ 9305.202538] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.222445] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.246536] atop D 0 6459 1 0x20020080
[ 9305.263597] Call Trace:
[ 9305.271548] ? __schedule+0x53e/0x59b
[ 9305.283242] schedule+0x7f/0x98
[ 9305.293336] io_schedule+0x16/0x38
[ 9305.304602] wait_on_page_bit_common+0x10c/0x199
[ 9305.319074] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9305.335182] __lock_page_or_retry+0x6c/0x75
[ 9305.348811] do_swap_page+0x21f/0x511
[ 9305.360470] __handle_mm_fault+0x9c2/0xa61
[ 9305.373441] ? zone_statistics+0x4c/0x60
[ 9305.385854] handle_mm_fault+0x11c/0x16b
[ 9305.398797] __get_user_pages+0x364/0x505
[ 9305.411513] get_user_pages_remote+0x8e/0x18d
[ 9305.425209] __access_remote_vm+0x8c/0x1b4
[ 9305.438135] proc_pid_cmdline_read+0x13b/0x3dd
[ 9305.452150] ? do_filp_open+0x99/0xc6
[ 9305.463786] ? __vfs_read+0x33/0xcc
[ 9305.474901] __vfs_read+0x33/0xcc
[ 9305.485476] ? current_kernel_time64+0x10/0x36
[ 9305.499916] vfs_read+0x97/0xcc
[ 9305.510013] ksys_read+0x5c/0x9f
[ 9305.520301] do_fast_syscall_32+0xaf/0xf3
[ 9305.533157] entry_SYSENTER_compat+0x7f/0x91
[ 9305.546619] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9305.568354] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.588237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.612340] zmtrigger.pl D 0 9981 9804 0x20020080
[ 9305.629607] Call Trace:
[ 9305.637714] ? __schedule+0x53e/0x59b
[ 9305.649350] schedule+0x7f/0x98
[ 9305.659441] io_schedule+0x16/0x38
[ 9305.670283] wait_on_page_bit_common+0x10c/0x199
[ 9305.684771] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9305.700839] shmem_getpage_gfp+0x2dd/0x975
[ 9305.713854] shmem_fault+0x188/0x1c3
[ 9305.725230] ? filemap_map_pages+0x6f/0x295
[ 9305.738416] __do_fault+0x1d/0x6e
[ 9305.749000] __handle_mm_fault+0x675/0xa61
[ 9305.761923] ? list_move+0x21/0x3a
[ 9305.772857] handle_mm_fault+0x11c/0x16b
[ 9305.785262] __do_page_fault+0x324/0x41c
[ 9305.797733] ? page_fault+0x8/0x30
[ 9305.808581] page_fault+0x1e/0x30
[ 9305.819429] RIP: 0023:0xf7d2d022
[ 9305.829797] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9305.846085] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9305.868212] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9305.890279] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9305.912266] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9305.934291] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9305.956295] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9305.978927] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9305.998652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.022742] /usr/sbin/apach D 0 11868 11311 0x20020080
[ 9306.040041] Call Trace:
[ 9306.048029] ? __schedule+0x53e/0x59b
[ 9306.059703] schedule+0x7f/0x98
[ 9306.069772] io_schedule+0x16/0x38
[ 9306.080615] wait_on_page_bit_common+0x10c/0x199
[ 9306.095146] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.111423] shmem_getpage_gfp+0x2dd/0x975
[ 9306.124384] shmem_fault+0x188/0x1c3
[ 9306.135746] ? filemap_map_pages+0x6f/0x295
[ 9306.148943] __do_fault+0x1d/0x6e
[ 9306.159540] __handle_mm_fault+0x675/0xa61
[ 9306.172531] handle_mm_fault+0x11c/0x16b
[ 9306.184947] __do_page_fault+0x324/0x41c
[ 9306.197544] ? page_fault+0x8/0x30
[ 9306.208405] page_fault+0x1e/0x30
[ 9306.219016] RIP: 0023:0xf7aaa89e
[ 9306.229524] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9306.245994] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8
[ 9306.268285] RDX: 00000000f3664820 RSI: 0000000056e8ad20 RDI: 00000000f7aaee40
[ 9306.290641] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 9306.313028] R10: 0000000000000000 R11: 0000000000200282 R12: 0000000000000000
[ 9306.335441] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9306.357849] INFO: task pgrep:5570 blocked for more than 120 seconds.
[ 9306.377918] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9306.397989] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.422282] pgrep D 0 5570 5567 0x20020080
[ 9306.439795] Call Trace:
[ 9306.447754] ? __schedule+0x53e/0x59b
[ 9306.459419] schedule+0x7f/0x98
[ 9306.469460] io_schedule+0x16/0x38
[ 9306.480275] wait_on_page_bit_common+0x10c/0x199
[ 9306.494738] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.510783] __lock_page_or_retry+0x6c/0x75
[ 9306.523970] do_swap_page+0x21f/0x511
[ 9306.535593] ? btrfs_bio_counter_sub+0x1e/0x4c
[ 9306.549560] __handle_mm_fault+0x9c2/0xa61
[ 9306.562483] ? ktime_get+0x31/0x41
[ 9306.573324] handle_mm_fault+0x11c/0x16b
[ 9306.585723] __get_user_pages+0x364/0x505
[ 9306.598377] get_user_pages_remote+0x8e/0x18d
[ 9306.612083] __access_remote_vm+0x8c/0x1b4
[ 9306.625009] proc_pid_cmdline_read+0x13b/0x3dd
[ 9306.638977] ? do_filp_open+0x99/0xc6
[ 9306.650599] ? __vfs_read+0x33/0xcc
[ 9306.661690] __vfs_read+0x33/0xcc
[ 9306.672257] ? current_kernel_time64+0x10/0x36
[ 9306.686182] vfs_read+0x97/0xcc
[ 9306.696363] ksys_read+0x5c/0x9f
[ 9306.706638] do_fast_syscall_32+0xaf/0xf3
[ 9306.719293] entry_SYSENTER_compat+0x7f/0x91
[ 9306.732717] INFO: task pgrep:5579 blocked for more than 120 seconds.
[ 9306.752547] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9306.772223] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9306.796319] pgrep D 0 5579 4327 0x20020080
[ 9306.813577] Call Trace:
[ 9306.821562] ? __schedule+0x53e/0x59b
[ 9306.833180] schedule+0x7f/0x98
[ 9306.843275] io_schedule+0x16/0x38
[ 9306.854164] wait_on_page_bit_common+0x10c/0x199
[ 9306.868651] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9306.884730] __lock_page_or_retry+0x6c/0x75
[ 9306.897942] do_swap_page+0x21f/0x511
[ 9306.909566] ? aac_scsi_cmd+0x455/0x117b [aacraid]
[ 9306.924630] __handle_mm_fault+0x9c2/0xa61
[ 9306.937594] ? ktime_get+0x31/0x41
[ 9306.948478] handle_mm_fault+0x11c/0x16b
[ 9306.961081] __get_user_pages+0x364/0x505
[ 9306.973748] get_user_pages_remote+0x8e/0x18d
[ 9306.987598] __access_remote_vm+0x8c/0x1b4
[ 9307.000744] proc_pid_cmdline_read+0x13b/0x3dd
[ 9307.014868] ? do_filp_open+0x99/0xc6
[ 9307.026676] ? __vfs_read+0x33/0xcc
[ 9307.037927] __vfs_read+0x33/0xcc
[ 9307.048670] ? current_kernel_time64+0x10/0x36
[ 9307.062789] vfs_read+0x97/0xcc
[ 9307.072860] ksys_read+0x5c/0x9f
[ 9307.083171] do_fast_syscall_32+0xaf/0xf3
[ 9307.095810] entry_SYSENTER_compat+0x7f/0x91
[ 9428.060462] INFO: task atop:6459 blocked for more than 120 seconds.
[ 9428.080112] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.099816] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.123975] atop D 0 6459 1 0x20020080
[ 9428.141515] Call Trace:
[ 9428.149507] ? __schedule+0x53e/0x59b
[ 9428.161247] schedule+0x7f/0x98
[ 9428.171320] io_schedule+0x16/0x38
[ 9428.182181] wait_on_page_bit_common+0x10c/0x199
[ 9428.196674] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.212767] __lock_page_or_retry+0x6c/0x75
[ 9428.225983] do_swap_page+0x21f/0x511
[ 9428.237625] __handle_mm_fault+0x9c2/0xa61
[ 9428.250579] ? zone_statistics+0x4c/0x60
[ 9428.262977] handle_mm_fault+0x11c/0x16b
[ 9428.275429] __get_user_pages+0x364/0x505
[ 9428.288117] get_user_pages_remote+0x8e/0x18d
[ 9428.301873] __access_remote_vm+0x8c/0x1b4
[ 9428.314854] proc_pid_cmdline_read+0x13b/0x3dd
[ 9428.328897] ? do_filp_open+0x99/0xc6
[ 9428.340536] ? __vfs_read+0x33/0xcc
[ 9428.351626] __vfs_read+0x33/0xcc
[ 9428.362181] ? current_kernel_time64+0x10/0x36
[ 9428.376107] vfs_read+0x97/0xcc
[ 9428.386112] ksys_read+0x5c/0x9f
[ 9428.396406] do_fast_syscall_32+0xaf/0xf3
[ 9428.409029] entry_SYSENTER_compat+0x7f/0x91
[ 9428.422465] INFO: task zmtrigger.pl:9981 blocked for more than 120 seconds.
[ 9428.443947] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.463635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.487740] zmtrigger.pl D 0 9981 9804 0x20020080
[ 9428.504827] Call Trace:
[ 9428.512804] ? __schedule+0x53e/0x59b
[ 9428.524426] schedule+0x7f/0x98
[ 9428.534486] io_schedule+0x16/0x38
[ 9428.545325] wait_on_page_bit_common+0x10c/0x199
[ 9428.559813] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.575860] shmem_getpage_gfp+0x2dd/0x975
[ 9428.588802] shmem_fault+0x188/0x1c3
[ 9428.600161] ? filemap_map_pages+0x6f/0x295
[ 9428.613348] __do_fault+0x1d/0x6e
[ 9428.623960] __handle_mm_fault+0x675/0xa61
[ 9428.636934] ? list_move+0x21/0x3a
[ 9428.647968] handle_mm_fault+0x11c/0x16b
[ 9428.660517] __do_page_fault+0x324/0x41c
[ 9428.672990] ? page_fault+0x8/0x30
[ 9428.683846] page_fault+0x1e/0x30
[ 9428.694437] RIP: 0023:0xf7d2d022
[ 9428.704744] RSP: 002b:00000000ffeb9fe8 EFLAGS: 00010202
[ 9428.721025] RAX: 00000000f7eed000 RBX: 00000000567e6000 RCX: 0000000000000004
[ 9428.743062] RDX: 00000000587fecb0 RSI: 000000005876538c RDI: 0000000000000004
[ 9428.765091] RBP: 0000000058185160 R08: 0000000000000000 R09: 0000000000000000
[ 9428.787241] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[ 9428.809265] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 9428.831501] INFO: task /usr/sbin/apach:11868 blocked for more than 120 seconds.
[ 9428.854070] Not tainted 4.17.6-amd64-preempt-sysrq-20180818 #4
[ 9428.873815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9428.897937] /usr/sbin/apach D 0 11868 11311 0x20020080
[ 9428.915023] Call Trace:
[ 9428.923002] ? __schedule+0x53e/0x59b
[ 9428.934625] schedule+0x7f/0x98
[ 9428.944687] io_schedule+0x16/0x38
[ 9428.955525] wait_on_page_bit_common+0x10c/0x199
[ 9428.970014] ? file_check_and_advance_wb_err+0xd7/0xd7
[ 9428.986074] shmem_getpage_gfp+0x2dd/0x975
[ 9428.999006] shmem_fault+0x188/0x1c3
[ 9429.010366] ? filemap_map_pages+0x6f/0x295
[ 9429.023551] __do_fault+0x1d/0x6e
[ 9429.034140] __handle_mm_fault+0x675/0xa61
[ 9429.047056] handle_mm_fault+0x11c/0x16b
[ 9429.059461] __do_page_fault+0x324/0x41c
[ 9429.071857] ? page_fault+0x8/0x30
[ 9429.082703] page_fault+0x1e/0x30
[ 9429.093276] RIP: 0023:0xf7aaa89e
[ 9429.103591] RSP: 002b:00000000ffb26f70 EFLAGS: 00210297
[ 9429.119892] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000f7eb9ed8
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6) 2018-07-17 20:32 btrfs check (not lowmem) and OOM-like hangs (4.17.6) Marc MERLIN @ 2018-07-17 20:59 ` Marc MERLIN 2018-07-18 0:05 ` Qu Wenruo 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2018-07-17 20:59 UTC (permalink / raw) To: Su Yue; +Cc: Su Yue, quwenruo.btrfs, linux-btrfs Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers. Details below: On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: > Here is what I got when the system was not doing well (it took minutes to run): > > total used free shared buffers cached > Mem: 32643788 32070952 572836 0 102160 4378772 > -/+ buffers/cache: 27590020 5053768 > Swap: 15616764 973596 14643168 ok, the reason it was not that close to 0 was due to /dev/shm it seems. I cleared that, and now I can get it to go to near 0 again. I'm wrong about the system being fully crashed, it's not, it's just very close to being hung. I can type killall -9 btrfs in the serial console and wait a few minutes. The system eventually recovers, but it's impossible to fix anything via ssh apparently because networking does not get to run when I'm in this state. I'm not sure why my system reproduces this easy while Qu's system does not, but Qu was right that the kernel is not dead and that it's merely a problem of userspace taking all the RAM and somehow not being killed by OOM I checked the PID and don't see why it's not being killed: gargamel:/proc/31006# grep . oom* oom_adj:0 oom_score:221 << this increases a lot, but OOM never kills it oom_score_adj:0 I have these variables: /proc/sys/vm/oom_dump_tasks:1 /proc/sys/vm/oom_kill_allocating_task:0 /proc/sys/vm/overcommit_kbytes:0 /proc/sys/vm/overcommit_memory:0 /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) Here is my system when it virtually died: ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 total used free shared buffers cached Mem: 32643788 32180100 463688 0 44664 119508 -/+ buffers/cache: 32015928 627860 Swap: 15616764 443676 15173088 MemTotal: 32643788 kB MemFree: 463440 kB MemAvailable: 44864 kB Buffers: 44664 kB Cached: 120360 kB SwapCached: 87064 kB Active: 30381404 kB Inactive: 585952 kB Active(anon): 30334696 kB Inactive(anon): 474624 kB Active(file): 46708 kB Inactive(file): 111328 kB Unevictable: 5616 kB Mlocked: 5616 kB SwapTotal: 15616764 kB SwapFree: 15173088 kB Dirty: 1636 kB Writeback: 4 kB AnonPages: 30734240 kB Mapped: 67236 kB Shmem: 3036 kB Slab: 267884 kB SReclaimable: 51528 kB SUnreclaim: 216356 kB KernelStack: 10144 kB PageTables: 69284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 31938656 kB Committed_AS: 32865492 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 16384 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 560404 kB DirectMap2M: 32692224 kB -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6) 2018-07-17 20:59 ` Marc MERLIN @ 2018-07-18 0:05 ` Qu Wenruo 2018-07-18 0:24 ` Marc MERLIN 2018-07-18 19:42 ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov 0 siblings, 2 replies; 479+ messages in thread From: Qu Wenruo @ 2018-07-18 0:05 UTC (permalink / raw) To: Marc MERLIN, Su Yue; +Cc: Su Yue, linux-btrfs On 2018年07月18日 04:59, Marc MERLIN wrote: > Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. > It just takes all the memory until linux hangs everywhere, and somehow (no idea why) > the OOM killer never triggers. No OOM triggers? That's a little strange. Maybe it's related to how kernel handles memory over-commit? And for the hang, I think it's related to some memory allocation failure and error handler just didn't handle it well, so it's causing deadlock for certain page. ENOMEM handling is pretty common but hardly verified, so it's not that strange, but we must locate the problem. > Details below: > > On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: >> Here is what I got when the system was not doing well (it took minutes to run): >> >> total used free shared buffers cached >> Mem: 32643788 32070952 572836 0 102160 4378772 >> -/+ buffers/cache: 27590020 5053768 >> Swap: 15616764 973596 14643168 > > ok, the reason it was not that close to 0 was due to /dev/shm it seems. > I cleared that, and now I can get it to go to near 0 again. > I'm wrong about the system being fully crashed, it's not, it's just very > close to being hung. > I can type killall -9 btrfs in the serial console and wait a few minutes. > The system eventually recovers, but it's impossible to fix anything via ssh > apparently because networking does not get to run when I'm in this state. > > I'm not sure why my system reproduces this easy while Qu's system does not, > but Qu was right that the kernel is not dead and that it's merely a problem of userspace > taking all the RAM and somehow not being killed by OOM In my system, at least I'm not using btrfs as root fs, and for the memory eating program I normally ensure it's eating all the memory + swap, so OOM killer is always triggered, maybe that's the cause. So in your case, maybe it's btrfs not really taking up all memory, thus OOM killer not triggered. > > I checked the PID and don't see why it's not being killed: > gargamel:/proc/31006# grep . oom* > oom_adj:0 > oom_score:221 << this increases a lot, but OOM never kills it > oom_score_adj:0 > > I have these variables: > /proc/sys/vm/oom_dump_tasks:1 > /proc/sys/vm/oom_kill_allocating_task:0 > /proc/sys/vm/overcommit_kbytes:0 > /proc/sys/vm/overcommit_memory:0 > /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) Any kernel dmesg about OOM killer triggered? > > Here is my system when it virtually died: > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > total used free shared buffers cached > Mem: 32643788 32180100 463688 0 44664 119508 > -/+ buffers/cache: 32015928 627860 > Swap: 15616764 443676 15173088 For swap, it looks like only some other program's memory is swapped out, not btrfs'. And unfortunately, I'm not so familiar with OOM/MM code outside of filesystem. Any help from other experienced developers would definitely help to solve why memory of 'btrfs check' is not swapped out or why OOM killer is not triggered. Thanks, Qu > > MemTotal: 32643788 kB > MemFree: 463440 kB > MemAvailable: 44864 kB > Buffers: 44664 kB > Cached: 120360 kB > SwapCached: 87064 kB > Active: 30381404 kB > Inactive: 585952 kB > Active(anon): 30334696 kB > Inactive(anon): 474624 kB > Active(file): 46708 kB > Inactive(file): 111328 kB > Unevictable: 5616 kB > Mlocked: 5616 kB > SwapTotal: 15616764 kB > SwapFree: 15173088 kB > Dirty: 1636 kB > Writeback: 4 kB > AnonPages: 30734240 kB > Mapped: 67236 kB > Shmem: 3036 kB > Slab: 267884 kB > SReclaimable: 51528 kB > SUnreclaim: 216356 kB > KernelStack: 10144 kB > PageTables: 69284 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 31938656 kB > Committed_AS: 32865492 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > HardwareCorrupted: 0 kB > AnonHugePages: 0 kB > ShmemHugePages: 0 kB > ShmemPmdMapped: 0 kB > CmaTotal: 16384 kB > CmaFree: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > Hugetlb: 0 kB > DirectMap4k: 560404 kB > DirectMap2M: 32692224 kB > > ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6) 2018-07-18 0:05 ` Qu Wenruo @ 2018-07-18 0:24 ` Marc MERLIN 2018-07-19 9:22 ` Qu Wenruo ` (2 more replies) 2018-07-18 19:42 ` btrfs check (not lowmem) and OOM-like hangs (4.17.6) Andrei Borzenkov 1 sibling, 3 replies; 479+ messages in thread From: Marc MERLIN @ 2018-07-18 0:24 UTC (permalink / raw) To: Qu Wenruo; +Cc: linux-btrfs On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: > No OOM triggers? That's a little strange. > Maybe it's related to how kernel handles memory over-commit? Yes, I think you are correct. > And for the hang, I think it's related to some memory allocation failure > and error handler just didn't handle it well, so it's causing deadlock > for certain page. That indeed matches what I'm seeing. > ENOMEM handling is pretty common but hardly verified, so it's not that > strange, but we must locate the problem. I seem to be getting deadlocks in the kernel, so I'm hoping that at least it's checked there, but maybe not? > In my system, at least I'm not using btrfs as root fs, and for the > memory eating program I normally ensure it's eating all the memory + > swap, so OOM killer is always triggered, maybe that's the cause. > > So in your case, maybe it's btrfs not really taking up all memory, thus > OOM killer not triggered. Correct, the swap is not used. > Any kernel dmesg about OOM killer triggered? Nothing at all. It never gets triggered. > > Here is my system when it virtually died: > > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 See how btrs was taking 29GB in that ps output (that's before it takes everything and I can't even type ps anymore) Note that VSZ is almost equal to RSS. Nothing gets swapped. Then see free output: > > total used free shared buffers cached > > Mem: 32643788 32180100 463688 0 44664 119508 > > -/+ buffers/cache: 32015928 627860 > > Swap: 15616764 443676 15173088 > > For swap, it looks like only some other program's memory is swapped out, > not btrfs'. That's exactly correct. btrfs check never goes to swap, I'm not sure why, and because there is virtual memory free, maybe that's why OOM does not trigger? So I guess I can probably "fix" my problem by removing swap, but ultimately it would be useful to know why memory taken by btrfs check does not end up in swap. > And unfortunately, I'm not so familiar with OOM/MM code outside of > filesystem. > Any help from other experienced developers would definitely help to > solve why memory of 'btrfs check' is not swapped out or why OOM killer > is not triggered. Do you have someone from linux-vm you might be able to ask, or should we Cc this thread there? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6) 2018-07-18 0:24 ` Marc MERLIN @ 2018-07-19 9:22 ` Qu Wenruo 2019-10-18 2:56 ` 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 Marc MERLIN 2020-05-24 21:30 ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN 2 siblings, 0 replies; 479+ messages in thread From: Qu Wenruo @ 2018-07-19 9:22 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs [-- Attachment #1.1: Type: text/plain, Size: 3526 bytes --] On 2018年07月18日 08:24, Marc MERLIN wrote: > On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: >> No OOM triggers? That's a little strange. >> Maybe it's related to how kernel handles memory over-commit? > > Yes, I think you are correct. > >> And for the hang, I think it's related to some memory allocation failure >> and error handler just didn't handle it well, so it's causing deadlock >> for certain page. > > That indeed matches what I'm seeing. > >> ENOMEM handling is pretty common but hardly verified, so it's not that >> strange, but we must locate the problem. > > I seem to be getting deadlocks in the kernel, so I'm hoping that at least > it's checked there, but maybe not? > >> In my system, at least I'm not using btrfs as root fs, and for the >> memory eating program I normally ensure it's eating all the memory + >> swap, so OOM killer is always triggered, maybe that's the cause. >> >> So in your case, maybe it's btrfs not really taking up all memory, thus >> OOM killer not triggered. > > Correct, the swap is not used. > >> Any kernel dmesg about OOM killer triggered? > > Nothing at all. It never gets triggered. > >>> Here is my system when it virtually died: >>> ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND >>> root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > See how btrs was taking 29GB in that ps output (that's before it takes > everything and I can't even type ps anymore) > Note that VSZ is almost equal to RSS. Nothing gets swapped. > > Then see free output: > >>> total used free shared buffers cached >>> Mem: 32643788 32180100 463688 0 44664 119508 >>> -/+ buffers/cache: 32015928 627860 >>> Swap: 15616764 443676 15173088 >> >> For swap, it looks like only some other program's memory is swapped out, >> not btrfs'. > > That's exactly correct. btrfs check never goes to swap, I'm not sure why, > and because there is virtual memory free, maybe that's why OOM does not > trigger? > So I guess I can probably "fix" my problem by removing swap, but ultimately > it would be useful to know why memory taken by btrfs check does not end up > in swap. > >> And unfortunately, I'm not so familiar with OOM/MM code outside of >> filesystem. >> Any help from other experienced developers would definitely help to >> solve why memory of 'btrfs check' is not swapped out or why OOM killer >> is not triggered. > > Do you have someone from linux-vm you might be able to ask, or should we Cc > this thread there? Michal Hocho gives me a brief session about this. Which is super helpful in this case, thank you Michal! Firstly, btrfs-progs usage of malloc() will result anonymous pages, thus they can be swapped out. Secondly, kernel doesn't like to swap out anonymous pages at all, thus kernel won't try to aggressively swap out such pages. Thirdly, for user anonymous memory, there is LRU-like algorithm to determine which memory should go swapped out. But considering how btrfs check uses pages, it would only make it harder to be swapped out. So it's not an easy way thing to aggressively swap out memory of btrfs check to swap. Although Michal mentioned some cgroup way to limit the memory usage so it can be more aggressively swapped out, I'm still digging into it. Thanks, Qu > > Thanks, > Marc > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 479+ messages in thread
* 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 @ 2019-10-18 2:56 ` Marc MERLIN 2019-10-19 3:07 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2019-10-18 2:56 UTC (permalink / raw) To: Linux BTRFS Mailinglist This happened almost after a resume from suspend to disk. First corruption and read only I got a very long time. Could they be related? [26062.126505] ------------[ cut here ]------------ [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda _intel [26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.126636] Call Trace: [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 [26062.126653] ? __switch_to_asm+0x41/0x70 [26062.126655] ? __switch_to_asm+0x35/0x70 [26062.126658] ? __switch_to_asm+0x41/0x70 [26062.126662] ? __switch_to+0x13d/0x3d5 [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 [26062.126672] btrfs_commit_transaction+0x55/0x7c8 [26062.126676] ? start_transaction+0x347/0x3cb [26062.126679] transaction_kthread+0xc9/0x135 [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 [26062.126688] kthread+0xeb/0xf0 [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.126695] ret_from_fork+0x35/0x40 [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 [26062.126708] extent refs 2 gen 2369265 flags 1 [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 [26062.126711] ref#1: shared data backref parent 437615230976 count 1 [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 [26062.126715] extent refs 2 gen 2369025 flags 1 [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 [26062.126718] ref#1: shared data backref parent 437615230976 count 1 [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 [26062.126722] extent refs 11 gen 1800715 flags 1 [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 [26062.126724] ref#1: shared data backref parent 508153839616 count 1 [26062.126726] ref#2: shared data backref parent 493541244928 count 1 [26062.126727] ref#3: shared data backref parent 492677332992 count 1 [26062.126728] ref#4: shared data backref parent 492328566784 count 1 [26062.126730] ref#5: shared data backref parent 492245188608 count 1 [26062.126731] ref#6: shared data backref parent 475885109248 count 1 [26062.126732] ref#7: shared data backref parent 471036608512 count 1 [26062.126733] ref#8: shared data backref parent 436114243584 count 1 [26062.126735] ref#9: shared data backref parent 886718464 count 1 [26062.126736] ref#10: shared data backref parent 90259456 count 1 [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 [26062.126740] extent refs 2 gen 2420471 flags 1 [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 [26062.126743] ref#1: shared data backref parent 41020112896 count 1 [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 [26062.126746] extent refs 1 gen 2422941 flags 1 [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 [26062.126750] extent refs 1 gen 1444854 flags 1 [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 [26062.126754] extent refs 1 gen 1444854 flags 1 [26062.126755] ref#0: shared data backref parent 214892544 count 1 [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 [26062.126758] extent refs 1 gen 1444854 flags 1 [26062.126759] ref#0: shared data backref parent 102547456 count 1 [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 [26062.126762] extent refs 2 gen 2422644 flags 1 [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 [26062.126765] ref#1: shared data backref parent 470658482176 count 1 [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 [26062.126769] extent refs 2 gen 2369217 flags 1 [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 [26062.126772] ref#1: shared data backref parent 437615230976 count 1 [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 [26062.126775] extent refs 2 gen 2369242 flags 1 [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 [26062.126778] ref#1: shared data backref parent 437615230976 count 1 [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 [26062.126781] extent refs 1 gen 2369305 flags 1 [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 [26062.126786] extent refs 2 gen 2369052 flags 1 [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 [26062.126788] ref#1: shared data backref parent 437615230976 count 1 [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 [26062.126792] extent refs 1 gen 2369305 flags 1 [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 [26062.126796] extent refs 2 gen 2368754 flags 1 [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 [26062.126799] ref#1: shared data backref parent 437615230976 count 1 [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 [26062.126802] extent refs 11 gen 1800682 flags 1 [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 [26062.126805] ref#1: shared data backref parent 508153069568 count 1 [26062.126806] ref#2: shared data backref parent 508031270912 count 1 [26062.126808] ref#3: shared data backref parent 493564215296 count 1 [26062.126809] ref#4: shared data backref parent 493540098048 count 1 [26062.126811] ref#5: shared data backref parent 492676038656 count 1 [26062.126813] ref#6: shared data backref parent 475884142592 count 1 [26062.126814] ref#7: shared data backref parent 471032840192 count 1 [26062.126816] ref#8: shared data backref parent 436112146432 count 1 [26062.126817] ref#9: shared data backref parent 883752960 count 1 [26062.126818] ref#10: shared data backref parent 86654976 count 1 [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 [26062.126822] extent refs 2 gen 2369347 flags 1 [26062.126823] ref#0: shared data backref parent 492981370880 count 1 [26062.126824] ref#1: shared data backref parent 438017720320 count 1 [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 [26062.126827] extent refs 2 gen 2369347 flags 1 [26062.126828] ref#0: shared data backref parent 492981370880 count 1 [26062.126830] ref#1: shared data backref parent 438017720320 count 1 [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 [26062.126833] extent refs 2 gen 2369347 flags 1 [26062.126834] ref#0: shared data backref parent 492981370880 count 1 [26062.126836] ref#1: shared data backref parent 438017720320 count 1 [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 [26062.126840] extent refs 1 gen 2369352 flags 1 [26062.126841] ref#0: shared data backref parent 510058790912 count 1 [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 [26062.126845] extent refs 1 gen 2369352 flags 1 [26062.126845] ref#0: shared data backref parent 510058790912 count 1 [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 [26062.126849] extent refs 1 gen 2369352 flags 1 [26062.126850] ref#0: shared data backref parent 510058790912 count 1 [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 [26062.126853] extent refs 2 gen 2369347 flags 1 [26062.126854] ref#0: shared data backref parent 492981370880 count 1 [26062.126856] ref#1: shared data backref parent 438017720320 count 1 [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 [26062.126859] extent refs 2 gen 2369347 flags 1 [26062.126860] ref#0: shared data backref parent 492981370880 count 1 [26062.126862] ref#1: shared data backref parent 438017720320 count 1 [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 [26062.126865] extent refs 2 gen 2369347 flags 1 [26062.126866] ref#0: shared data backref parent 492981370880 count 1 [26062.126868] ref#1: shared data backref parent 438017720320 count 1 [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 [26062.126871] extent refs 2 gen 2369347 flags 1 [26062.126873] ref#0: shared data backref parent 492981370880 count 1 [26062.126874] ref#1: shared data backref parent 438017720320 count 1 [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 [26062.126877] extent refs 2 gen 2369347 flags 1 [26062.126878] ref#0: shared data backref parent 492981370880 count 1 [26062.126880] ref#1: shared data backref parent 438017720320 count 1 [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 [26062.126883] extent refs 2 gen 2369347 flags 1 [26062.126884] ref#0: shared data backref parent 492981370880 count 1 [26062.126885] ref#1: shared data backref parent 438017720320 count 1 [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 [26062.126889] extent refs 2 gen 2369347 flags 1 [26062.126890] ref#0: shared data backref parent 492981370880 count 1 [26062.126891] ref#1: shared data backref parent 438017720320 count 1 [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 [26062.126894] extent refs 3 gen 2369347 flags 1 [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 [26062.126897] ref#1: shared data backref parent 492981370880 count 1 [26062.126898] ref#2: shared data backref parent 438017720320 count 1 [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 [26062.126902] extent refs 2 gen 2369347 flags 1 [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 [26062.126904] ref#1: shared data backref parent 438017720320 count 1 [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 [26062.126907] extent refs 1 gen 2369352 flags 1 [26062.126908] ref#0: shared data backref parent 510058790912 count 1 [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 [26062.126912] extent refs 6 gen 1800750 flags 1 [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 [26062.126915] ref#1: shared data backref parent 509782933504 count 2 [26062.126917] ref#2: shared data backref parent 508543795200 count 2 [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 [26062.126920] extent refs 2 gen 2369075 flags 1 [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 [26062.126923] ref#1: shared data backref parent 437615230976 count 1 [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 [26062.126926] extent refs 3 gen 1800733 flags 1 [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 [26062.126929] ref#1: shared data backref parent 509783719936 count 1 [26062.126930] ref#2: shared data backref parent 509351231488 count 1 [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 [26062.126934] extent refs 2 gen 2420471 flags 1 [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 [26062.126936] ref#1: shared data backref parent 41020112896 count 1 [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 [26062.126939] extent refs 2 gen 2420471 flags 1 [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 [26062.126943] ref#1: shared data backref parent 41020112896 count 1 [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 [26062.126946] extent refs 3 gen 1800733 flags 1 [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 [26062.126949] ref#1: shared data backref parent 509783719936 count 1 [26062.126950] ref#2: shared data backref parent 509351231488 count 1 [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 [26062.126954] extent refs 2 gen 2420502 flags 1 [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 [26062.126957] ref#1: shared data backref parent 492274728960 count 1 [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 [26062.126961] extent refs 2 gen 2420478 flags 1 [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 [26062.126964] ref#1: shared data backref parent 41020112896 count 1 [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 [26062.126967] extent refs 1 gen 1444854 flags 1 [26062.126968] ref#0: shared data backref parent 214908928 count 1 [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 [26062.126971] extent refs 1 gen 2376479 flags 1 [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 [26062.126976] extent refs 1 gen 1444854 flags 1 [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 [26062.126981] extent refs 1 gen 1444854 flags 1 [26062.126982] ref#0: shared data backref parent 93732864 count 1 [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 [26062.126985] extent refs 1 gen 1444854 flags 1 [26062.126986] ref#0: shared data backref parent 93732864 count 1 [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 [26062.126990] extent refs 1 gen 1444854 flags 1 [26062.126991] ref#0: shared data backref parent 93732864 count 1 [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 [26062.126994] extent refs 1 gen 1444854 flags 1 [26062.126995] ref#0: shared data backref parent 214908928 count 1 [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 [26062.126998] extent refs 4 gen 2419085 flags 1 [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 [26062.127001] ref#1: shared data backref parent 494075035648 count 1 [26062.127002] ref#2: shared data backref parent 470850240512 count 1 [26062.127004] ref#3: shared data backref parent 437034745856 count 1 [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 [26062.127008] extent refs 3 gen 1800733 flags 1 [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 [26062.127011] ref#1: shared data backref parent 509783719936 count 1 [26062.127013] ref#2: shared data backref parent 509351231488 count 1 [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 [26062.127016] extent refs 2 gen 2368859 flags 1 [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 [26062.127019] ref#1: shared data backref parent 437615230976 count 1 [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 [26062.127023] extent refs 1 gen 2376479 flags 1 [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 [26062.127028] extent refs 11 gen 1800674 flags 1 [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 [26062.127031] ref#1: shared data backref parent 508153118720 count 1 [26062.127032] ref#2: shared data backref parent 508031533056 count 1 [26062.127034] ref#3: shared data backref parent 493564477440 count 1 [26062.127036] ref#4: shared data backref parent 493540179968 count 1 [26062.127038] ref#5: shared data backref parent 492675366912 count 1 [26062.127039] ref#6: shared data backref parent 475132116992 count 1 [26062.127041] ref#7: shared data backref parent 471032922112 count 1 [26062.127042] ref#8: shared data backref parent 436112244736 count 1 [26062.127044] ref#9: shared data backref parent 883998720 count 1 [26062.127046] ref#10: shared data backref parent 87015424 count 1 [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 [26062.127050] extent refs 1 gen 2408131 flags 1 [26062.127052] ref#0: shared data backref parent 509278633984 count 1 [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 [26062.127056] extent refs 11 gen 1800715 flags 1 [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 [26062.127059] ref#1: shared data backref parent 508152889344 count 1 [26062.127061] ref#2: shared data backref parent 508031139840 count 1 [26062.127062] ref#3: shared data backref parent 493563478016 count 1 [26062.127065] ref#4: shared data backref parent 493539983360 count 1 [26062.127066] ref#5: shared data backref parent 492674990080 count 1 [26062.127068] ref#6: shared data backref parent 475132166144 count 1 [26062.127070] ref#7: shared data backref parent 471032348672 count 1 [26062.127071] ref#8: shared data backref parent 436111769600 count 1 [26062.127073] ref#9: shared data backref parent 883359744 count 1 [26062.127074] ref#10: shared data backref parent 85082112 count 1 [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 [26062.127078] extent refs 4 gen 2419085 flags 1 [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 [26062.127085] ref#1: shared data backref parent 494075035648 count 1 [26062.127088] ref#2: shared data backref parent 470850240512 count 1 [26062.127089] ref#3: shared data backref parent 437034745856 count 1 [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 [26062.127095] extent refs 2 gen 2421893 flags 1 [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 [26062.127099] ref#1: shared data backref parent 493703069696 count 1 [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 [26062.127104] extent refs 1 gen 1444854 flags 1 [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 [26062.127111] extent refs 2 gen 2368700 flags 1 [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 [26062.127114] ref#1: shared data backref parent 494105804800 count 1 [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 [26062.127120] extent refs 1 gen 1444854 flags 1 [26062.127121] ref#0: shared data backref parent 93732864 count 1 [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 [26062.127127] extent refs 1 gen 1444854 flags 1 [26062.127128] ref#0: shared data backref parent 214433792 count 1 [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 [26062.127132] extent refs 1 gen 1444854 flags 1 [26062.127133] ref#0: shared data backref parent 214908928 count 1 [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 [26062.127138] extent refs 3 gen 1800750 flags 1 [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 [26062.127141] ref#1: shared data backref parent 509783556096 count 1 [26062.127143] ref#2: shared data backref parent 508544974848 count 1 [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 [26062.127148] extent refs 2 gen 2368707 flags 1 [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 [26062.127151] ref#1: shared data backref parent 437615230976 count 1 [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 [26062.127155] extent refs 11 gen 1800674 flags 1 [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 [26062.127157] ref#1: shared data backref parent 508152905728 count 1 [26062.127159] ref#2: shared data backref parent 508031057920 count 1 [26062.127160] ref#3: shared data backref parent 493563166720 count 1 [26062.127161] ref#4: shared data backref parent 493539934208 count 1 [26062.127163] ref#5: shared data backref parent 492674908160 count 1 [26062.127164] ref#6: shared data backref parent 475884208128 count 1 [26062.127165] ref#7: shared data backref parent 471032266752 count 1 [26062.127167] ref#8: shared data backref parent 436111720448 count 1 [26062.127168] ref#9: shared data backref parent 883179520 count 1 [26062.127170] ref#10: shared data backref parent 83017728 count 1 [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 [26062.127173] extent refs 1 gen 1444854 flags 1 [26062.127174] ref#0: shared data backref parent 214908928 count 1 [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 [26062.127177] extent refs 2 gen 2381729 flags 1 [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 [26062.127180] ref#1: shared data backref parent 437808545792 count 1 [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 [26062.127183] extent refs 2 gen 2369098 flags 1 [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 [26062.127186] ref#1: shared data backref parent 437615230976 count 1 [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 [26062.127189] extent refs 2 gen 2368700 flags 1 [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 [26062.127191] ref#1: shared data backref parent 494105804800 count 1 [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 [26062.127195] extent refs 1 gen 1444854 flags 1 [26062.127195] ref#0: shared data backref parent 102547456 count 1 [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 [26062.127198] extent refs 3 gen 1800750 flags 1 [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 [26062.127201] ref#1: shared data backref parent 509783408640 count 1 [26062.127202] ref#2: shared data backref parent 508543451136 count 1 [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 [26062.127205] extent refs 1 gen 1444854 flags 1 [26062.127206] ref#0: shared data backref parent 93732864 count 1 [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 [26062.127209] extent refs 1 gen 1444854 flags 1 [26062.127210] ref#0: shared data backref parent 93732864 count 1 [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 [26062.127214] extent refs 1 gen 1444854 flags 1 [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 [26062.127219] extent refs 4 gen 2419085 flags 1 [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 [26062.127222] ref#1: shared data backref parent 494075035648 count 1 [26062.127223] ref#2: shared data backref parent 470850240512 count 1 [26062.127225] ref#3: shared data backref parent 437034745856 count 1 [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 [26062.127228] extent refs 3 gen 1800750 flags 1 [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 [26062.127230] ref#1: shared data backref parent 509783408640 count 1 [26062.127232] ref#2: shared data backref parent 508543451136 count 1 [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 [26062.127235] extent refs 2 gen 2368731 flags 1 [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 [26062.127237] ref#1: shared data backref parent 437615230976 count 1 [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 [26062.127241] extent refs 1 gen 1444854 flags 1 [26062.127241] ref#0: shared data backref parent 93732864 count 1 [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 [26062.127244] extent refs 1 gen 1444854 flags 1 [26062.127245] ref#0: shared data backref parent 93732864 count 1 [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 [26062.127249] extent refs 1 gen 1444854 flags 1 [26062.127249] ref#0: shared data backref parent 93732864 count 1 [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 [26062.127253] extent refs 1 gen 1444854 flags 1 [26062.127254] ref#0: shared data backref parent 93732864 count 1 [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 [26062.127258] extent refs 1 gen 1444854 flags 1 [26062.127259] ref#0: shared data backref parent 93732864 count 1 [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 [26062.127263] extent refs 1 gen 2381741 flags 1 [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 [26062.127268] extent refs 2 gen 2368882 flags 1 [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 [26062.127270] ref#1: shared data backref parent 437615230976 count 1 [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 [26062.127274] extent refs 2 gen 2368909 flags 1 [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 [26062.127277] ref#1: shared data backref parent 437615230976 count 1 [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 [26062.127280] extent refs 2 gen 2369146 flags 1 [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 [26062.127283] ref#1: shared data backref parent 437615230976 count 1 [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 [26062.127286] extent refs 1 gen 2369353 flags 1 [26062.127287] ref#0: shared data backref parent 510167498752 count 1 [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 [26062.127294] extent refs 3 gen 1800733 flags 1 [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 [26062.127296] ref#1: shared data backref parent 509783752704 count 1 [26062.127298] ref#2: shared data backref parent 509351313408 count 1 [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 [26062.127303] extent refs 22 gen 1800674 flags 1 [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 [26062.127306] ref#1: shared data backref parent 508152889344 count 2 [26062.127307] ref#2: shared data backref parent 508031139840 count 2 [26062.127308] ref#3: shared data backref parent 493563478016 count 2 [26062.127310] ref#4: shared data backref parent 493539983360 count 2 [26062.127311] ref#5: shared data backref parent 492674990080 count 2 [26062.127312] ref#6: shared data backref parent 475132166144 count 2 [26062.127314] ref#7: shared data backref parent 471032348672 count 2 [26062.127315] ref#8: shared data backref parent 436111769600 count 2 [26062.127317] ref#9: shared data backref parent 883359744 count 2 [26062.127318] ref#10: shared data backref parent 85082112 count 2 [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 [26062.127321] extent refs 33 gen 1800674 flags 1 [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 [26062.127324] ref#1: shared data backref parent 508152889344 count 3 [26062.127325] ref#2: shared data backref parent 508031139840 count 3 [26062.127327] ref#3: shared data backref parent 493563478016 count 3 [26062.127328] ref#4: shared data backref parent 493539983360 count 3 [26062.127329] ref#5: shared data backref parent 492674990080 count 3 [26062.127331] ref#6: shared data backref parent 475132166144 count 3 [26062.127333] ref#7: shared data backref parent 471032348672 count 3 [26062.127334] ref#8: shared data backref parent 436111769600 count 3 [26062.127335] ref#9: shared data backref parent 883359744 count 3 [26062.127337] ref#10: shared data backref parent 85082112 count 3 [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 [26062.127340] extent refs 4 gen 2419085 flags 1 [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 [26062.127342] ref#1: shared data backref parent 494075035648 count 1 [26062.127344] ref#2: shared data backref parent 470850240512 count 1 [26062.127345] ref#3: shared data backref parent 437034745856 count 1 [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 [26062.127348] extent refs 2 gen 2368932 flags 1 [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 [26062.127351] ref#1: shared data backref parent 437615230976 count 1 [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 [26062.127354] extent refs 1 gen 2369353 flags 1 [26062.127355] ref#0: shared data backref parent 510165155840 count 1 [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 [26062.127358] extent refs 2 gen 2369168 flags 1 [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 [26062.127360] ref#1: shared data backref parent 437615230976 count 1 [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 [26062.127364] extent refs 1 gen 2369305 flags 1 [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 [26062.127369] extent refs 4 gen 2419085 flags 1 [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 [26062.127372] ref#1: shared data backref parent 494075035648 count 1 [26062.127374] ref#2: shared data backref parent 470850240512 count 1 [26062.127375] ref#3: shared data backref parent 437034745856 count 1 [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 [26062.127378] extent refs 2 gen 2368955 flags 1 [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 [26062.127382] ref#1: shared data backref parent 437615230976 count 1 [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 [26062.127385] extent refs 1 gen 2369352 flags 1 [26062.127386] ref#0: shared data backref parent 510178295808 count 1 [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 [26062.127390] extent refs 1 gen 2369352 flags 1 [26062.127391] ref#0: shared data backref parent 510058790912 count 1 [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 [26062.127394] extent refs 1 gen 2369352 flags 1 [26062.127395] ref#0: shared data backref parent 510059184128 count 1 [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 [26062.127399] extent refs 1 gen 2378552 flags 1 [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 [26062.127405] extent refs 1 gen 2378552 flags 1 [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 [26062.127409] extent refs 1 gen 2381741 flags 1 [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 [26062.127414] extent refs 2 gen 2368700 flags 1 [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 [26062.127416] ref#1: shared data backref parent 494105804800 count 1 [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 [26062.127420] extent refs 2 gen 2368700 flags 1 [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 [26062.127423] ref#1: shared data backref parent 494105804800 count 1 [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 [26062.127426] extent refs 2 gen 2368700 flags 1 [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 [26062.127430] ref#1: shared data backref parent 494105804800 count 1 [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 [26062.127434] extent refs 11 gen 1800699 flags 1 [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 [26062.127437] ref#1: shared data backref parent 508152987648 count 1 [26062.127439] ref#2: shared data backref parent 508031172608 count 1 [26062.127440] ref#3: shared data backref parent 493563822080 count 1 [26062.127441] ref#4: shared data backref parent 493540130816 count 1 [26062.127443] ref#5: shared data backref parent 492675170304 count 1 [26062.127444] ref#6: shared data backref parent 475884584960 count 1 [26062.127446] ref#7: shared data backref parent 471032496128 count 1 [26062.127447] ref#8: shared data backref parent 436111884288 count 1 [26062.127448] ref#9: shared data backref parent 883703808 count 1 [26062.127450] ref#10: shared data backref parent 85671936 count 1 [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 [26062.127453] extent refs 1 gen 1444854 flags 1 [26062.127454] ref#0: shared data backref parent 214908928 count 1 [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 [26062.127458] extent refs 1 gen 2426255 flags 1 [26062.127459] ref#0: shared data backref parent 436271005696 count 1 [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 [26062.127463] extent refs 22 gen 1800674 flags 1 [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 [26062.127465] ref#1: shared data backref parent 508152987648 count 2 [26062.127467] ref#2: shared data backref parent 508031172608 count 2 [26062.127468] ref#3: shared data backref parent 493563822080 count 2 [26062.127470] ref#4: shared data backref parent 493540130816 count 2 [26062.127472] ref#5: shared data backref parent 492675170304 count 2 [26062.127473] ref#6: shared data backref parent 475884584960 count 2 [26062.127475] ref#7: shared data backref parent 471032496128 count 2 [26062.127476] ref#8: shared data backref parent 436111884288 count 2 [26062.127478] ref#9: shared data backref parent 883703808 count 2 [26062.127480] ref#10: shared data backref parent 85671936 count 2 [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 [26062.127486] extent refs 1 gen 1444854 flags 1 [26062.127488] ref#0: shared data backref parent 167198720 count 1 [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 [26062.127494] extent refs 2 gen 1400856 flags 1 [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 [26062.127498] ref#1: shared data backref parent 470954196992 count 1 [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 [26062.127503] extent refs 1 gen 1444854 flags 1 [26062.127504] ref#0: shared data backref parent 93732864 count 1 [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 [26062.127510] extent refs 1 gen 2381741 flags 1 [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 [26062.127520] extent refs 2 gen 2368701 flags 1 [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 [26062.127523] ref#1: shared data backref parent 494107131904 count 1 [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 [26062.127527] extent refs 44 gen 1800674 flags 1 [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 [26062.127530] ref#1: shared data backref parent 508152987648 count 4 [26062.127532] ref#2: shared data backref parent 508031172608 count 4 [26062.127533] ref#3: shared data backref parent 493563822080 count 4 [26062.127535] ref#4: shared data backref parent 493540130816 count 4 [26062.127537] ref#5: shared data backref parent 492675170304 count 4 [26062.127539] ref#6: shared data backref parent 475884584960 count 4 [26062.127540] ref#7: shared data backref parent 471032496128 count 4 [26062.127542] ref#8: shared data backref parent 436111884288 count 4 [26062.127543] ref#9: shared data backref parent 883703808 count 4 [26062.127545] ref#10: shared data backref parent 85671936 count 4 [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 [26062.127549] extent refs 3 gen 1800750 flags 1 [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 [26062.127554] ref#1: shared data backref parent 509783130112 count 1 [26062.127557] ref#2: shared data backref parent 508544876544 count 1 [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 [26062.127561] extent refs 1 gen 1444854 flags 1 [26062.127562] ref#0: shared data backref parent 93732864 count 1 [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 [26062.127566] extent refs 1 gen 1444854 flags 1 [26062.127568] ref#0: shared data backref parent 214908928 count 1 [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 [26062.127572] extent refs 1 gen 1444854 flags 1 [26062.127573] ref#0: shared data backref parent 93732864 count 1 [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 [26062.127583] extent refs 1 gen 1444854 flags 1 [26062.127584] ref#0: shared data backref parent 167198720 count 1 [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 [26062.127587] extent refs 1 gen 1444854 flags 1 [26062.127588] ref#0: shared data backref parent 167198720 count 1 [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 [26062.127591] extent refs 1 gen 1444854 flags 1 [26062.127592] ref#0: shared data backref parent 167198720 count 1 [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 [26062.127595] extent refs 1 gen 1444854 flags 1 [26062.127596] ref#0: shared data backref parent 167198720 count 1 [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 [26062.127599] extent refs 1 gen 1444854 flags 1 [26062.127600] ref#0: shared data backref parent 167198720 count 1 [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 [26062.127603] extent refs 1 gen 1444854 flags 1 [26062.127604] ref#0: shared data backref parent 167198720 count 1 [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 [26062.127607] extent refs 11 gen 2409230 flags 1 [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 [26062.127610] ref#1: shared data backref parent 508153675776 count 1 [26062.127611] ref#2: shared data backref parent 493566640128 count 1 [26062.127613] ref#3: shared data backref parent 493540999168 count 1 [26062.127614] ref#4: shared data backref parent 492676497408 count 1 [26062.127616] ref#5: shared data backref parent 475884634112 count 1 [26062.127617] ref#6: shared data backref parent 475691106304 count 1 [26062.127619] ref#7: shared data backref parent 471034494976 count 1 [26062.127620] ref#8: shared data backref parent 436113883136 count 1 [26062.127622] ref#9: shared data backref parent 886407168 count 1 [26062.127623] ref#10: shared data backref parent 89440256 count 1 [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 [26062.127626] extent refs 10 gen 2413323 flags 1 [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 [26062.127631] ref#1: shared data backref parent 508153675776 count 1 [26062.127633] ref#2: shared data backref parent 493566640128 count 1 [26062.127635] ref#3: shared data backref parent 493540999168 count 1 [26062.127636] ref#4: shared data backref parent 492676497408 count 1 [26062.127638] ref#5: shared data backref parent 475691106304 count 1 [26062.127640] ref#6: shared data backref parent 471034494976 count 1 [26062.127642] ref#7: shared data backref parent 436113883136 count 1 [26062.127643] ref#8: shared data backref parent 886407168 count 1 [26062.127645] ref#9: shared data backref parent 89440256 count 1 [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 [26062.127648] extent refs 1 gen 2407793 flags 1 [26062.127649] ref#0: shared data backref parent 475884634112 count 1 [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 [26062.127655] extent refs 11 gen 1800729 flags 1 [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 [26062.127658] ref#1: shared data backref parent 508153069568 count 1 [26062.127659] ref#2: shared data backref parent 508031270912 count 1 [26062.127661] ref#3: shared data backref parent 493564215296 count 1 [26062.127663] ref#4: shared data backref parent 493540098048 count 1 [26062.127664] ref#5: shared data backref parent 492676038656 count 1 [26062.127666] ref#6: shared data backref parent 475884142592 count 1 [26062.127668] ref#7: shared data backref parent 471032840192 count 1 [26062.127669] ref#8: shared data backref parent 436112146432 count 1 [26062.127671] ref#9: shared data backref parent 883752960 count 1 [26062.127674] ref#10: shared data backref parent 86654976 count 1 [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 [26062.127680] ------------[ cut here ]------------ [26062.127682] BTRFS: Transaction aborted (error -2) [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_hda _intel [26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [26062.127823] Call Trace: [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 [26062.127837] ? __switch_to_asm+0x41/0x70 [26062.127839] ? __switch_to_asm+0x35/0x70 [26062.127841] ? __switch_to_asm+0x41/0x70 [26062.127848] ? __switch_to+0x13d/0x3d5 [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 [26062.127855] btrfs_commit_transaction+0x55/0x7c8 [26062.127858] ? start_transaction+0x347/0x3cb [26062.127862] transaction_kthread+0xc9/0x135 [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 [26062.127869] kthread+0xeb/0xf0 [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 [26062.127875] ret_from_fork+0x35/0x40 [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry [26062.127917] BTRFS info (device dm-2): forced readonly [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 2019-10-18 2:56 ` 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 Marc MERLIN @ 2019-10-19 3:07 ` Marc MERLIN 2019-10-23 0:55 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2019-10-19 3:07 UTC (permalink / raw) To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue; +Cc: Su Yue Ok, so before blowing the filesystem away after it was apparently badly damaged by a suspend to disk, I tried check --repair and I hit an infinite loop. Let me know if you'd like anything off the FS before I delete it. Thanks, Marc enabling repair mode repair mode will force to clear out log tree, are you sure? [y/N]: y Checking filesystem on /dev/mapper/pool1 UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23 checking extents Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0 Backref 415334400 parent 179634176 root 179634176 not found in extent tree Incorrect global backref count on 415334400 found 2 wanted 1 backpointer mismatch on [415334400 16384] repair deleting extent record: key 415334400 169 0 adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176 Repaired extent references for 415334400 ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1 repair deleting extent record: key 101995261952 168 4096 adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1 Repaired extent references for 101995261952 Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0 Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 backpointer mismatch on [458640384000 86016] repair deleting extent record: key 458640384000 168 86016 adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1 adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1 Repaired extent references for 458640384000 Fixed 0 roots. checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots Deleting bad dir index [10138517,96,436945] root 456 Deleting bad dir index [10138518,96,646273] root 456 Deleting bad dir index [10138517,96,437016] root 456 Deleting bad dir index [10138518,96,808999] root 456 Deleting bad dir index [10215134,96,149427] root 456 Deleting bad dir index [10240541,96,268037] root 456 Deleting bad dir index [10138517,96,540247] root 456 Deleting bad dir index [10138518,96,825234] root 456 Deleting bad dir index [10138517,96,736673] root 456 Deleting bad dir index [10138518,96,1118221] root 456 Deleting bad dir index [10240541,96,439703] root 456 Deleting bad dir index [10138517,96,752282] root 456 root 456 inode 75431563 errors 100, file extent discount Found file extent holes: start: 4096, len: 4096 root 456 inode 75431568 errors 100, file extent discount Found file extent holes: start: 0, len: 1638400 root 456 inode 75431583 errors 100, file extent discount Found file extent holes: start: 0, len: 147456 root 456 inode 75431585 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431591 errors 100, file extent discount Found file extent holes: start: 0, len: 2523136 root 456 inode 75431730 errors 100, file extent discount Found file extent holes: start: 0, len: 208896 root 456 inode 75431744 errors 100, file extent discount Found file extent holes: start: 0, len: 2084864 root 456 inode 75431751 errors 100, file extent discount Found file extent holes: start: 0, len: 172032 root 456 inode 75431756 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 456 inode 75431760 errors 100, file extent discount Found file extent holes: start: 0, len: 12288 root 456 inode 75431765 errors 100, file extent discount Found file extent holes: start: 0, len: 32768 root 456 inode 75431773 errors 100, file extent discount Found file extent holes: start: 0, len: 90112 Fixed discount file extents for inode: 75432421 in root: 456 Fixed discount file extents for inode: 75432429 in root: 456 root 456 inode 75432429 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432430 in root: 456 Fixed discount file extents for inode: 75432432 in root: 456 Fixed discount file extents for inode: 75432433 in root: 456 root 456 inode 75432433 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 root 456 inode 75432434 errors 100, file extent discount Found file extent holes: start: 0, len: 724992 Fixed discount file extents for inode: 75432436 in root: 456 Fixed discount file extents for inode: 75432437 in root: 456 root 456 inode 75432437 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432438 in root: 456 Fixed discount file extents for inode: 75432441 in root: 456 root 456 inode 75432441 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432444 in root: 456 root 456 inode 75432444 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432445 in root: 456 Fixed discount file extents for inode: 75432447 in root: 456 Fixed discount file extents for inode: 75432448 in root: 456 Fixed discount file extents for inode: 75432454 in root: 456 Fixed discount file extents for inode: 75432457 in root: 456 root 456 inode 75432457 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 root 456 inode 75432471 errors 100, file extent discount Found file extent holes: start: 0, len: 20480 Fixed discount file extents for inode: 75432473 in root: 456 root 456 inode 75432473 errors 100, file extent discount Found file extent holes: start: 0, len: 20480 Fixed discount file extents for inode: 75432482 in root: 456 Fixed discount file extents for inode: 75432483 in root: 456 Fixed discount file extents for inode: 75432484 in root: 456 Fixed discount file extents for inode: 75432487 in root: 456 Fixed discount file extents for inode: 75432489 in root: 456 Fixed discount file extents for inode: 75432494 in root: 456 Fixed discount file extents for inode: 75432611 in root: 456 Fixed discount file extents for inode: 75432622 in root: 456 Fixed discount file extents for inode: 75432626 in root: 456 Fixed discount file extents for inode: 75432628 in root: 456 Fixed discount file extents for inode: 75432636 in root: 456 Fixed discount file extents for inode: 75432637 in root: 456 root 456 inode 75432637 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432643 in root: 456 root 456 inode 75432643 errors 100, file extent discount Found file extent holes: start: 0, len: 135168 Fixed discount file extents for inode: 75432650 in root: 456 root 456 inode 75432650 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432656 in root: 456 root 456 inode 75432656 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432657 in root: 456 root 456 inode 75432657 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432659 in root: 456 Fixed discount file extents for inode: 75432662 in root: 456 root 456 inode 75432662 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432663 in root: 456 root 456 inode 75432663 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432664 in root: 456 root 456 inode 75432664 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432674 in root: 456 root 456 inode 75432674 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432692 in root: 456 root 456 inode 75432692 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432708 in root: 456 Fixed discount file extents for inode: 75432709 in root: 456 root 456 inode 75432709 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432712 in root: 456 Fixed discount file extents for inode: 75432717 in root: 456 Fixed discount file extents for inode: 75432720 in root: 456 Fixed discount file extents for inode: 75432725 in root: 456 Fixed discount file extents for inode: 75432731 in root: 456 Fixed discount file extents for inode: 75432732 in root: 456 Fixed discount file extents for inode: 75432749 in root: 456 Fixed discount file extents for inode: 75432762 in root: 456 Fixed discount file extents for inode: 75432765 in root: 456 Fixed discount file extents for inode: 75432783 in root: 456 Fixed discount file extents for inode: 75432784 in root: 456 Fixed discount file extents for inode: 75432785 in root: 456 root 456 inode 75432785 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432786 in root: 456 root 456 inode 75432786 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432796 in root: 456 root 456 inode 75432796 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432799 in root: 456 root 456 inode 75432799 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432801 in root: 456 root 456 inode 75432801 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432807 in root: 456 Fixed discount file extents for inode: 75432817 in root: 456 Fixed discount file extents for inode: 75432829 in root: 456 Fixed discount file extents for inode: 75432860 in root: 456 root 456 inode 75432860 errors 100, file extent discount Found file extent holes: start: 0, len: 4096 Fixed discount file extents for inode: 75432862 in root: 456 Fixed discount file extents for inode: 75432863 in root: 456 Fixed discount file extents for inode: 75432869 in root: 456 Fixed discount file extents for inode: 75432870 in root: 456 Fixed discount file extents for inode: 75432871 in root: 456 Fixed discount file extents for inode: 75432872 in root: 456 Fixed discount file extents for inode: 75432875 in root: 456 Fixed discount file extents for inode: 75432877 in root: 456 Fixed discount file extents for inode: 75432882 in root: 456 Fixed discount file extents for inode: 75432883 in root: 456 Fixed discount file extents for inode: 75432893 in root: 456 Fixed discount file extents for inode: 75432894 in root: 456 Fixed discount file extents for inode: 75432897 in root: 456 Fixed discount file extents for inode: 75432899 in root: 456 Fixed discount file extents for inode: 75432900 in root: 456 Fixed discount file extents for inode: 75432905 in root: 456 Fixed discount file extents for inode: 75432906 in root: 456 Fixed discount file extents for inode: 75432916 in root: 456 Fixed discount file extents for inode: 75432917 in root: 456 Fixed discount file extents for inode: 75432919 in root: 456 Fixed discount file extents for inode: 75432920 in root: 456 Fixed discount file extents for inode: 75432923 in root: 456 Fixed discount file extents for inode: 75432942 in root: 456 Fixed discount file extents for inode: 75432944 in root: 456 Fixed discount file extents for inode: 75432948 in root: 456 root 456 inode 75432948 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 Fixed discount file extents for inode: 75432949 in root: 456 root 456 inode 75432949 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 and it loops forever on 456 On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote: > This happened almost after a resume from suspend to disk. > First corruption and read only I got a very long time. > > Could they be related? > > [26062.126505] ------------[ cut here ]------------ > [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 > [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h da_intel > [26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 > [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 > [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 > [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 > [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 > [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 > [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [26062.126636] Call Trace: > [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 > [26062.126653] ? __switch_to_asm+0x41/0x70 > [26062.126655] ? __switch_to_asm+0x35/0x70 > [26062.126658] ? __switch_to_asm+0x41/0x70 > [26062.126662] ? __switch_to+0x13d/0x3d5 > [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 > [26062.126672] btrfs_commit_transaction+0x55/0x7c8 > [26062.126676] ? start_transaction+0x347/0x3cb > [26062.126679] transaction_kthread+0xc9/0x135 > [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 > [26062.126688] kthread+0xeb/0xf0 > [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 > [26062.126695] ret_from_fork+0x35/0x40 > [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- > [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 > [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 > [26062.126708] extent refs 2 gen 2369265 flags 1 > [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 > [26062.126711] ref#1: shared data backref parent 437615230976 count 1 > [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 > [26062.126715] extent refs 2 gen 2369025 flags 1 > [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 > [26062.126718] ref#1: shared data backref parent 437615230976 count 1 > [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 > [26062.126722] extent refs 11 gen 1800715 flags 1 > [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 > [26062.126724] ref#1: shared data backref parent 508153839616 count 1 > [26062.126726] ref#2: shared data backref parent 493541244928 count 1 > [26062.126727] ref#3: shared data backref parent 492677332992 count 1 > [26062.126728] ref#4: shared data backref parent 492328566784 count 1 > [26062.126730] ref#5: shared data backref parent 492245188608 count 1 > [26062.126731] ref#6: shared data backref parent 475885109248 count 1 > [26062.126732] ref#7: shared data backref parent 471036608512 count 1 > [26062.126733] ref#8: shared data backref parent 436114243584 count 1 > [26062.126735] ref#9: shared data backref parent 886718464 count 1 > [26062.126736] ref#10: shared data backref parent 90259456 count 1 > [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 > [26062.126740] extent refs 2 gen 2420471 flags 1 > [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 > [26062.126743] ref#1: shared data backref parent 41020112896 count 1 > [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 > [26062.126746] extent refs 1 gen 2422941 flags 1 > [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 > [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 > [26062.126750] extent refs 1 gen 1444854 flags 1 > [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 > [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 > [26062.126754] extent refs 1 gen 1444854 flags 1 > [26062.126755] ref#0: shared data backref parent 214892544 count 1 > [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 > [26062.126758] extent refs 1 gen 1444854 flags 1 > [26062.126759] ref#0: shared data backref parent 102547456 count 1 > [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 > [26062.126762] extent refs 2 gen 2422644 flags 1 > [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 > [26062.126765] ref#1: shared data backref parent 470658482176 count 1 > [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 > [26062.126769] extent refs 2 gen 2369217 flags 1 > [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 > [26062.126772] ref#1: shared data backref parent 437615230976 count 1 > [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 > [26062.126775] extent refs 2 gen 2369242 flags 1 > [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 > [26062.126778] ref#1: shared data backref parent 437615230976 count 1 > [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 > [26062.126781] extent refs 1 gen 2369305 flags 1 > [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 > [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 > [26062.126786] extent refs 2 gen 2369052 flags 1 > [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 > [26062.126788] ref#1: shared data backref parent 437615230976 count 1 > [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 > [26062.126792] extent refs 1 gen 2369305 flags 1 > [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 > [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 > [26062.126796] extent refs 2 gen 2368754 flags 1 > [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 > [26062.126799] ref#1: shared data backref parent 437615230976 count 1 > [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 > [26062.126802] extent refs 11 gen 1800682 flags 1 > [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 > [26062.126805] ref#1: shared data backref parent 508153069568 count 1 > [26062.126806] ref#2: shared data backref parent 508031270912 count 1 > [26062.126808] ref#3: shared data backref parent 493564215296 count 1 > [26062.126809] ref#4: shared data backref parent 493540098048 count 1 > [26062.126811] ref#5: shared data backref parent 492676038656 count 1 > [26062.126813] ref#6: shared data backref parent 475884142592 count 1 > [26062.126814] ref#7: shared data backref parent 471032840192 count 1 > [26062.126816] ref#8: shared data backref parent 436112146432 count 1 > [26062.126817] ref#9: shared data backref parent 883752960 count 1 > [26062.126818] ref#10: shared data backref parent 86654976 count 1 > [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 > [26062.126822] extent refs 2 gen 2369347 flags 1 > [26062.126823] ref#0: shared data backref parent 492981370880 count 1 > [26062.126824] ref#1: shared data backref parent 438017720320 count 1 > [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 > [26062.126827] extent refs 2 gen 2369347 flags 1 > [26062.126828] ref#0: shared data backref parent 492981370880 count 1 > [26062.126830] ref#1: shared data backref parent 438017720320 count 1 > [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 > [26062.126833] extent refs 2 gen 2369347 flags 1 > [26062.126834] ref#0: shared data backref parent 492981370880 count 1 > [26062.126836] ref#1: shared data backref parent 438017720320 count 1 > [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 > [26062.126840] extent refs 1 gen 2369352 flags 1 > [26062.126841] ref#0: shared data backref parent 510058790912 count 1 > [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 > [26062.126845] extent refs 1 gen 2369352 flags 1 > [26062.126845] ref#0: shared data backref parent 510058790912 count 1 > [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 > [26062.126849] extent refs 1 gen 2369352 flags 1 > [26062.126850] ref#0: shared data backref parent 510058790912 count 1 > [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 > [26062.126853] extent refs 2 gen 2369347 flags 1 > [26062.126854] ref#0: shared data backref parent 492981370880 count 1 > [26062.126856] ref#1: shared data backref parent 438017720320 count 1 > [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 > [26062.126859] extent refs 2 gen 2369347 flags 1 > [26062.126860] ref#0: shared data backref parent 492981370880 count 1 > [26062.126862] ref#1: shared data backref parent 438017720320 count 1 > [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 > [26062.126865] extent refs 2 gen 2369347 flags 1 > [26062.126866] ref#0: shared data backref parent 492981370880 count 1 > [26062.126868] ref#1: shared data backref parent 438017720320 count 1 > [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 > [26062.126871] extent refs 2 gen 2369347 flags 1 > [26062.126873] ref#0: shared data backref parent 492981370880 count 1 > [26062.126874] ref#1: shared data backref parent 438017720320 count 1 > [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 > [26062.126877] extent refs 2 gen 2369347 flags 1 > [26062.126878] ref#0: shared data backref parent 492981370880 count 1 > [26062.126880] ref#1: shared data backref parent 438017720320 count 1 > [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 > [26062.126883] extent refs 2 gen 2369347 flags 1 > [26062.126884] ref#0: shared data backref parent 492981370880 count 1 > [26062.126885] ref#1: shared data backref parent 438017720320 count 1 > [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 > [26062.126889] extent refs 2 gen 2369347 flags 1 > [26062.126890] ref#0: shared data backref parent 492981370880 count 1 > [26062.126891] ref#1: shared data backref parent 438017720320 count 1 > [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 > [26062.126894] extent refs 3 gen 2369347 flags 1 > [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 > [26062.126897] ref#1: shared data backref parent 492981370880 count 1 > [26062.126898] ref#2: shared data backref parent 438017720320 count 1 > [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 > [26062.126902] extent refs 2 gen 2369347 flags 1 > [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 > [26062.126904] ref#1: shared data backref parent 438017720320 count 1 > [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 > [26062.126907] extent refs 1 gen 2369352 flags 1 > [26062.126908] ref#0: shared data backref parent 510058790912 count 1 > [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 > [26062.126912] extent refs 6 gen 1800750 flags 1 > [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 > [26062.126915] ref#1: shared data backref parent 509782933504 count 2 > [26062.126917] ref#2: shared data backref parent 508543795200 count 2 > [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 > [26062.126920] extent refs 2 gen 2369075 flags 1 > [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 > [26062.126923] ref#1: shared data backref parent 437615230976 count 1 > [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 > [26062.126926] extent refs 3 gen 1800733 flags 1 > [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 > [26062.126929] ref#1: shared data backref parent 509783719936 count 1 > [26062.126930] ref#2: shared data backref parent 509351231488 count 1 > [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 > [26062.126934] extent refs 2 gen 2420471 flags 1 > [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 > [26062.126936] ref#1: shared data backref parent 41020112896 count 1 > [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 > [26062.126939] extent refs 2 gen 2420471 flags 1 > [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 > [26062.126943] ref#1: shared data backref parent 41020112896 count 1 > [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 > [26062.126946] extent refs 3 gen 1800733 flags 1 > [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 > [26062.126949] ref#1: shared data backref parent 509783719936 count 1 > [26062.126950] ref#2: shared data backref parent 509351231488 count 1 > [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 > [26062.126954] extent refs 2 gen 2420502 flags 1 > [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 > [26062.126957] ref#1: shared data backref parent 492274728960 count 1 > [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 > [26062.126961] extent refs 2 gen 2420478 flags 1 > [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 > [26062.126964] ref#1: shared data backref parent 41020112896 count 1 > [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 > [26062.126967] extent refs 1 gen 1444854 flags 1 > [26062.126968] ref#0: shared data backref parent 214908928 count 1 > [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 > [26062.126971] extent refs 1 gen 2376479 flags 1 > [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 > [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 > [26062.126976] extent refs 1 gen 1444854 flags 1 > [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 > [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 > [26062.126981] extent refs 1 gen 1444854 flags 1 > [26062.126982] ref#0: shared data backref parent 93732864 count 1 > [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 > [26062.126985] extent refs 1 gen 1444854 flags 1 > [26062.126986] ref#0: shared data backref parent 93732864 count 1 > [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 > [26062.126990] extent refs 1 gen 1444854 flags 1 > [26062.126991] ref#0: shared data backref parent 93732864 count 1 > [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 > [26062.126994] extent refs 1 gen 1444854 flags 1 > [26062.126995] ref#0: shared data backref parent 214908928 count 1 > [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 > [26062.126998] extent refs 4 gen 2419085 flags 1 > [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 > [26062.127001] ref#1: shared data backref parent 494075035648 count 1 > [26062.127002] ref#2: shared data backref parent 470850240512 count 1 > [26062.127004] ref#3: shared data backref parent 437034745856 count 1 > [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 > [26062.127008] extent refs 3 gen 1800733 flags 1 > [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 > [26062.127011] ref#1: shared data backref parent 509783719936 count 1 > [26062.127013] ref#2: shared data backref parent 509351231488 count 1 > [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 > [26062.127016] extent refs 2 gen 2368859 flags 1 > [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 > [26062.127019] ref#1: shared data backref parent 437615230976 count 1 > [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 > [26062.127023] extent refs 1 gen 2376479 flags 1 > [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 > [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 > [26062.127028] extent refs 11 gen 1800674 flags 1 > [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 > [26062.127031] ref#1: shared data backref parent 508153118720 count 1 > [26062.127032] ref#2: shared data backref parent 508031533056 count 1 > [26062.127034] ref#3: shared data backref parent 493564477440 count 1 > [26062.127036] ref#4: shared data backref parent 493540179968 count 1 > [26062.127038] ref#5: shared data backref parent 492675366912 count 1 > [26062.127039] ref#6: shared data backref parent 475132116992 count 1 > [26062.127041] ref#7: shared data backref parent 471032922112 count 1 > [26062.127042] ref#8: shared data backref parent 436112244736 count 1 > [26062.127044] ref#9: shared data backref parent 883998720 count 1 > [26062.127046] ref#10: shared data backref parent 87015424 count 1 > [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 > [26062.127050] extent refs 1 gen 2408131 flags 1 > [26062.127052] ref#0: shared data backref parent 509278633984 count 1 > [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 > [26062.127056] extent refs 11 gen 1800715 flags 1 > [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 > [26062.127059] ref#1: shared data backref parent 508152889344 count 1 > [26062.127061] ref#2: shared data backref parent 508031139840 count 1 > [26062.127062] ref#3: shared data backref parent 493563478016 count 1 > [26062.127065] ref#4: shared data backref parent 493539983360 count 1 > [26062.127066] ref#5: shared data backref parent 492674990080 count 1 > [26062.127068] ref#6: shared data backref parent 475132166144 count 1 > [26062.127070] ref#7: shared data backref parent 471032348672 count 1 > [26062.127071] ref#8: shared data backref parent 436111769600 count 1 > [26062.127073] ref#9: shared data backref parent 883359744 count 1 > [26062.127074] ref#10: shared data backref parent 85082112 count 1 > [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 > [26062.127078] extent refs 4 gen 2419085 flags 1 > [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 > [26062.127085] ref#1: shared data backref parent 494075035648 count 1 > [26062.127088] ref#2: shared data backref parent 470850240512 count 1 > [26062.127089] ref#3: shared data backref parent 437034745856 count 1 > [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 > [26062.127095] extent refs 2 gen 2421893 flags 1 > [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 > [26062.127099] ref#1: shared data backref parent 493703069696 count 1 > [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 > [26062.127104] extent refs 1 gen 1444854 flags 1 > [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 > [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 > [26062.127111] extent refs 2 gen 2368700 flags 1 > [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 > [26062.127114] ref#1: shared data backref parent 494105804800 count 1 > [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 > [26062.127120] extent refs 1 gen 1444854 flags 1 > [26062.127121] ref#0: shared data backref parent 93732864 count 1 > [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 > [26062.127127] extent refs 1 gen 1444854 flags 1 > [26062.127128] ref#0: shared data backref parent 214433792 count 1 > [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 > [26062.127132] extent refs 1 gen 1444854 flags 1 > [26062.127133] ref#0: shared data backref parent 214908928 count 1 > [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 > [26062.127138] extent refs 3 gen 1800750 flags 1 > [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 > [26062.127141] ref#1: shared data backref parent 509783556096 count 1 > [26062.127143] ref#2: shared data backref parent 508544974848 count 1 > [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 > [26062.127148] extent refs 2 gen 2368707 flags 1 > [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 > [26062.127151] ref#1: shared data backref parent 437615230976 count 1 > [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 > [26062.127155] extent refs 11 gen 1800674 flags 1 > [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 > [26062.127157] ref#1: shared data backref parent 508152905728 count 1 > [26062.127159] ref#2: shared data backref parent 508031057920 count 1 > [26062.127160] ref#3: shared data backref parent 493563166720 count 1 > [26062.127161] ref#4: shared data backref parent 493539934208 count 1 > [26062.127163] ref#5: shared data backref parent 492674908160 count 1 > [26062.127164] ref#6: shared data backref parent 475884208128 count 1 > [26062.127165] ref#7: shared data backref parent 471032266752 count 1 > [26062.127167] ref#8: shared data backref parent 436111720448 count 1 > [26062.127168] ref#9: shared data backref parent 883179520 count 1 > [26062.127170] ref#10: shared data backref parent 83017728 count 1 > [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 > [26062.127173] extent refs 1 gen 1444854 flags 1 > [26062.127174] ref#0: shared data backref parent 214908928 count 1 > [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 > [26062.127177] extent refs 2 gen 2381729 flags 1 > [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 > [26062.127180] ref#1: shared data backref parent 437808545792 count 1 > [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 > [26062.127183] extent refs 2 gen 2369098 flags 1 > [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 > [26062.127186] ref#1: shared data backref parent 437615230976 count 1 > [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 > [26062.127189] extent refs 2 gen 2368700 flags 1 > [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 > [26062.127191] ref#1: shared data backref parent 494105804800 count 1 > [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 > [26062.127195] extent refs 1 gen 1444854 flags 1 > [26062.127195] ref#0: shared data backref parent 102547456 count 1 > [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 > [26062.127198] extent refs 3 gen 1800750 flags 1 > [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 > [26062.127201] ref#1: shared data backref parent 509783408640 count 1 > [26062.127202] ref#2: shared data backref parent 508543451136 count 1 > [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 > [26062.127205] extent refs 1 gen 1444854 flags 1 > [26062.127206] ref#0: shared data backref parent 93732864 count 1 > [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 > [26062.127209] extent refs 1 gen 1444854 flags 1 > [26062.127210] ref#0: shared data backref parent 93732864 count 1 > [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 > [26062.127214] extent refs 1 gen 1444854 flags 1 > [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 > [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 > [26062.127219] extent refs 4 gen 2419085 flags 1 > [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 > [26062.127222] ref#1: shared data backref parent 494075035648 count 1 > [26062.127223] ref#2: shared data backref parent 470850240512 count 1 > [26062.127225] ref#3: shared data backref parent 437034745856 count 1 > [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 > [26062.127228] extent refs 3 gen 1800750 flags 1 > [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 > [26062.127230] ref#1: shared data backref parent 509783408640 count 1 > [26062.127232] ref#2: shared data backref parent 508543451136 count 1 > [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 > [26062.127235] extent refs 2 gen 2368731 flags 1 > [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 > [26062.127237] ref#1: shared data backref parent 437615230976 count 1 > [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 > [26062.127241] extent refs 1 gen 1444854 flags 1 > [26062.127241] ref#0: shared data backref parent 93732864 count 1 > [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 > [26062.127244] extent refs 1 gen 1444854 flags 1 > [26062.127245] ref#0: shared data backref parent 93732864 count 1 > [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 > [26062.127249] extent refs 1 gen 1444854 flags 1 > [26062.127249] ref#0: shared data backref parent 93732864 count 1 > [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 > [26062.127253] extent refs 1 gen 1444854 flags 1 > [26062.127254] ref#0: shared data backref parent 93732864 count 1 > [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 > [26062.127258] extent refs 1 gen 1444854 flags 1 > [26062.127259] ref#0: shared data backref parent 93732864 count 1 > [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 > [26062.127263] extent refs 1 gen 2381741 flags 1 > [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 > [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 > [26062.127268] extent refs 2 gen 2368882 flags 1 > [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 > [26062.127270] ref#1: shared data backref parent 437615230976 count 1 > [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 > [26062.127274] extent refs 2 gen 2368909 flags 1 > [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 > [26062.127277] ref#1: shared data backref parent 437615230976 count 1 > [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 > [26062.127280] extent refs 2 gen 2369146 flags 1 > [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 > [26062.127283] ref#1: shared data backref parent 437615230976 count 1 > [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 > [26062.127286] extent refs 1 gen 2369353 flags 1 > [26062.127287] ref#0: shared data backref parent 510167498752 count 1 > [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 > [26062.127294] extent refs 3 gen 1800733 flags 1 > [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 > [26062.127296] ref#1: shared data backref parent 509783752704 count 1 > [26062.127298] ref#2: shared data backref parent 509351313408 count 1 > [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 > [26062.127303] extent refs 22 gen 1800674 flags 1 > [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 > [26062.127306] ref#1: shared data backref parent 508152889344 count 2 > [26062.127307] ref#2: shared data backref parent 508031139840 count 2 > [26062.127308] ref#3: shared data backref parent 493563478016 count 2 > [26062.127310] ref#4: shared data backref parent 493539983360 count 2 > [26062.127311] ref#5: shared data backref parent 492674990080 count 2 > [26062.127312] ref#6: shared data backref parent 475132166144 count 2 > [26062.127314] ref#7: shared data backref parent 471032348672 count 2 > [26062.127315] ref#8: shared data backref parent 436111769600 count 2 > [26062.127317] ref#9: shared data backref parent 883359744 count 2 > [26062.127318] ref#10: shared data backref parent 85082112 count 2 > [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 > [26062.127321] extent refs 33 gen 1800674 flags 1 > [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 > [26062.127324] ref#1: shared data backref parent 508152889344 count 3 > [26062.127325] ref#2: shared data backref parent 508031139840 count 3 > [26062.127327] ref#3: shared data backref parent 493563478016 count 3 > [26062.127328] ref#4: shared data backref parent 493539983360 count 3 > [26062.127329] ref#5: shared data backref parent 492674990080 count 3 > [26062.127331] ref#6: shared data backref parent 475132166144 count 3 > [26062.127333] ref#7: shared data backref parent 471032348672 count 3 > [26062.127334] ref#8: shared data backref parent 436111769600 count 3 > [26062.127335] ref#9: shared data backref parent 883359744 count 3 > [26062.127337] ref#10: shared data backref parent 85082112 count 3 > [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 > [26062.127340] extent refs 4 gen 2419085 flags 1 > [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 > [26062.127342] ref#1: shared data backref parent 494075035648 count 1 > [26062.127344] ref#2: shared data backref parent 470850240512 count 1 > [26062.127345] ref#3: shared data backref parent 437034745856 count 1 > [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 > [26062.127348] extent refs 2 gen 2368932 flags 1 > [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 > [26062.127351] ref#1: shared data backref parent 437615230976 count 1 > [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 > [26062.127354] extent refs 1 gen 2369353 flags 1 > [26062.127355] ref#0: shared data backref parent 510165155840 count 1 > [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 > [26062.127358] extent refs 2 gen 2369168 flags 1 > [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 > [26062.127360] ref#1: shared data backref parent 437615230976 count 1 > [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 > [26062.127364] extent refs 1 gen 2369305 flags 1 > [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 > [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 > [26062.127369] extent refs 4 gen 2419085 flags 1 > [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 > [26062.127372] ref#1: shared data backref parent 494075035648 count 1 > [26062.127374] ref#2: shared data backref parent 470850240512 count 1 > [26062.127375] ref#3: shared data backref parent 437034745856 count 1 > [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 > [26062.127378] extent refs 2 gen 2368955 flags 1 > [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 > [26062.127382] ref#1: shared data backref parent 437615230976 count 1 > [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 > [26062.127385] extent refs 1 gen 2369352 flags 1 > [26062.127386] ref#0: shared data backref parent 510178295808 count 1 > [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 > [26062.127390] extent refs 1 gen 2369352 flags 1 > [26062.127391] ref#0: shared data backref parent 510058790912 count 1 > [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 > [26062.127394] extent refs 1 gen 2369352 flags 1 > [26062.127395] ref#0: shared data backref parent 510059184128 count 1 > [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 > [26062.127399] extent refs 1 gen 2378552 flags 1 > [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 > [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 > [26062.127405] extent refs 1 gen 2378552 flags 1 > [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 > [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 > [26062.127409] extent refs 1 gen 2381741 flags 1 > [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 > [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 > [26062.127414] extent refs 2 gen 2368700 flags 1 > [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 > [26062.127416] ref#1: shared data backref parent 494105804800 count 1 > [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 > [26062.127420] extent refs 2 gen 2368700 flags 1 > [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 > [26062.127423] ref#1: shared data backref parent 494105804800 count 1 > [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 > [26062.127426] extent refs 2 gen 2368700 flags 1 > [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 > [26062.127430] ref#1: shared data backref parent 494105804800 count 1 > [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 > [26062.127434] extent refs 11 gen 1800699 flags 1 > [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 > [26062.127437] ref#1: shared data backref parent 508152987648 count 1 > [26062.127439] ref#2: shared data backref parent 508031172608 count 1 > [26062.127440] ref#3: shared data backref parent 493563822080 count 1 > [26062.127441] ref#4: shared data backref parent 493540130816 count 1 > [26062.127443] ref#5: shared data backref parent 492675170304 count 1 > [26062.127444] ref#6: shared data backref parent 475884584960 count 1 > [26062.127446] ref#7: shared data backref parent 471032496128 count 1 > [26062.127447] ref#8: shared data backref parent 436111884288 count 1 > [26062.127448] ref#9: shared data backref parent 883703808 count 1 > [26062.127450] ref#10: shared data backref parent 85671936 count 1 > [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 > [26062.127453] extent refs 1 gen 1444854 flags 1 > [26062.127454] ref#0: shared data backref parent 214908928 count 1 > [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 > [26062.127458] extent refs 1 gen 2426255 flags 1 > [26062.127459] ref#0: shared data backref parent 436271005696 count 1 > [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 > [26062.127463] extent refs 22 gen 1800674 flags 1 > [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 > [26062.127465] ref#1: shared data backref parent 508152987648 count 2 > [26062.127467] ref#2: shared data backref parent 508031172608 count 2 > [26062.127468] ref#3: shared data backref parent 493563822080 count 2 > [26062.127470] ref#4: shared data backref parent 493540130816 count 2 > [26062.127472] ref#5: shared data backref parent 492675170304 count 2 > [26062.127473] ref#6: shared data backref parent 475884584960 count 2 > [26062.127475] ref#7: shared data backref parent 471032496128 count 2 > [26062.127476] ref#8: shared data backref parent 436111884288 count 2 > [26062.127478] ref#9: shared data backref parent 883703808 count 2 > [26062.127480] ref#10: shared data backref parent 85671936 count 2 > [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 > [26062.127486] extent refs 1 gen 1444854 flags 1 > [26062.127488] ref#0: shared data backref parent 167198720 count 1 > [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 > [26062.127494] extent refs 2 gen 1400856 flags 1 > [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 > [26062.127498] ref#1: shared data backref parent 470954196992 count 1 > [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 > [26062.127503] extent refs 1 gen 1444854 flags 1 > [26062.127504] ref#0: shared data backref parent 93732864 count 1 > [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 > [26062.127510] extent refs 1 gen 2381741 flags 1 > [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 > [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 > [26062.127520] extent refs 2 gen 2368701 flags 1 > [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 > [26062.127523] ref#1: shared data backref parent 494107131904 count 1 > [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 > [26062.127527] extent refs 44 gen 1800674 flags 1 > [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 > [26062.127530] ref#1: shared data backref parent 508152987648 count 4 > [26062.127532] ref#2: shared data backref parent 508031172608 count 4 > [26062.127533] ref#3: shared data backref parent 493563822080 count 4 > [26062.127535] ref#4: shared data backref parent 493540130816 count 4 > [26062.127537] ref#5: shared data backref parent 492675170304 count 4 > [26062.127539] ref#6: shared data backref parent 475884584960 count 4 > [26062.127540] ref#7: shared data backref parent 471032496128 count 4 > [26062.127542] ref#8: shared data backref parent 436111884288 count 4 > [26062.127543] ref#9: shared data backref parent 883703808 count 4 > [26062.127545] ref#10: shared data backref parent 85671936 count 4 > [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 > [26062.127549] extent refs 3 gen 1800750 flags 1 > [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 > [26062.127554] ref#1: shared data backref parent 509783130112 count 1 > [26062.127557] ref#2: shared data backref parent 508544876544 count 1 > [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 > [26062.127561] extent refs 1 gen 1444854 flags 1 > [26062.127562] ref#0: shared data backref parent 93732864 count 1 > [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 > [26062.127566] extent refs 1 gen 1444854 flags 1 > [26062.127568] ref#0: shared data backref parent 214908928 count 1 > [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 > [26062.127572] extent refs 1 gen 1444854 flags 1 > [26062.127573] ref#0: shared data backref parent 93732864 count 1 > [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 > [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 > [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 > [26062.127583] extent refs 1 gen 1444854 flags 1 > [26062.127584] ref#0: shared data backref parent 167198720 count 1 > [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 > [26062.127587] extent refs 1 gen 1444854 flags 1 > [26062.127588] ref#0: shared data backref parent 167198720 count 1 > [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 > [26062.127591] extent refs 1 gen 1444854 flags 1 > [26062.127592] ref#0: shared data backref parent 167198720 count 1 > [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 > [26062.127595] extent refs 1 gen 1444854 flags 1 > [26062.127596] ref#0: shared data backref parent 167198720 count 1 > [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 > [26062.127599] extent refs 1 gen 1444854 flags 1 > [26062.127600] ref#0: shared data backref parent 167198720 count 1 > [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 > [26062.127603] extent refs 1 gen 1444854 flags 1 > [26062.127604] ref#0: shared data backref parent 167198720 count 1 > [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 > [26062.127607] extent refs 11 gen 2409230 flags 1 > [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 > [26062.127610] ref#1: shared data backref parent 508153675776 count 1 > [26062.127611] ref#2: shared data backref parent 493566640128 count 1 > [26062.127613] ref#3: shared data backref parent 493540999168 count 1 > [26062.127614] ref#4: shared data backref parent 492676497408 count 1 > [26062.127616] ref#5: shared data backref parent 475884634112 count 1 > [26062.127617] ref#6: shared data backref parent 475691106304 count 1 > [26062.127619] ref#7: shared data backref parent 471034494976 count 1 > [26062.127620] ref#8: shared data backref parent 436113883136 count 1 > [26062.127622] ref#9: shared data backref parent 886407168 count 1 > [26062.127623] ref#10: shared data backref parent 89440256 count 1 > [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 > [26062.127626] extent refs 10 gen 2413323 flags 1 > [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 > [26062.127631] ref#1: shared data backref parent 508153675776 count 1 > [26062.127633] ref#2: shared data backref parent 493566640128 count 1 > [26062.127635] ref#3: shared data backref parent 493540999168 count 1 > [26062.127636] ref#4: shared data backref parent 492676497408 count 1 > [26062.127638] ref#5: shared data backref parent 475691106304 count 1 > [26062.127640] ref#6: shared data backref parent 471034494976 count 1 > [26062.127642] ref#7: shared data backref parent 436113883136 count 1 > [26062.127643] ref#8: shared data backref parent 886407168 count 1 > [26062.127645] ref#9: shared data backref parent 89440256 count 1 > [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 > [26062.127648] extent refs 1 gen 2407793 flags 1 > [26062.127649] ref#0: shared data backref parent 475884634112 count 1 > [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 > [26062.127655] extent refs 11 gen 1800729 flags 1 > [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 > [26062.127658] ref#1: shared data backref parent 508153069568 count 1 > [26062.127659] ref#2: shared data backref parent 508031270912 count 1 > [26062.127661] ref#3: shared data backref parent 493564215296 count 1 > [26062.127663] ref#4: shared data backref parent 493540098048 count 1 > [26062.127664] ref#5: shared data backref parent 492676038656 count 1 > [26062.127666] ref#6: shared data backref parent 475884142592 count 1 > [26062.127668] ref#7: shared data backref parent 471032840192 count 1 > [26062.127669] ref#8: shared data backref parent 436112146432 count 1 > [26062.127671] ref#9: shared data backref parent 883752960 count 1 > [26062.127674] ref#10: shared data backref parent 86654976 count 1 > [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 > [26062.127680] ------------[ cut here ]------------ > [26062.127682] BTRFS: Transaction aborted (error -2) > [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 > [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd_h da_intel > [26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 > [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df > [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 > [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 > [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 > [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 > [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [26062.127823] Call Trace: > [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 > [26062.127837] ? __switch_to_asm+0x41/0x70 > [26062.127839] ? __switch_to_asm+0x35/0x70 > [26062.127841] ? __switch_to_asm+0x41/0x70 > [26062.127848] ? __switch_to+0x13d/0x3d5 > [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 > [26062.127855] btrfs_commit_transaction+0x55/0x7c8 > [26062.127858] ? start_transaction+0x347/0x3cb > [26062.127862] transaction_kthread+0xc9/0x135 > [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 > [26062.127869] kthread+0xeb/0xf0 > [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 > [26062.127875] ret_from_fork+0x35/0x40 > [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- > [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry > [26062.127917] BTRFS info (device dm-2): forced readonly > [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry On Tue, Jul 17, 2018 at 05:24:51PM -0700, Marc MERLIN wrote: > On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: > > No OOM triggers? That's a little strange. > > Maybe it's related to how kernel handles memory over-commit? > > Yes, I think you are correct. > > > And for the hang, I think it's related to some memory allocation failure > > and error handler just didn't handle it well, so it's causing deadlock > > for certain page. > > That indeed matches what I'm seeing. > > > ENOMEM handling is pretty common but hardly verified, so it's not that > > strange, but we must locate the problem. > > I seem to be getting deadlocks in the kernel, so I'm hoping that at least > it's checked there, but maybe not? > > > In my system, at least I'm not using btrfs as root fs, and for the > > memory eating program I normally ensure it's eating all the memory + > > swap, so OOM killer is always triggered, maybe that's the cause. > > > > So in your case, maybe it's btrfs not really taking up all memory, thus > > OOM killer not triggered. > > Correct, the swap is not used. > > > Any kernel dmesg about OOM killer triggered? > > Nothing at all. It never gets triggered. > > > > Here is my system when it virtually died: > > > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > See how btrs was taking 29GB in that ps output (that's before it takes > everything and I can't even type ps anymore) > Note that VSZ is almost equal to RSS. Nothing gets swapped. > > Then see free output: > > > > total used free shared buffers cached > > > Mem: 32643788 32180100 463688 0 44664 119508 > > > -/+ buffers/cache: 32015928 627860 > > > Swap: 15616764 443676 15173088 > > > > For swap, it looks like only some other program's memory is swapped out, > > not btrfs'. > > That's exactly correct. btrfs check never goes to swap, I'm not sure why, > and because there is virtual memory free, maybe that's why OOM does not > trigger? > So I guess I can probably "fix" my problem by removing swap, but ultimately > it would be useful to know why memory taken by btrfs check does not end up > in swap. > > > And unfortunately, I'm not so familiar with OOM/MM code outside of > > filesystem. > > Any help from other experienced developers would definitely help to > > solve why memory of 'btrfs check' is not swapped out or why OOM killer > > is not triggered. > > Do you have someone from linux-vm you might be able to ask, or should we Cc > this thread there? > > Thanks, > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ On Tue, Jul 17, 2018 at 01:59:05PM -0700, Marc MERLIN wrote: > Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. > It just takes all the memory until linux hangs everywhere, and somehow (no idea why) > the OOM killer never triggers. > Details below: > > On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: > > Here is what I got when the system was not doing well (it took minutes to run): > > > > total used free shared buffers cached > > Mem: 32643788 32070952 572836 0 102160 4378772 > > -/+ buffers/cache: 27590020 5053768 > > Swap: 15616764 973596 14643168 > > ok, the reason it was not that close to 0 was due to /dev/shm it seems. > I cleared that, and now I can get it to go to near 0 again. > I'm wrong about the system being fully crashed, it's not, it's just very > close to being hung. > I can type killall -9 btrfs in the serial console and wait a few minutes. > The system eventually recovers, but it's impossible to fix anything via ssh > apparently because networking does not get to run when I'm in this state. > > I'm not sure why my system reproduces this easy while Qu's system does not, > but Qu was right that the kernel is not dead and that it's merely a problem of userspace > taking all the RAM and somehow not being killed by OOM > > I checked the PID and don't see why it's not being killed: > gargamel:/proc/31006# grep . oom* > oom_adj:0 > oom_score:221 << this increases a lot, but OOM never kills it > oom_score_adj:0 > > I have these variables: > /proc/sys/vm/oom_dump_tasks:1 > /proc/sys/vm/oom_kill_allocating_task:0 > /proc/sys/vm/overcommit_kbytes:0 > /proc/sys/vm/overcommit_memory:0 > /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) > > Here is my system when it virtually died: > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > total used free shared buffers cached > Mem: 32643788 32180100 463688 0 44664 119508 > -/+ buffers/cache: 32015928 627860 > Swap: 15616764 443676 15173088 > > MemTotal: 32643788 kB > MemFree: 463440 kB > MemAvailable: 44864 kB > Buffers: 44664 kB > Cached: 120360 kB > SwapCached: 87064 kB > Active: 30381404 kB > Inactive: 585952 kB > Active(anon): 30334696 kB > Inactive(anon): 474624 kB > Active(file): 46708 kB > Inactive(file): 111328 kB > Unevictable: 5616 kB > Mlocked: 5616 kB > SwapTotal: 15616764 kB > SwapFree: 15173088 kB > Dirty: 1636 kB > Writeback: 4 kB > AnonPages: 30734240 kB > Mapped: 67236 kB > Shmem: 3036 kB > Slab: 267884 kB > SReclaimable: 51528 kB > SUnreclaim: 216356 kB > KernelStack: 10144 kB > PageTables: 69284 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 31938656 kB > Committed_AS: 32865492 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > HardwareCorrupted: 0 kB > AnonHugePages: 0 kB > ShmemHugePages: 0 kB > ShmemPmdMapped: 0 kB > CmaTotal: 16384 kB > CmaFree: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > Hugetlb: 0 kB > DirectMap4k: 560404 kB > DirectMap2M: 32692224 kB > > > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 2019-10-19 3:07 ` Marc MERLIN @ 2019-10-23 0:55 ` Marc MERLIN 2019-10-26 3:36 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2019-10-23 0:55 UTC (permalink / raw) To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue On Fri, Oct 18, 2019 at 08:07:28PM -0700, Marc MERLIN wrote: > Ok, so before blowing the filesystem away after it was apparently badly > damaged by a suspend to disk, I tried check --repair and I hit an > infinite loop. > > Let me know if you'd like anything off the FS before I delete it. I heard nothing back, so I deleted the FS and restored from backup. But now I'm scared of ever doing a suspend to disk again. Could someone please look at the logs and give me some idea of what happened, if at all possible? Non recoverable data corruption on my laptop when I travel and backups/restores are complicated, is a bit unnerving... Thanks, Marc > Thanks, > Marc > > enabling repair mode > repair mode will force to clear out log tree, are you sure? [y/N]: y > Checking filesystem on /dev/mapper/pool1 > UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23 > checking extents > Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0 > Backref 415334400 parent 179634176 root 179634176 not found in extent tree > Incorrect global backref count on 415334400 found 2 wanted 1 > backpointer mismatch on [415334400 16384] > repair deleting extent record: key 415334400 169 0 > adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176 > Repaired extent references for 415334400 > ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1 > repair deleting extent record: key 101995261952 168 4096 > adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1 > Repaired extent references for 101995261952 > Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 > Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0 > Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree > Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 > backpointer mismatch on [458640384000 86016] > repair deleting extent record: key 458640384000 168 86016 > adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1 > adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1 > Repaired extent references for 458640384000 > Fixed 0 roots. > checking free space cache > cache and super generation don't match, space cache will be invalidated > checking fs roots > Deleting bad dir index [10138517,96,436945] root 456 > Deleting bad dir index [10138518,96,646273] root 456 > Deleting bad dir index [10138517,96,437016] root 456 > Deleting bad dir index [10138518,96,808999] root 456 > Deleting bad dir index [10215134,96,149427] root 456 > Deleting bad dir index [10240541,96,268037] root 456 > Deleting bad dir index [10138517,96,540247] root 456 > Deleting bad dir index [10138518,96,825234] root 456 > Deleting bad dir index [10138517,96,736673] root 456 > Deleting bad dir index [10138518,96,1118221] root 456 > Deleting bad dir index [10240541,96,439703] root 456 > Deleting bad dir index [10138517,96,752282] root 456 > root 456 inode 75431563 errors 100, file extent discount > Found file extent holes: > start: 4096, len: 4096 > root 456 inode 75431568 errors 100, file extent discount > Found file extent holes: > start: 0, len: 1638400 > root 456 inode 75431583 errors 100, file extent discount > Found file extent holes: > start: 0, len: 147456 > root 456 inode 75431585 errors 100, file extent discount > Found file extent holes: > start: 0, len: 208896 > root 456 inode 75431591 errors 100, file extent discount > Found file extent holes: > start: 0, len: 2523136 > root 456 inode 75431730 errors 100, file extent discount > Found file extent holes: > start: 0, len: 208896 > root 456 inode 75431744 errors 100, file extent discount > Found file extent holes: > start: 0, len: 2084864 > root 456 inode 75431751 errors 100, file extent discount > Found file extent holes: > start: 0, len: 172032 > root 456 inode 75431756 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 456 inode 75431760 errors 100, file extent discount > Found file extent holes: > start: 0, len: 12288 > root 456 inode 75431765 errors 100, file extent discount > Found file extent holes: > start: 0, len: 32768 > root 456 inode 75431773 errors 100, file extent discount > Found file extent holes: > start: 0, len: 90112 > Fixed discount file extents for inode: 75432421 in root: 456 > Fixed discount file extents for inode: 75432429 in root: 456 > root 456 inode 75432429 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432430 in root: 456 > Fixed discount file extents for inode: 75432432 in root: 456 > Fixed discount file extents for inode: 75432433 in root: 456 > root 456 inode 75432433 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > root 456 inode 75432434 errors 100, file extent discount > Found file extent holes: > start: 0, len: 724992 > Fixed discount file extents for inode: 75432436 in root: 456 > Fixed discount file extents for inode: 75432437 in root: 456 > root 456 inode 75432437 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432438 in root: 456 > Fixed discount file extents for inode: 75432441 in root: 456 > root 456 inode 75432441 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432444 in root: 456 > root 456 inode 75432444 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432445 in root: 456 > Fixed discount file extents for inode: 75432447 in root: 456 > Fixed discount file extents for inode: 75432448 in root: 456 > Fixed discount file extents for inode: 75432454 in root: 456 > Fixed discount file extents for inode: 75432457 in root: 456 > root 456 inode 75432457 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > root 456 inode 75432471 errors 100, file extent discount > Found file extent holes: > start: 0, len: 20480 > Fixed discount file extents for inode: 75432473 in root: 456 > root 456 inode 75432473 errors 100, file extent discount > Found file extent holes: > start: 0, len: 20480 > Fixed discount file extents for inode: 75432482 in root: 456 > Fixed discount file extents for inode: 75432483 in root: 456 > Fixed discount file extents for inode: 75432484 in root: 456 > Fixed discount file extents for inode: 75432487 in root: 456 > Fixed discount file extents for inode: 75432489 in root: 456 > Fixed discount file extents for inode: 75432494 in root: 456 > Fixed discount file extents for inode: 75432611 in root: 456 > Fixed discount file extents for inode: 75432622 in root: 456 > Fixed discount file extents for inode: 75432626 in root: 456 > Fixed discount file extents for inode: 75432628 in root: 456 > Fixed discount file extents for inode: 75432636 in root: 456 > Fixed discount file extents for inode: 75432637 in root: 456 > root 456 inode 75432637 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432643 in root: 456 > root 456 inode 75432643 errors 100, file extent discount > Found file extent holes: > start: 0, len: 135168 > Fixed discount file extents for inode: 75432650 in root: 456 > root 456 inode 75432650 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432656 in root: 456 > root 456 inode 75432656 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432657 in root: 456 > root 456 inode 75432657 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432659 in root: 456 > Fixed discount file extents for inode: 75432662 in root: 456 > root 456 inode 75432662 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432663 in root: 456 > root 456 inode 75432663 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432664 in root: 456 > root 456 inode 75432664 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432674 in root: 456 > root 456 inode 75432674 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432692 in root: 456 > root 456 inode 75432692 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432708 in root: 456 > Fixed discount file extents for inode: 75432709 in root: 456 > root 456 inode 75432709 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432712 in root: 456 > Fixed discount file extents for inode: 75432717 in root: 456 > Fixed discount file extents for inode: 75432720 in root: 456 > Fixed discount file extents for inode: 75432725 in root: 456 > Fixed discount file extents for inode: 75432731 in root: 456 > Fixed discount file extents for inode: 75432732 in root: 456 > Fixed discount file extents for inode: 75432749 in root: 456 > Fixed discount file extents for inode: 75432762 in root: 456 > Fixed discount file extents for inode: 75432765 in root: 456 > Fixed discount file extents for inode: 75432783 in root: 456 > Fixed discount file extents for inode: 75432784 in root: 456 > Fixed discount file extents for inode: 75432785 in root: 456 > root 456 inode 75432785 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432786 in root: 456 > root 456 inode 75432786 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432796 in root: 456 > root 456 inode 75432796 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432799 in root: 456 > root 456 inode 75432799 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432801 in root: 456 > root 456 inode 75432801 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432807 in root: 456 > Fixed discount file extents for inode: 75432817 in root: 456 > Fixed discount file extents for inode: 75432829 in root: 456 > Fixed discount file extents for inode: 75432860 in root: 456 > root 456 inode 75432860 errors 100, file extent discount > Found file extent holes: > start: 0, len: 4096 > Fixed discount file extents for inode: 75432862 in root: 456 > Fixed discount file extents for inode: 75432863 in root: 456 > Fixed discount file extents for inode: 75432869 in root: 456 > Fixed discount file extents for inode: 75432870 in root: 456 > Fixed discount file extents for inode: 75432871 in root: 456 > Fixed discount file extents for inode: 75432872 in root: 456 > Fixed discount file extents for inode: 75432875 in root: 456 > Fixed discount file extents for inode: 75432877 in root: 456 > Fixed discount file extents for inode: 75432882 in root: 456 > Fixed discount file extents for inode: 75432883 in root: 456 > Fixed discount file extents for inode: 75432893 in root: 456 > Fixed discount file extents for inode: 75432894 in root: 456 > Fixed discount file extents for inode: 75432897 in root: 456 > Fixed discount file extents for inode: 75432899 in root: 456 > Fixed discount file extents for inode: 75432900 in root: 456 > Fixed discount file extents for inode: 75432905 in root: 456 > Fixed discount file extents for inode: 75432906 in root: 456 > Fixed discount file extents for inode: 75432916 in root: 456 > Fixed discount file extents for inode: 75432917 in root: 456 > Fixed discount file extents for inode: 75432919 in root: 456 > Fixed discount file extents for inode: 75432920 in root: 456 > Fixed discount file extents for inode: 75432923 in root: 456 > Fixed discount file extents for inode: 75432942 in root: 456 > Fixed discount file extents for inode: 75432944 in root: 456 > Fixed discount file extents for inode: 75432948 in root: 456 > root 456 inode 75432948 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > Fixed discount file extents for inode: 75432949 in root: 456 > root 456 inode 75432949 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > and it loops forever on 456 > > On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote: > > This happened almost after a resume from suspend to disk. > > First corruption and read only I got a very long time. > > > > Could they be related? > > > > [26062.126505] ------------[ cut here ]------------ > > [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 > > [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd _hda_intel > > [26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > > [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > > [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > > [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 > > [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 > > [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 > > [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 > > [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 > > [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > > [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 > > [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > > [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > > [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > > [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > [26062.126636] Call Trace: > > [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 > > [26062.126653] ? __switch_to_asm+0x41/0x70 > > [26062.126655] ? __switch_to_asm+0x35/0x70 > > [26062.126658] ? __switch_to_asm+0x41/0x70 > > [26062.126662] ? __switch_to+0x13d/0x3d5 > > [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 > > [26062.126672] btrfs_commit_transaction+0x55/0x7c8 > > [26062.126676] ? start_transaction+0x347/0x3cb > > [26062.126679] transaction_kthread+0xc9/0x135 > > [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 > > [26062.126688] kthread+0xeb/0xf0 > > [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 > > [26062.126695] ret_from_fork+0x35/0x40 > > [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- > > [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 > > [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 > > [26062.126708] extent refs 2 gen 2369265 flags 1 > > [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 > > [26062.126711] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 > > [26062.126715] extent refs 2 gen 2369025 flags 1 > > [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 > > [26062.126718] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 > > [26062.126722] extent refs 11 gen 1800715 flags 1 > > [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 > > [26062.126724] ref#1: shared data backref parent 508153839616 count 1 > > [26062.126726] ref#2: shared data backref parent 493541244928 count 1 > > [26062.126727] ref#3: shared data backref parent 492677332992 count 1 > > [26062.126728] ref#4: shared data backref parent 492328566784 count 1 > > [26062.126730] ref#5: shared data backref parent 492245188608 count 1 > > [26062.126731] ref#6: shared data backref parent 475885109248 count 1 > > [26062.126732] ref#7: shared data backref parent 471036608512 count 1 > > [26062.126733] ref#8: shared data backref parent 436114243584 count 1 > > [26062.126735] ref#9: shared data backref parent 886718464 count 1 > > [26062.126736] ref#10: shared data backref parent 90259456 count 1 > > [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 > > [26062.126740] extent refs 2 gen 2420471 flags 1 > > [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 > > [26062.126743] ref#1: shared data backref parent 41020112896 count 1 > > [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 > > [26062.126746] extent refs 1 gen 2422941 flags 1 > > [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 > > [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 > > [26062.126750] extent refs 1 gen 1444854 flags 1 > > [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 > > [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 > > [26062.126754] extent refs 1 gen 1444854 flags 1 > > [26062.126755] ref#0: shared data backref parent 214892544 count 1 > > [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 > > [26062.126758] extent refs 1 gen 1444854 flags 1 > > [26062.126759] ref#0: shared data backref parent 102547456 count 1 > > [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 > > [26062.126762] extent refs 2 gen 2422644 flags 1 > > [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 > > [26062.126765] ref#1: shared data backref parent 470658482176 count 1 > > [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 > > [26062.126769] extent refs 2 gen 2369217 flags 1 > > [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 > > [26062.126772] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 > > [26062.126775] extent refs 2 gen 2369242 flags 1 > > [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 > > [26062.126778] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 > > [26062.126781] extent refs 1 gen 2369305 flags 1 > > [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 > > [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 > > [26062.126786] extent refs 2 gen 2369052 flags 1 > > [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 > > [26062.126788] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 > > [26062.126792] extent refs 1 gen 2369305 flags 1 > > [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 > > [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 > > [26062.126796] extent refs 2 gen 2368754 flags 1 > > [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 > > [26062.126799] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 > > [26062.126802] extent refs 11 gen 1800682 flags 1 > > [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 > > [26062.126805] ref#1: shared data backref parent 508153069568 count 1 > > [26062.126806] ref#2: shared data backref parent 508031270912 count 1 > > [26062.126808] ref#3: shared data backref parent 493564215296 count 1 > > [26062.126809] ref#4: shared data backref parent 493540098048 count 1 > > [26062.126811] ref#5: shared data backref parent 492676038656 count 1 > > [26062.126813] ref#6: shared data backref parent 475884142592 count 1 > > [26062.126814] ref#7: shared data backref parent 471032840192 count 1 > > [26062.126816] ref#8: shared data backref parent 436112146432 count 1 > > [26062.126817] ref#9: shared data backref parent 883752960 count 1 > > [26062.126818] ref#10: shared data backref parent 86654976 count 1 > > [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 > > [26062.126822] extent refs 2 gen 2369347 flags 1 > > [26062.126823] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126824] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 > > [26062.126827] extent refs 2 gen 2369347 flags 1 > > [26062.126828] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126830] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 > > [26062.126833] extent refs 2 gen 2369347 flags 1 > > [26062.126834] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126836] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 > > [26062.126840] extent refs 1 gen 2369352 flags 1 > > [26062.126841] ref#0: shared data backref parent 510058790912 count 1 > > [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 > > [26062.126845] extent refs 1 gen 2369352 flags 1 > > [26062.126845] ref#0: shared data backref parent 510058790912 count 1 > > [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 > > [26062.126849] extent refs 1 gen 2369352 flags 1 > > [26062.126850] ref#0: shared data backref parent 510058790912 count 1 > > [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 > > [26062.126853] extent refs 2 gen 2369347 flags 1 > > [26062.126854] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126856] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 > > [26062.126859] extent refs 2 gen 2369347 flags 1 > > [26062.126860] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126862] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 > > [26062.126865] extent refs 2 gen 2369347 flags 1 > > [26062.126866] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126868] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 > > [26062.126871] extent refs 2 gen 2369347 flags 1 > > [26062.126873] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126874] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 > > [26062.126877] extent refs 2 gen 2369347 flags 1 > > [26062.126878] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126880] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 > > [26062.126883] extent refs 2 gen 2369347 flags 1 > > [26062.126884] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126885] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 > > [26062.126889] extent refs 2 gen 2369347 flags 1 > > [26062.126890] ref#0: shared data backref parent 492981370880 count 1 > > [26062.126891] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 > > [26062.126894] extent refs 3 gen 2369347 flags 1 > > [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 > > [26062.126897] ref#1: shared data backref parent 492981370880 count 1 > > [26062.126898] ref#2: shared data backref parent 438017720320 count 1 > > [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 > > [26062.126902] extent refs 2 gen 2369347 flags 1 > > [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 > > [26062.126904] ref#1: shared data backref parent 438017720320 count 1 > > [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 > > [26062.126907] extent refs 1 gen 2369352 flags 1 > > [26062.126908] ref#0: shared data backref parent 510058790912 count 1 > > [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 > > [26062.126912] extent refs 6 gen 1800750 flags 1 > > [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 > > [26062.126915] ref#1: shared data backref parent 509782933504 count 2 > > [26062.126917] ref#2: shared data backref parent 508543795200 count 2 > > [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 > > [26062.126920] extent refs 2 gen 2369075 flags 1 > > [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 > > [26062.126923] ref#1: shared data backref parent 437615230976 count 1 > > [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 > > [26062.126926] extent refs 3 gen 1800733 flags 1 > > [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 > > [26062.126929] ref#1: shared data backref parent 509783719936 count 1 > > [26062.126930] ref#2: shared data backref parent 509351231488 count 1 > > [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 > > [26062.126934] extent refs 2 gen 2420471 flags 1 > > [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 > > [26062.126936] ref#1: shared data backref parent 41020112896 count 1 > > [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 > > [26062.126939] extent refs 2 gen 2420471 flags 1 > > [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 > > [26062.126943] ref#1: shared data backref parent 41020112896 count 1 > > [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 > > [26062.126946] extent refs 3 gen 1800733 flags 1 > > [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 > > [26062.126949] ref#1: shared data backref parent 509783719936 count 1 > > [26062.126950] ref#2: shared data backref parent 509351231488 count 1 > > [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 > > [26062.126954] extent refs 2 gen 2420502 flags 1 > > [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 > > [26062.126957] ref#1: shared data backref parent 492274728960 count 1 > > [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 > > [26062.126961] extent refs 2 gen 2420478 flags 1 > > [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 > > [26062.126964] ref#1: shared data backref parent 41020112896 count 1 > > [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 > > [26062.126967] extent refs 1 gen 1444854 flags 1 > > [26062.126968] ref#0: shared data backref parent 214908928 count 1 > > [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 > > [26062.126971] extent refs 1 gen 2376479 flags 1 > > [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 > > [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 > > [26062.126976] extent refs 1 gen 1444854 flags 1 > > [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 > > [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 > > [26062.126981] extent refs 1 gen 1444854 flags 1 > > [26062.126982] ref#0: shared data backref parent 93732864 count 1 > > [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 > > [26062.126985] extent refs 1 gen 1444854 flags 1 > > [26062.126986] ref#0: shared data backref parent 93732864 count 1 > > [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 > > [26062.126990] extent refs 1 gen 1444854 flags 1 > > [26062.126991] ref#0: shared data backref parent 93732864 count 1 > > [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 > > [26062.126994] extent refs 1 gen 1444854 flags 1 > > [26062.126995] ref#0: shared data backref parent 214908928 count 1 > > [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 > > [26062.126998] extent refs 4 gen 2419085 flags 1 > > [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 > > [26062.127001] ref#1: shared data backref parent 494075035648 count 1 > > [26062.127002] ref#2: shared data backref parent 470850240512 count 1 > > [26062.127004] ref#3: shared data backref parent 437034745856 count 1 > > [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 > > [26062.127008] extent refs 3 gen 1800733 flags 1 > > [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 > > [26062.127011] ref#1: shared data backref parent 509783719936 count 1 > > [26062.127013] ref#2: shared data backref parent 509351231488 count 1 > > [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 > > [26062.127016] extent refs 2 gen 2368859 flags 1 > > [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 > > [26062.127019] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 > > [26062.127023] extent refs 1 gen 2376479 flags 1 > > [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 > > [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 > > [26062.127028] extent refs 11 gen 1800674 flags 1 > > [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 > > [26062.127031] ref#1: shared data backref parent 508153118720 count 1 > > [26062.127032] ref#2: shared data backref parent 508031533056 count 1 > > [26062.127034] ref#3: shared data backref parent 493564477440 count 1 > > [26062.127036] ref#4: shared data backref parent 493540179968 count 1 > > [26062.127038] ref#5: shared data backref parent 492675366912 count 1 > > [26062.127039] ref#6: shared data backref parent 475132116992 count 1 > > [26062.127041] ref#7: shared data backref parent 471032922112 count 1 > > [26062.127042] ref#8: shared data backref parent 436112244736 count 1 > > [26062.127044] ref#9: shared data backref parent 883998720 count 1 > > [26062.127046] ref#10: shared data backref parent 87015424 count 1 > > [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 > > [26062.127050] extent refs 1 gen 2408131 flags 1 > > [26062.127052] ref#0: shared data backref parent 509278633984 count 1 > > [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 > > [26062.127056] extent refs 11 gen 1800715 flags 1 > > [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 > > [26062.127059] ref#1: shared data backref parent 508152889344 count 1 > > [26062.127061] ref#2: shared data backref parent 508031139840 count 1 > > [26062.127062] ref#3: shared data backref parent 493563478016 count 1 > > [26062.127065] ref#4: shared data backref parent 493539983360 count 1 > > [26062.127066] ref#5: shared data backref parent 492674990080 count 1 > > [26062.127068] ref#6: shared data backref parent 475132166144 count 1 > > [26062.127070] ref#7: shared data backref parent 471032348672 count 1 > > [26062.127071] ref#8: shared data backref parent 436111769600 count 1 > > [26062.127073] ref#9: shared data backref parent 883359744 count 1 > > [26062.127074] ref#10: shared data backref parent 85082112 count 1 > > [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 > > [26062.127078] extent refs 4 gen 2419085 flags 1 > > [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 > > [26062.127085] ref#1: shared data backref parent 494075035648 count 1 > > [26062.127088] ref#2: shared data backref parent 470850240512 count 1 > > [26062.127089] ref#3: shared data backref parent 437034745856 count 1 > > [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 > > [26062.127095] extent refs 2 gen 2421893 flags 1 > > [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 > > [26062.127099] ref#1: shared data backref parent 493703069696 count 1 > > [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 > > [26062.127104] extent refs 1 gen 1444854 flags 1 > > [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 > > [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 > > [26062.127111] extent refs 2 gen 2368700 flags 1 > > [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 > > [26062.127114] ref#1: shared data backref parent 494105804800 count 1 > > [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 > > [26062.127120] extent refs 1 gen 1444854 flags 1 > > [26062.127121] ref#0: shared data backref parent 93732864 count 1 > > [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 > > [26062.127127] extent refs 1 gen 1444854 flags 1 > > [26062.127128] ref#0: shared data backref parent 214433792 count 1 > > [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 > > [26062.127132] extent refs 1 gen 1444854 flags 1 > > [26062.127133] ref#0: shared data backref parent 214908928 count 1 > > [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 > > [26062.127138] extent refs 3 gen 1800750 flags 1 > > [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 > > [26062.127141] ref#1: shared data backref parent 509783556096 count 1 > > [26062.127143] ref#2: shared data backref parent 508544974848 count 1 > > [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 > > [26062.127148] extent refs 2 gen 2368707 flags 1 > > [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 > > [26062.127151] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 > > [26062.127155] extent refs 11 gen 1800674 flags 1 > > [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 > > [26062.127157] ref#1: shared data backref parent 508152905728 count 1 > > [26062.127159] ref#2: shared data backref parent 508031057920 count 1 > > [26062.127160] ref#3: shared data backref parent 493563166720 count 1 > > [26062.127161] ref#4: shared data backref parent 493539934208 count 1 > > [26062.127163] ref#5: shared data backref parent 492674908160 count 1 > > [26062.127164] ref#6: shared data backref parent 475884208128 count 1 > > [26062.127165] ref#7: shared data backref parent 471032266752 count 1 > > [26062.127167] ref#8: shared data backref parent 436111720448 count 1 > > [26062.127168] ref#9: shared data backref parent 883179520 count 1 > > [26062.127170] ref#10: shared data backref parent 83017728 count 1 > > [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 > > [26062.127173] extent refs 1 gen 1444854 flags 1 > > [26062.127174] ref#0: shared data backref parent 214908928 count 1 > > [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 > > [26062.127177] extent refs 2 gen 2381729 flags 1 > > [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 > > [26062.127180] ref#1: shared data backref parent 437808545792 count 1 > > [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 > > [26062.127183] extent refs 2 gen 2369098 flags 1 > > [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 > > [26062.127186] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 > > [26062.127189] extent refs 2 gen 2368700 flags 1 > > [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 > > [26062.127191] ref#1: shared data backref parent 494105804800 count 1 > > [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 > > [26062.127195] extent refs 1 gen 1444854 flags 1 > > [26062.127195] ref#0: shared data backref parent 102547456 count 1 > > [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 > > [26062.127198] extent refs 3 gen 1800750 flags 1 > > [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 > > [26062.127201] ref#1: shared data backref parent 509783408640 count 1 > > [26062.127202] ref#2: shared data backref parent 508543451136 count 1 > > [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 > > [26062.127205] extent refs 1 gen 1444854 flags 1 > > [26062.127206] ref#0: shared data backref parent 93732864 count 1 > > [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 > > [26062.127209] extent refs 1 gen 1444854 flags 1 > > [26062.127210] ref#0: shared data backref parent 93732864 count 1 > > [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 > > [26062.127214] extent refs 1 gen 1444854 flags 1 > > [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 > > [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 > > [26062.127219] extent refs 4 gen 2419085 flags 1 > > [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 > > [26062.127222] ref#1: shared data backref parent 494075035648 count 1 > > [26062.127223] ref#2: shared data backref parent 470850240512 count 1 > > [26062.127225] ref#3: shared data backref parent 437034745856 count 1 > > [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 > > [26062.127228] extent refs 3 gen 1800750 flags 1 > > [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 > > [26062.127230] ref#1: shared data backref parent 509783408640 count 1 > > [26062.127232] ref#2: shared data backref parent 508543451136 count 1 > > [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 > > [26062.127235] extent refs 2 gen 2368731 flags 1 > > [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 > > [26062.127237] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 > > [26062.127241] extent refs 1 gen 1444854 flags 1 > > [26062.127241] ref#0: shared data backref parent 93732864 count 1 > > [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 > > [26062.127244] extent refs 1 gen 1444854 flags 1 > > [26062.127245] ref#0: shared data backref parent 93732864 count 1 > > [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 > > [26062.127249] extent refs 1 gen 1444854 flags 1 > > [26062.127249] ref#0: shared data backref parent 93732864 count 1 > > [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 > > [26062.127253] extent refs 1 gen 1444854 flags 1 > > [26062.127254] ref#0: shared data backref parent 93732864 count 1 > > [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 > > [26062.127258] extent refs 1 gen 1444854 flags 1 > > [26062.127259] ref#0: shared data backref parent 93732864 count 1 > > [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 > > [26062.127263] extent refs 1 gen 2381741 flags 1 > > [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 > > [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 > > [26062.127268] extent refs 2 gen 2368882 flags 1 > > [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 > > [26062.127270] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 > > [26062.127274] extent refs 2 gen 2368909 flags 1 > > [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 > > [26062.127277] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 > > [26062.127280] extent refs 2 gen 2369146 flags 1 > > [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 > > [26062.127283] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 > > [26062.127286] extent refs 1 gen 2369353 flags 1 > > [26062.127287] ref#0: shared data backref parent 510167498752 count 1 > > [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 > > [26062.127294] extent refs 3 gen 1800733 flags 1 > > [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 > > [26062.127296] ref#1: shared data backref parent 509783752704 count 1 > > [26062.127298] ref#2: shared data backref parent 509351313408 count 1 > > [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 > > [26062.127303] extent refs 22 gen 1800674 flags 1 > > [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 > > [26062.127306] ref#1: shared data backref parent 508152889344 count 2 > > [26062.127307] ref#2: shared data backref parent 508031139840 count 2 > > [26062.127308] ref#3: shared data backref parent 493563478016 count 2 > > [26062.127310] ref#4: shared data backref parent 493539983360 count 2 > > [26062.127311] ref#5: shared data backref parent 492674990080 count 2 > > [26062.127312] ref#6: shared data backref parent 475132166144 count 2 > > [26062.127314] ref#7: shared data backref parent 471032348672 count 2 > > [26062.127315] ref#8: shared data backref parent 436111769600 count 2 > > [26062.127317] ref#9: shared data backref parent 883359744 count 2 > > [26062.127318] ref#10: shared data backref parent 85082112 count 2 > > [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 > > [26062.127321] extent refs 33 gen 1800674 flags 1 > > [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 > > [26062.127324] ref#1: shared data backref parent 508152889344 count 3 > > [26062.127325] ref#2: shared data backref parent 508031139840 count 3 > > [26062.127327] ref#3: shared data backref parent 493563478016 count 3 > > [26062.127328] ref#4: shared data backref parent 493539983360 count 3 > > [26062.127329] ref#5: shared data backref parent 492674990080 count 3 > > [26062.127331] ref#6: shared data backref parent 475132166144 count 3 > > [26062.127333] ref#7: shared data backref parent 471032348672 count 3 > > [26062.127334] ref#8: shared data backref parent 436111769600 count 3 > > [26062.127335] ref#9: shared data backref parent 883359744 count 3 > > [26062.127337] ref#10: shared data backref parent 85082112 count 3 > > [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 > > [26062.127340] extent refs 4 gen 2419085 flags 1 > > [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 > > [26062.127342] ref#1: shared data backref parent 494075035648 count 1 > > [26062.127344] ref#2: shared data backref parent 470850240512 count 1 > > [26062.127345] ref#3: shared data backref parent 437034745856 count 1 > > [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 > > [26062.127348] extent refs 2 gen 2368932 flags 1 > > [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 > > [26062.127351] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 > > [26062.127354] extent refs 1 gen 2369353 flags 1 > > [26062.127355] ref#0: shared data backref parent 510165155840 count 1 > > [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 > > [26062.127358] extent refs 2 gen 2369168 flags 1 > > [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 > > [26062.127360] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 > > [26062.127364] extent refs 1 gen 2369305 flags 1 > > [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 > > [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 > > [26062.127369] extent refs 4 gen 2419085 flags 1 > > [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 > > [26062.127372] ref#1: shared data backref parent 494075035648 count 1 > > [26062.127374] ref#2: shared data backref parent 470850240512 count 1 > > [26062.127375] ref#3: shared data backref parent 437034745856 count 1 > > [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 > > [26062.127378] extent refs 2 gen 2368955 flags 1 > > [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 > > [26062.127382] ref#1: shared data backref parent 437615230976 count 1 > > [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 > > [26062.127385] extent refs 1 gen 2369352 flags 1 > > [26062.127386] ref#0: shared data backref parent 510178295808 count 1 > > [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 > > [26062.127390] extent refs 1 gen 2369352 flags 1 > > [26062.127391] ref#0: shared data backref parent 510058790912 count 1 > > [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 > > [26062.127394] extent refs 1 gen 2369352 flags 1 > > [26062.127395] ref#0: shared data backref parent 510059184128 count 1 > > [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 > > [26062.127399] extent refs 1 gen 2378552 flags 1 > > [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 > > [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 > > [26062.127405] extent refs 1 gen 2378552 flags 1 > > [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 > > [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 > > [26062.127409] extent refs 1 gen 2381741 flags 1 > > [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 > > [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 > > [26062.127414] extent refs 2 gen 2368700 flags 1 > > [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 > > [26062.127416] ref#1: shared data backref parent 494105804800 count 1 > > [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 > > [26062.127420] extent refs 2 gen 2368700 flags 1 > > [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 > > [26062.127423] ref#1: shared data backref parent 494105804800 count 1 > > [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 > > [26062.127426] extent refs 2 gen 2368700 flags 1 > > [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 > > [26062.127430] ref#1: shared data backref parent 494105804800 count 1 > > [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 > > [26062.127434] extent refs 11 gen 1800699 flags 1 > > [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 > > [26062.127437] ref#1: shared data backref parent 508152987648 count 1 > > [26062.127439] ref#2: shared data backref parent 508031172608 count 1 > > [26062.127440] ref#3: shared data backref parent 493563822080 count 1 > > [26062.127441] ref#4: shared data backref parent 493540130816 count 1 > > [26062.127443] ref#5: shared data backref parent 492675170304 count 1 > > [26062.127444] ref#6: shared data backref parent 475884584960 count 1 > > [26062.127446] ref#7: shared data backref parent 471032496128 count 1 > > [26062.127447] ref#8: shared data backref parent 436111884288 count 1 > > [26062.127448] ref#9: shared data backref parent 883703808 count 1 > > [26062.127450] ref#10: shared data backref parent 85671936 count 1 > > [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 > > [26062.127453] extent refs 1 gen 1444854 flags 1 > > [26062.127454] ref#0: shared data backref parent 214908928 count 1 > > [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 > > [26062.127458] extent refs 1 gen 2426255 flags 1 > > [26062.127459] ref#0: shared data backref parent 436271005696 count 1 > > [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 > > [26062.127463] extent refs 22 gen 1800674 flags 1 > > [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 > > [26062.127465] ref#1: shared data backref parent 508152987648 count 2 > > [26062.127467] ref#2: shared data backref parent 508031172608 count 2 > > [26062.127468] ref#3: shared data backref parent 493563822080 count 2 > > [26062.127470] ref#4: shared data backref parent 493540130816 count 2 > > [26062.127472] ref#5: shared data backref parent 492675170304 count 2 > > [26062.127473] ref#6: shared data backref parent 475884584960 count 2 > > [26062.127475] ref#7: shared data backref parent 471032496128 count 2 > > [26062.127476] ref#8: shared data backref parent 436111884288 count 2 > > [26062.127478] ref#9: shared data backref parent 883703808 count 2 > > [26062.127480] ref#10: shared data backref parent 85671936 count 2 > > [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 > > [26062.127486] extent refs 1 gen 1444854 flags 1 > > [26062.127488] ref#0: shared data backref parent 167198720 count 1 > > [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 > > [26062.127494] extent refs 2 gen 1400856 flags 1 > > [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 > > [26062.127498] ref#1: shared data backref parent 470954196992 count 1 > > [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 > > [26062.127503] extent refs 1 gen 1444854 flags 1 > > [26062.127504] ref#0: shared data backref parent 93732864 count 1 > > [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 > > [26062.127510] extent refs 1 gen 2381741 flags 1 > > [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 > > [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 > > [26062.127520] extent refs 2 gen 2368701 flags 1 > > [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 > > [26062.127523] ref#1: shared data backref parent 494107131904 count 1 > > [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 > > [26062.127527] extent refs 44 gen 1800674 flags 1 > > [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 > > [26062.127530] ref#1: shared data backref parent 508152987648 count 4 > > [26062.127532] ref#2: shared data backref parent 508031172608 count 4 > > [26062.127533] ref#3: shared data backref parent 493563822080 count 4 > > [26062.127535] ref#4: shared data backref parent 493540130816 count 4 > > [26062.127537] ref#5: shared data backref parent 492675170304 count 4 > > [26062.127539] ref#6: shared data backref parent 475884584960 count 4 > > [26062.127540] ref#7: shared data backref parent 471032496128 count 4 > > [26062.127542] ref#8: shared data backref parent 436111884288 count 4 > > [26062.127543] ref#9: shared data backref parent 883703808 count 4 > > [26062.127545] ref#10: shared data backref parent 85671936 count 4 > > [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 > > [26062.127549] extent refs 3 gen 1800750 flags 1 > > [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 > > [26062.127554] ref#1: shared data backref parent 509783130112 count 1 > > [26062.127557] ref#2: shared data backref parent 508544876544 count 1 > > [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 > > [26062.127561] extent refs 1 gen 1444854 flags 1 > > [26062.127562] ref#0: shared data backref parent 93732864 count 1 > > [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 > > [26062.127566] extent refs 1 gen 1444854 flags 1 > > [26062.127568] ref#0: shared data backref parent 214908928 count 1 > > [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 > > [26062.127572] extent refs 1 gen 1444854 flags 1 > > [26062.127573] ref#0: shared data backref parent 93732864 count 1 > > [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 > > [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 > > [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 > > [26062.127583] extent refs 1 gen 1444854 flags 1 > > [26062.127584] ref#0: shared data backref parent 167198720 count 1 > > [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 > > [26062.127587] extent refs 1 gen 1444854 flags 1 > > [26062.127588] ref#0: shared data backref parent 167198720 count 1 > > [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 > > [26062.127591] extent refs 1 gen 1444854 flags 1 > > [26062.127592] ref#0: shared data backref parent 167198720 count 1 > > [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 > > [26062.127595] extent refs 1 gen 1444854 flags 1 > > [26062.127596] ref#0: shared data backref parent 167198720 count 1 > > [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 > > [26062.127599] extent refs 1 gen 1444854 flags 1 > > [26062.127600] ref#0: shared data backref parent 167198720 count 1 > > [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 > > [26062.127603] extent refs 1 gen 1444854 flags 1 > > [26062.127604] ref#0: shared data backref parent 167198720 count 1 > > [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 > > [26062.127607] extent refs 11 gen 2409230 flags 1 > > [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 > > [26062.127610] ref#1: shared data backref parent 508153675776 count 1 > > [26062.127611] ref#2: shared data backref parent 493566640128 count 1 > > [26062.127613] ref#3: shared data backref parent 493540999168 count 1 > > [26062.127614] ref#4: shared data backref parent 492676497408 count 1 > > [26062.127616] ref#5: shared data backref parent 475884634112 count 1 > > [26062.127617] ref#6: shared data backref parent 475691106304 count 1 > > [26062.127619] ref#7: shared data backref parent 471034494976 count 1 > > [26062.127620] ref#8: shared data backref parent 436113883136 count 1 > > [26062.127622] ref#9: shared data backref parent 886407168 count 1 > > [26062.127623] ref#10: shared data backref parent 89440256 count 1 > > [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 > > [26062.127626] extent refs 10 gen 2413323 flags 1 > > [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 > > [26062.127631] ref#1: shared data backref parent 508153675776 count 1 > > [26062.127633] ref#2: shared data backref parent 493566640128 count 1 > > [26062.127635] ref#3: shared data backref parent 493540999168 count 1 > > [26062.127636] ref#4: shared data backref parent 492676497408 count 1 > > [26062.127638] ref#5: shared data backref parent 475691106304 count 1 > > [26062.127640] ref#6: shared data backref parent 471034494976 count 1 > > [26062.127642] ref#7: shared data backref parent 436113883136 count 1 > > [26062.127643] ref#8: shared data backref parent 886407168 count 1 > > [26062.127645] ref#9: shared data backref parent 89440256 count 1 > > [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 > > [26062.127648] extent refs 1 gen 2407793 flags 1 > > [26062.127649] ref#0: shared data backref parent 475884634112 count 1 > > [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 > > [26062.127655] extent refs 11 gen 1800729 flags 1 > > [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 > > [26062.127658] ref#1: shared data backref parent 508153069568 count 1 > > [26062.127659] ref#2: shared data backref parent 508031270912 count 1 > > [26062.127661] ref#3: shared data backref parent 493564215296 count 1 > > [26062.127663] ref#4: shared data backref parent 493540098048 count 1 > > [26062.127664] ref#5: shared data backref parent 492676038656 count 1 > > [26062.127666] ref#6: shared data backref parent 475884142592 count 1 > > [26062.127668] ref#7: shared data backref parent 471032840192 count 1 > > [26062.127669] ref#8: shared data backref parent 436112146432 count 1 > > [26062.127671] ref#9: shared data backref parent 883752960 count 1 > > [26062.127674] ref#10: shared data backref parent 86654976 count 1 > > [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 > > [26062.127680] ------------[ cut here ]------------ > > [26062.127682] BTRFS: Transaction aborted (error -2) > > [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 > > [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc snd _hda_intel > > [26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > > [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > > [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > > [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 > > [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df > > [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 > > [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 > > [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 > > [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > > [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 > > [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > > [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > > [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > > [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > [26062.127823] Call Trace: > > [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 > > [26062.127837] ? __switch_to_asm+0x41/0x70 > > [26062.127839] ? __switch_to_asm+0x35/0x70 > > [26062.127841] ? __switch_to_asm+0x41/0x70 > > [26062.127848] ? __switch_to+0x13d/0x3d5 > > [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 > > [26062.127855] btrfs_commit_transaction+0x55/0x7c8 > > [26062.127858] ? start_transaction+0x347/0x3cb > > [26062.127862] transaction_kthread+0xc9/0x135 > > [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 > > [26062.127869] kthread+0xeb/0xf0 > > [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 > > [26062.127875] ret_from_fork+0x35/0x40 > > [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- > > [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry > > [26062.127917] BTRFS info (device dm-2): forced readonly > > [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry > > On Tue, Jul 17, 2018 at 05:24:51PM -0700, Marc MERLIN wrote: > > On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: > > > No OOM triggers? That's a little strange. > > > Maybe it's related to how kernel handles memory over-commit? > > > > Yes, I think you are correct. > > > > > And for the hang, I think it's related to some memory allocation failure > > > and error handler just didn't handle it well, so it's causing deadlock > > > for certain page. > > > > That indeed matches what I'm seeing. > > > > > ENOMEM handling is pretty common but hardly verified, so it's not that > > > strange, but we must locate the problem. > > > > I seem to be getting deadlocks in the kernel, so I'm hoping that at least > > it's checked there, but maybe not? > > > > > In my system, at least I'm not using btrfs as root fs, and for the > > > memory eating program I normally ensure it's eating all the memory + > > > swap, so OOM killer is always triggered, maybe that's the cause. > > > > > > So in your case, maybe it's btrfs not really taking up all memory, thus > > > OOM killer not triggered. > > > > Correct, the swap is not used. > > > > > Any kernel dmesg about OOM killer triggered? > > > > Nothing at all. It never gets triggered. > > > > > > Here is my system when it virtually died: > > > > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > > > See how btrs was taking 29GB in that ps output (that's before it takes > > everything and I can't even type ps anymore) > > Note that VSZ is almost equal to RSS. Nothing gets swapped. > > > > Then see free output: > > > > > > total used free shared buffers cached > > > > Mem: 32643788 32180100 463688 0 44664 119508 > > > > -/+ buffers/cache: 32015928 627860 > > > > Swap: 15616764 443676 15173088 > > > > > > For swap, it looks like only some other program's memory is swapped out, > > > not btrfs'. > > > > That's exactly correct. btrfs check never goes to swap, I'm not sure why, > > and because there is virtual memory free, maybe that's why OOM does not > > trigger? > > So I guess I can probably "fix" my problem by removing swap, but ultimately > > it would be useful to know why memory taken by btrfs check does not end up > > in swap. > > > > > And unfortunately, I'm not so familiar with OOM/MM code outside of > > > filesystem. > > > Any help from other experienced developers would definitely help to > > > solve why memory of 'btrfs check' is not swapped out or why OOM killer > > > is not triggered. > > > > Do you have someone from linux-vm you might be able to ask, or should we Cc > > this thread there? > > > > Thanks, > > Marc > > -- > > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Microsoft is to operating systems .... > > .... what McDonalds is to gourmet cooking > > Home page: http://marc.merlins.org/ > > On Tue, Jul 17, 2018 at 01:59:05PM -0700, Marc MERLIN wrote: > > Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. > > It just takes all the memory until linux hangs everywhere, and somehow (no idea why) > > the OOM killer never triggers. > > Details below: > > > > On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: > > > Here is what I got when the system was not doing well (it took minutes to run): > > > > > > total used free shared buffers cached > > > Mem: 32643788 32070952 572836 0 102160 4378772 > > > -/+ buffers/cache: 27590020 5053768 > > > Swap: 15616764 973596 14643168 > > > > ok, the reason it was not that close to 0 was due to /dev/shm it seems. > > I cleared that, and now I can get it to go to near 0 again. > > I'm wrong about the system being fully crashed, it's not, it's just very > > close to being hung. > > I can type killall -9 btrfs in the serial console and wait a few minutes. > > The system eventually recovers, but it's impossible to fix anything via ssh > > apparently because networking does not get to run when I'm in this state. > > > > I'm not sure why my system reproduces this easy while Qu's system does not, > > but Qu was right that the kernel is not dead and that it's merely a problem of userspace > > taking all the RAM and somehow not being killed by OOM > > > > I checked the PID and don't see why it's not being killed: > > gargamel:/proc/31006# grep . oom* > > oom_adj:0 > > oom_score:221 << this increases a lot, but OOM never kills it > > oom_score_adj:0 > > > > I have these variables: > > /proc/sys/vm/oom_dump_tasks:1 > > /proc/sys/vm/oom_kill_allocating_task:0 > > /proc/sys/vm/overcommit_kbytes:0 > > /proc/sys/vm/overcommit_memory:0 > > /proc/sys/vm/overcommit_ratio:50 << is this bad (seems default) > > > > Here is my system when it virtually died: > > ER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 31006 21.2 90.7 29639020 29623180 pts/19 D+ 13:49 1:35 ./btrfs check /dev/mapper/dshelf2 > > > > total used free shared buffers cached > > Mem: 32643788 32180100 463688 0 44664 119508 > > -/+ buffers/cache: 32015928 627860 > > Swap: 15616764 443676 15173088 > > > > MemTotal: 32643788 kB > > MemFree: 463440 kB > > MemAvailable: 44864 kB > > Buffers: 44664 kB > > Cached: 120360 kB > > SwapCached: 87064 kB > > Active: 30381404 kB > > Inactive: 585952 kB > > Active(anon): 30334696 kB > > Inactive(anon): 474624 kB > > Active(file): 46708 kB > > Inactive(file): 111328 kB > > Unevictable: 5616 kB > > Mlocked: 5616 kB > > SwapTotal: 15616764 kB > > SwapFree: 15173088 kB > > Dirty: 1636 kB > > Writeback: 4 kB > > AnonPages: 30734240 kB > > Mapped: 67236 kB > > Shmem: 3036 kB > > Slab: 267884 kB > > SReclaimable: 51528 kB > > SUnreclaim: 216356 kB > > KernelStack: 10144 kB > > PageTables: 69284 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 31938656 kB > > Committed_AS: 32865492 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 0 kB > > VmallocChunk: 0 kB > > HardwareCorrupted: 0 kB > > AnonHugePages: 0 kB > > ShmemHugePages: 0 kB > > ShmemPmdMapped: 0 kB > > CmaTotal: 16384 kB > > CmaFree: 0 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > Hugetlb: 0 kB > > DirectMap4k: 560404 kB > > DirectMap2M: 32692224 kB > > > > > > -- > > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Microsoft is to operating systems .... > > .... what McDonalds is to gourmet cooking > > Home page: http://marc.merlins.org/ > > > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.1.21: fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 2019-10-23 0:55 ` Marc MERLIN @ 2019-10-26 3:36 ` Marc MERLIN 0 siblings, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2019-10-26 3:36 UTC (permalink / raw) To: Linux BTRFS Mailinglist, Qu Wenruo, Su Yue So, no reply at all. No one cares about massive filesystem corruption in a recent enough kernel? On Tue, Oct 22, 2019 at 05:55:53PM -0700, Marc MERLIN wrote: > On Fri, Oct 18, 2019 at 08:07:28PM -0700, Marc MERLIN wrote: > > Ok, so before blowing the filesystem away after it was apparently badly > > damaged by a suspend to disk, I tried check --repair and I hit an > > infinite loop. > > > > Let me know if you'd like anything off the FS before I delete it. > > I heard nothing back, so I deleted the FS and restored from backup. > > But now I'm scared of ever doing a suspend to disk again. > Could someone please look at the logs and give me some idea of what > happened, if at all possible? > > Non recoverable data corruption on my laptop when I travel and > backups/restores are complicated, is a bit unnerving... > > Thanks, > Marc > > > Thanks, > > Marc > > > > enabling repair mode > > repair mode will force to clear out log tree, are you sure? [y/N]: y > > Checking filesystem on /dev/mapper/pool1 > > UUID: fda628bc-1ca4-49c5-91c2-4260fe967a23 > > checking extents > > Backref 415334400 parent 36028797198598144 not referenced back 0x5648ef1870e0 > > Backref 415334400 parent 179634176 root 179634176 not found in extent tree > > Incorrect global backref count on 415334400 found 2 wanted 1 > > backpointer mismatch on [415334400 16384] > > repair deleting extent record: key 415334400 169 0 > > adding new tree backref on start 415334400 len 16384 parent 179634176 root 179634176 > > Repaired extent references for 415334400 > > ref mismatch on [101995261952 4096] extent item 36028797018963969, found 1 > > repair deleting extent record: key 101995261952 168 4096 > > adding new data backref on 101995261952 root 456 owner 74455677 offset 64892928 found 1 > > Repaired extent references for 101995261952 > > Incorrect local backref count on 458640384000 root 456 owner 81409181 offset 17039360 found 0 wanted 1 back 0x5648eefd3d10 > > Backref disk bytenr does not match extent record, bytenr=458640384000, ref bytenr=0 > > Backref 458640384000 root 456 owner 73020573 offset 17039360 num_refs 0 not found in extent tree > > Incorrect local backref count on 458640384000 root 456 owner 73020573 offset 17039360 found 1 wanted 0 back 0x5648b32a9600 > > backpointer mismatch on [458640384000 86016] > > repair deleting extent record: key 458640384000 168 86016 > > adding new data backref on 458640384000 parent 438017720320 owner 0 offset 0 found 1 > > adding new data backref on 458640384000 root 456 owner 73020573 offset 17039360 found 1 > > Repaired extent references for 458640384000 > > Fixed 0 roots. > > checking free space cache > > cache and super generation don't match, space cache will be invalidated > > checking fs roots > > Deleting bad dir index [10138517,96,436945] root 456 > > Deleting bad dir index [10138518,96,646273] root 456 > > Deleting bad dir index [10138517,96,437016] root 456 > > Deleting bad dir index [10138518,96,808999] root 456 > > Deleting bad dir index [10215134,96,149427] root 456 > > Deleting bad dir index [10240541,96,268037] root 456 > > Deleting bad dir index [10138517,96,540247] root 456 > > Deleting bad dir index [10138518,96,825234] root 456 > > Deleting bad dir index [10138517,96,736673] root 456 > > Deleting bad dir index [10138518,96,1118221] root 456 > > Deleting bad dir index [10240541,96,439703] root 456 > > Deleting bad dir index [10138517,96,752282] root 456 > > root 456 inode 75431563 errors 100, file extent discount > > Found file extent holes: > > start: 4096, len: 4096 > > root 456 inode 75431568 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 1638400 > > root 456 inode 75431583 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 147456 > > root 456 inode 75431585 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 208896 > > root 456 inode 75431591 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 2523136 > > root 456 inode 75431730 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 208896 > > root 456 inode 75431744 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 2084864 > > root 456 inode 75431751 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 172032 > > root 456 inode 75431756 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > root 456 inode 75431760 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 12288 > > root 456 inode 75431765 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 32768 > > root 456 inode 75431773 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 90112 > > Fixed discount file extents for inode: 75432421 in root: 456 > > Fixed discount file extents for inode: 75432429 in root: 456 > > root 456 inode 75432429 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432430 in root: 456 > > Fixed discount file extents for inode: 75432432 in root: 456 > > Fixed discount file extents for inode: 75432433 in root: 456 > > root 456 inode 75432433 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > root 456 inode 75432434 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 724992 > > Fixed discount file extents for inode: 75432436 in root: 456 > > Fixed discount file extents for inode: 75432437 in root: 456 > > root 456 inode 75432437 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432438 in root: 456 > > Fixed discount file extents for inode: 75432441 in root: 456 > > root 456 inode 75432441 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432444 in root: 456 > > root 456 inode 75432444 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432445 in root: 456 > > Fixed discount file extents for inode: 75432447 in root: 456 > > Fixed discount file extents for inode: 75432448 in root: 456 > > Fixed discount file extents for inode: 75432454 in root: 456 > > Fixed discount file extents for inode: 75432457 in root: 456 > > root 456 inode 75432457 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > root 456 inode 75432471 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 20480 > > Fixed discount file extents for inode: 75432473 in root: 456 > > root 456 inode 75432473 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 20480 > > Fixed discount file extents for inode: 75432482 in root: 456 > > Fixed discount file extents for inode: 75432483 in root: 456 > > Fixed discount file extents for inode: 75432484 in root: 456 > > Fixed discount file extents for inode: 75432487 in root: 456 > > Fixed discount file extents for inode: 75432489 in root: 456 > > Fixed discount file extents for inode: 75432494 in root: 456 > > Fixed discount file extents for inode: 75432611 in root: 456 > > Fixed discount file extents for inode: 75432622 in root: 456 > > Fixed discount file extents for inode: 75432626 in root: 456 > > Fixed discount file extents for inode: 75432628 in root: 456 > > Fixed discount file extents for inode: 75432636 in root: 456 > > Fixed discount file extents for inode: 75432637 in root: 456 > > root 456 inode 75432637 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432643 in root: 456 > > root 456 inode 75432643 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 135168 > > Fixed discount file extents for inode: 75432650 in root: 456 > > root 456 inode 75432650 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432656 in root: 456 > > root 456 inode 75432656 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432657 in root: 456 > > root 456 inode 75432657 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432659 in root: 456 > > Fixed discount file extents for inode: 75432662 in root: 456 > > root 456 inode 75432662 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432663 in root: 456 > > root 456 inode 75432663 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432664 in root: 456 > > root 456 inode 75432664 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432674 in root: 456 > > root 456 inode 75432674 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432692 in root: 456 > > root 456 inode 75432692 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432708 in root: 456 > > Fixed discount file extents for inode: 75432709 in root: 456 > > root 456 inode 75432709 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432712 in root: 456 > > Fixed discount file extents for inode: 75432717 in root: 456 > > Fixed discount file extents for inode: 75432720 in root: 456 > > Fixed discount file extents for inode: 75432725 in root: 456 > > Fixed discount file extents for inode: 75432731 in root: 456 > > Fixed discount file extents for inode: 75432732 in root: 456 > > Fixed discount file extents for inode: 75432749 in root: 456 > > Fixed discount file extents for inode: 75432762 in root: 456 > > Fixed discount file extents for inode: 75432765 in root: 456 > > Fixed discount file extents for inode: 75432783 in root: 456 > > Fixed discount file extents for inode: 75432784 in root: 456 > > Fixed discount file extents for inode: 75432785 in root: 456 > > root 456 inode 75432785 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432786 in root: 456 > > root 456 inode 75432786 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432796 in root: 456 > > root 456 inode 75432796 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432799 in root: 456 > > root 456 inode 75432799 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432801 in root: 456 > > root 456 inode 75432801 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432807 in root: 456 > > Fixed discount file extents for inode: 75432817 in root: 456 > > Fixed discount file extents for inode: 75432829 in root: 456 > > Fixed discount file extents for inode: 75432860 in root: 456 > > root 456 inode 75432860 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 4096 > > Fixed discount file extents for inode: 75432862 in root: 456 > > Fixed discount file extents for inode: 75432863 in root: 456 > > Fixed discount file extents for inode: 75432869 in root: 456 > > Fixed discount file extents for inode: 75432870 in root: 456 > > Fixed discount file extents for inode: 75432871 in root: 456 > > Fixed discount file extents for inode: 75432872 in root: 456 > > Fixed discount file extents for inode: 75432875 in root: 456 > > Fixed discount file extents for inode: 75432877 in root: 456 > > Fixed discount file extents for inode: 75432882 in root: 456 > > Fixed discount file extents for inode: 75432883 in root: 456 > > Fixed discount file extents for inode: 75432893 in root: 456 > > Fixed discount file extents for inode: 75432894 in root: 456 > > Fixed discount file extents for inode: 75432897 in root: 456 > > Fixed discount file extents for inode: 75432899 in root: 456 > > Fixed discount file extents for inode: 75432900 in root: 456 > > Fixed discount file extents for inode: 75432905 in root: 456 > > Fixed discount file extents for inode: 75432906 in root: 456 > > Fixed discount file extents for inode: 75432916 in root: 456 > > Fixed discount file extents for inode: 75432917 in root: 456 > > Fixed discount file extents for inode: 75432919 in root: 456 > > Fixed discount file extents for inode: 75432920 in root: 456 > > Fixed discount file extents for inode: 75432923 in root: 456 > > Fixed discount file extents for inode: 75432942 in root: 456 > > Fixed discount file extents for inode: 75432944 in root: 456 > > Fixed discount file extents for inode: 75432948 in root: 456 > > root 456 inode 75432948 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > Fixed discount file extents for inode: 75432949 in root: 456 > > root 456 inode 75432949 errors 100, file extent discount > > Found file extent holes: > > start: 0, len: 8192 > > and it loops forever on 456 > > > > On Thu, Oct 17, 2019 at 07:56:04PM -0700, Marc MERLIN wrote: > > > This happened almost after a resume from suspend to disk. > > > First corruption and read only I got a very long time. > > > > > > Could they be related? > > > > > > [26062.126505] ------------[ cut here ]------------ > > > [26062.126524] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7100 __btrfs_free_extent+0x18b/0x921 > > > [26062.126526] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s nd_hda_intel > > > [26062.126561] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > > > [26062.126597] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > > > [26062.126599] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > > > [26062.126604] RIP: 0010:__btrfs_free_extent+0x18b/0x921 > > > [26062.126606] Code: 00 8b 45 40 44 29 e0 83 f8 05 0f 8f 2e 05 00 00 41 ff cc eb a5 83 f8 fe 0f 85 29 07 00 00 48 c7 c7 f8 67 f0 89 e8 f6 cb dd ff <0f> 0b 48 8b 7d 00 e8 e5 54 00 00 4c 89 fa 48 c7 c6 85 e0 f4 89 41 > > > [26062.126608] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010246 > > > [26062.126611] RAX: 0000000000000024 RBX: ffff9abca20884e0 RCX: 0000000000000000 > > > [26062.126613] RDX: 0000000000000000 RSI: ffff9abccf5d6558 RDI: ffff9abccf5d6558 > > > [26062.126617] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > > > [26062.126618] R10: 0000000000000002 R11: ffffb2d9c46e7b2f R12: 0000000000000169 > > > [26062.126622] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > > > [26062.126625] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > > > [26062.126627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [26062.126629] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > > > [26062.126633] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > [26062.126634] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > [26062.126636] Call Trace: > > > [26062.126647] __btrfs_run_delayed_refs+0x750/0xc36 > > > [26062.126653] ? __switch_to_asm+0x41/0x70 > > > [26062.126655] ? __switch_to_asm+0x35/0x70 > > > [26062.126658] ? __switch_to_asm+0x41/0x70 > > > [26062.126662] ? __switch_to+0x13d/0x3d5 > > > [26062.126668] btrfs_run_delayed_refs+0x5d/0x132 > > > [26062.126672] btrfs_commit_transaction+0x55/0x7c8 > > > [26062.126676] ? start_transaction+0x347/0x3cb > > > [26062.126679] transaction_kthread+0xc9/0x135 > > > [26062.126683] ? btrfs_cleanup_transaction+0x403/0x403 > > > [26062.126688] kthread+0xeb/0xf0 > > > [26062.126692] ? kthread_create_worker_on_cpu+0x65/0x65 > > > [26062.126695] ret_from_fork+0x35/0x40 > > > [26062.126698] ---[ end trace 4c1a6b3749a2f650 ]--- > > > [26062.126703] BTRFS info (device dm-2): leaf 510067163136 gen 2427077 total ptrs 130 free space 4329 owner 2 > > > [26062.126706] item 0 key (458630676480 168 65536) itemoff 16217 itemsize 66 > > > [26062.126708] extent refs 2 gen 2369265 flags 1 > > > [26062.126709] ref#0: extent data backref root 456 objectid 72925787 offset 5472256 count 1 > > > [26062.126711] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126714] item 1 key (458630856704 168 69632) itemoff 16151 itemsize 66 > > > [26062.126715] extent refs 2 gen 2369025 flags 1 > > > [26062.126716] ref#0: extent data backref root 456 objectid 72925787 offset 4796416 count 1 > > > [26062.126718] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126720] item 2 key (458631012352 168 16384) itemoff 15968 itemsize 183 > > > [26062.126722] extent refs 11 gen 1800715 flags 1 > > > [26062.126722] ref#0: extent data backref root 456 objectid 56413614 offset 63946752 count 1 > > > [26062.126724] ref#1: shared data backref parent 508153839616 count 1 > > > [26062.126726] ref#2: shared data backref parent 493541244928 count 1 > > > [26062.126727] ref#3: shared data backref parent 492677332992 count 1 > > > [26062.126728] ref#4: shared data backref parent 492328566784 count 1 > > > [26062.126730] ref#5: shared data backref parent 492245188608 count 1 > > > [26062.126731] ref#6: shared data backref parent 475885109248 count 1 > > > [26062.126732] ref#7: shared data backref parent 471036608512 count 1 > > > [26062.126733] ref#8: shared data backref parent 436114243584 count 1 > > > [26062.126735] ref#9: shared data backref parent 886718464 count 1 > > > [26062.126736] ref#10: shared data backref parent 90259456 count 1 > > > [26062.126738] item 3 key (458631028736 168 524288) itemoff 15902 itemsize 66 > > > [26062.126740] extent refs 2 gen 2420471 flags 1 > > > [26062.126741] ref#0: extent data backref root 456 objectid 75167213 offset 2097152 count 1 > > > [26062.126743] ref#1: shared data backref parent 41020112896 count 1 > > > [26062.126745] item 4 key (458631553024 168 69632) itemoff 15849 itemsize 53 > > > [26062.126746] extent refs 1 gen 2422941 flags 1 > > > [26062.126747] ref#0: extent data backref root 456 objectid 75259665 offset 0 count 1 > > > [26062.126749] item 5 key (458632052736 168 2269184) itemoff 15796 itemsize 53 > > > [26062.126750] extent refs 1 gen 1444854 flags 1 > > > [26062.126751] ref#0: extent data backref root 456 objectid 50397136 offset 0 count 1 > > > [26062.126753] item 6 key (458634321920 168 311296) itemoff 15759 itemsize 37 > > > [26062.126754] extent refs 1 gen 1444854 flags 1 > > > [26062.126755] ref#0: shared data backref parent 214892544 count 1 > > > [26062.126757] item 7 key (458635517952 168 1032192) itemoff 15722 itemsize 37 > > > [26062.126758] extent refs 1 gen 1444854 flags 1 > > > [26062.126759] ref#0: shared data backref parent 102547456 count 1 > > > [26062.126761] item 8 key (458636550144 168 438272) itemoff 15656 itemsize 66 > > > [26062.126762] extent refs 2 gen 2422644 flags 1 > > > [26062.126763] ref#0: extent data backref root 456 objectid 75244720 offset 0 count 1 > > > [26062.126765] ref#1: shared data backref parent 470658482176 count 1 > > > [26062.126768] item 9 key (458637037568 168 73728) itemoff 15590 itemsize 66 > > > [26062.126769] extent refs 2 gen 2369217 flags 1 > > > [26062.126770] ref#0: extent data backref root 456 objectid 72925787 offset 5332992 count 1 > > > [26062.126772] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126774] item 10 key (458637152256 168 73728) itemoff 15524 itemsize 66 > > > [26062.126775] extent refs 2 gen 2369242 flags 1 > > > [26062.126776] ref#0: extent data backref root 456 objectid 72925787 offset 5402624 count 1 > > > [26062.126778] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126780] item 11 key (458637328384 168 45056) itemoff 15471 itemsize 53 > > > [26062.126781] extent refs 1 gen 2369305 flags 1 > > > [26062.126782] ref#0: extent data backref root 456 objectid 73017376 offset 131072 count 1 > > > [26062.126784] item 12 key (458637885440 168 69632) itemoff 15405 itemsize 66 > > > [26062.126786] extent refs 2 gen 2369052 flags 1 > > > [26062.126787] ref#0: extent data backref root 456 objectid 72925787 offset 4861952 count 1 > > > [26062.126788] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126790] item 13 key (458638147584 168 69632) itemoff 15352 itemsize 53 > > > [26062.126792] extent refs 1 gen 2369305 flags 1 > > > [26062.126793] ref#0: extent data backref root 456 objectid 73017334 offset 131072 count 1 > > > [26062.126795] item 14 key (458638561280 168 73728) itemoff 15286 itemsize 66 > > > [26062.126796] extent refs 2 gen 2368754 flags 1 > > > [26062.126797] ref#0: extent data backref root 456 objectid 72925787 offset 4063232 count 1 > > > [26062.126799] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126801] item 15 key (458638970880 168 90112) itemoff 15103 itemsize 183 > > > [26062.126802] extent refs 11 gen 1800682 flags 1 > > > [26062.126803] ref#0: extent data backref root 456 objectid 56413614 offset 40665088 count 1 > > > [26062.126805] ref#1: shared data backref parent 508153069568 count 1 > > > [26062.126806] ref#2: shared data backref parent 508031270912 count 1 > > > [26062.126808] ref#3: shared data backref parent 493564215296 count 1 > > > [26062.126809] ref#4: shared data backref parent 493540098048 count 1 > > > [26062.126811] ref#5: shared data backref parent 492676038656 count 1 > > > [26062.126813] ref#6: shared data backref parent 475884142592 count 1 > > > [26062.126814] ref#7: shared data backref parent 471032840192 count 1 > > > [26062.126816] ref#8: shared data backref parent 436112146432 count 1 > > > [26062.126817] ref#9: shared data backref parent 883752960 count 1 > > > [26062.126818] ref#10: shared data backref parent 86654976 count 1 > > > [26062.126820] item 16 key (458639060992 168 86016) itemoff 15053 itemsize 50 > > > [26062.126822] extent refs 2 gen 2369347 flags 1 > > > [26062.126823] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126824] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126826] item 17 key (458639147008 168 86016) itemoff 15003 itemsize 50 > > > [26062.126827] extent refs 2 gen 2369347 flags 1 > > > [26062.126828] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126830] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126832] item 18 key (458639233024 168 90112) itemoff 14953 itemsize 50 > > > [26062.126833] extent refs 2 gen 2369347 flags 1 > > > [26062.126834] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126836] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126839] item 19 key (458639323136 168 81920) itemoff 14916 itemsize 37 > > > [26062.126840] extent refs 1 gen 2369352 flags 1 > > > [26062.126841] ref#0: shared data backref parent 510058790912 count 1 > > > [26062.126843] item 20 key (458639405056 168 77824) itemoff 14879 itemsize 37 > > > [26062.126845] extent refs 1 gen 2369352 flags 1 > > > [26062.126845] ref#0: shared data backref parent 510058790912 count 1 > > > [26062.126848] item 21 key (458639482880 168 69632) itemoff 14842 itemsize 37 > > > [26062.126849] extent refs 1 gen 2369352 flags 1 > > > [26062.126850] ref#0: shared data backref parent 510058790912 count 1 > > > [26062.126852] item 22 key (458639585280 168 86016) itemoff 14792 itemsize 50 > > > [26062.126853] extent refs 2 gen 2369347 flags 1 > > > [26062.126854] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126856] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126858] item 23 key (458639671296 168 90112) itemoff 14742 itemsize 50 > > > [26062.126859] extent refs 2 gen 2369347 flags 1 > > > [26062.126860] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126862] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126864] item 24 key (458639761408 168 81920) itemoff 14692 itemsize 50 > > > [26062.126865] extent refs 2 gen 2369347 flags 1 > > > [26062.126866] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126868] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126870] item 25 key (458640027648 168 90112) itemoff 14642 itemsize 50 > > > [26062.126871] extent refs 2 gen 2369347 flags 1 > > > [26062.126873] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126874] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126876] item 26 key (458640117760 168 90112) itemoff 14592 itemsize 50 > > > [26062.126877] extent refs 2 gen 2369347 flags 1 > > > [26062.126878] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126880] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126882] item 27 key (458640207872 168 86016) itemoff 14542 itemsize 50 > > > [26062.126883] extent refs 2 gen 2369347 flags 1 > > > [26062.126884] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126885] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126888] item 28 key (458640293888 168 90112) itemoff 14492 itemsize 50 > > > [26062.126889] extent refs 2 gen 2369347 flags 1 > > > [26062.126890] ref#0: shared data backref parent 492981370880 count 1 > > > [26062.126891] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126893] item 29 key (458640384000 168 86016) itemoff 14413 itemsize 79 > > > [26062.126894] extent refs 3 gen 2369347 flags 1 > > > [26062.126895] ref#0: extent data backref root 456 objectid 81409181 offset 17039360 count 1 > > > [26062.126897] ref#1: shared data backref parent 492981370880 count 1 > > > [26062.126898] ref#2: shared data backref parent 438017720320 count 1 > > > [26062.126900] item 30 key (458640470016 168 86016) itemoff 14347 itemsize 66 > > > [26062.126902] extent refs 2 gen 2369347 flags 1 > > > [26062.126903] ref#0: extent data backref root 456 objectid 73020573 offset 17170432 count 1 > > > [26062.126904] ref#1: shared data backref parent 438017720320 count 1 > > > [26062.126906] item 31 key (458640556032 168 77824) itemoff 14310 itemsize 37 > > > [26062.126907] extent refs 1 gen 2369352 flags 1 > > > [26062.126908] ref#0: shared data backref parent 510058790912 count 1 > > > [26062.126910] item 32 key (458640756736 168 24576) itemoff 14231 itemsize 79 > > > [26062.126912] extent refs 6 gen 1800750 flags 1 > > > [26062.126913] ref#0: extent data backref root 456 objectid 56413614 offset 697655296 count 2 > > > [26062.126915] ref#1: shared data backref parent 509782933504 count 2 > > > [26062.126917] ref#2: shared data backref parent 508543795200 count 2 > > > [26062.126919] item 33 key (458641227776 168 73728) itemoff 14165 itemsize 66 > > > [26062.126920] extent refs 2 gen 2369075 flags 1 > > > [26062.126921] ref#0: extent data backref root 456 objectid 72925787 offset 4927488 count 1 > > > [26062.126923] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.126925] item 34 key (458641346560 168 8192) itemoff 14086 itemsize 79 > > > [26062.126926] extent refs 3 gen 1800733 flags 1 > > > [26062.126927] ref#0: extent data backref root 456 objectid 56413614 offset 667910144 count 1 > > > [26062.126929] ref#1: shared data backref parent 509783719936 count 1 > > > [26062.126930] ref#2: shared data backref parent 509351231488 count 1 > > > [26062.126932] item 35 key (458641518592 168 524288) itemoff 14020 itemsize 66 > > > [26062.126934] extent refs 2 gen 2420471 flags 1 > > > [26062.126934] ref#0: extent data backref root 456 objectid 75167213 offset 2621440 count 1 > > > [26062.126936] ref#1: shared data backref parent 41020112896 count 1 > > > [26062.126938] item 36 key (458642042880 168 524288) itemoff 13954 itemsize 66 > > > [26062.126939] extent refs 2 gen 2420471 flags 1 > > > [26062.126940] ref#0: extent data backref root 456 objectid 75167213 offset 3145728 count 1 > > > [26062.126943] ref#1: shared data backref parent 41020112896 count 1 > > > [26062.126945] item 37 key (458642747392 168 8192) itemoff 13875 itemsize 79 > > > [26062.126946] extent refs 3 gen 1800733 flags 1 > > > [26062.126947] ref#0: extent data backref root 456 objectid 56413614 offset 668401664 count 1 > > > [26062.126949] ref#1: shared data backref parent 509783719936 count 1 > > > [26062.126950] ref#2: shared data backref parent 509351231488 count 1 > > > [26062.126953] item 38 key (458642841600 168 524288) itemoff 13809 itemsize 66 > > > [26062.126954] extent refs 2 gen 2420502 flags 1 > > > [26062.126955] ref#0: extent data backref root 456 objectid 75168659 offset 524288 count 1 > > > [26062.126957] ref#1: shared data backref parent 492274728960 count 1 > > > [26062.126959] item 39 key (458643431424 168 524288) itemoff 13743 itemsize 66 > > > [26062.126961] extent refs 2 gen 2420478 flags 1 > > > [26062.126962] ref#0: extent data backref root 456 objectid 75167502 offset 524288 count 1 > > > [26062.126964] ref#1: shared data backref parent 41020112896 count 1 > > > [26062.126966] item 40 key (458644230144 168 331776) itemoff 13706 itemsize 37 > > > [26062.126967] extent refs 1 gen 1444854 flags 1 > > > [26062.126968] ref#0: shared data backref parent 214908928 count 1 > > > [26062.126970] item 41 key (458644561920 168 1671168) itemoff 13653 itemsize 53 > > > [26062.126971] extent refs 1 gen 2376479 flags 1 > > > [26062.126972] ref#0: extent data backref root 456 objectid 73223285 offset 0 count 1 > > > [26062.126975] item 42 key (458646454272 168 2121728) itemoff 13600 itemsize 53 > > > [26062.126976] extent refs 1 gen 1444854 flags 1 > > > [26062.126977] ref#0: extent data backref root 456 objectid 50397133 offset 0 count 1 > > > [26062.126979] item 43 key (458648576000 168 1605632) itemoff 13563 itemsize 37 > > > [26062.126981] extent refs 1 gen 1444854 flags 1 > > > [26062.126982] ref#0: shared data backref parent 93732864 count 1 > > > [26062.126984] item 44 key (458650181632 168 1474560) itemoff 13526 itemsize 37 > > > [26062.126985] extent refs 1 gen 1444854 flags 1 > > > [26062.126986] ref#0: shared data backref parent 93732864 count 1 > > > [26062.126988] item 45 key (458651656192 168 2125824) itemoff 13489 itemsize 37 > > > [26062.126990] extent refs 1 gen 1444854 flags 1 > > > [26062.126991] ref#0: shared data backref parent 93732864 count 1 > > > [26062.126993] item 46 key (458653782016 168 356352) itemoff 13452 itemsize 37 > > > [26062.126994] extent refs 1 gen 1444854 flags 1 > > > [26062.126995] ref#0: shared data backref parent 214908928 count 1 > > > [26062.126997] item 47 key (458655539200 168 524288) itemoff 13360 itemsize 92 > > > [26062.126998] extent refs 4 gen 2419085 flags 1 > > > [26062.126999] ref#0: extent data backref root 456 objectid 75099153 offset 2621440 count 1 > > > [26062.127001] ref#1: shared data backref parent 494075035648 count 1 > > > [26062.127002] ref#2: shared data backref parent 470850240512 count 1 > > > [26062.127004] ref#3: shared data backref parent 437034745856 count 1 > > > [26062.127006] item 48 key (458656153600 168 8192) itemoff 13281 itemsize 79 > > > [26062.127008] extent refs 3 gen 1800733 flags 1 > > > [26062.127009] ref#0: extent data backref root 456 objectid 56413614 offset 668811264 count 1 > > > [26062.127011] ref#1: shared data backref parent 509783719936 count 1 > > > [26062.127013] ref#2: shared data backref parent 509351231488 count 1 > > > [26062.127015] item 49 key (458656161792 168 69632) itemoff 13215 itemsize 66 > > > [26062.127016] extent refs 2 gen 2368859 flags 1 > > > [26062.127017] ref#0: extent data backref root 456 objectid 72925787 offset 4325376 count 1 > > > [26062.127019] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127021] item 50 key (458657443840 168 1847296) itemoff 13162 itemsize 53 > > > [26062.127023] extent refs 1 gen 2376479 flags 1 > > > [26062.127024] ref#0: extent data backref root 456 objectid 73223286 offset 0 count 1 > > > [26062.127026] item 51 key (458659307520 168 98304) itemoff 12979 itemsize 183 > > > [26062.127028] extent refs 11 gen 1800674 flags 1 > > > [26062.127029] ref#0: extent data backref root 456 objectid 56413614 offset 41541632 count 1 > > > [26062.127031] ref#1: shared data backref parent 508153118720 count 1 > > > [26062.127032] ref#2: shared data backref parent 508031533056 count 1 > > > [26062.127034] ref#3: shared data backref parent 493564477440 count 1 > > > [26062.127036] ref#4: shared data backref parent 493540179968 count 1 > > > [26062.127038] ref#5: shared data backref parent 492675366912 count 1 > > > [26062.127039] ref#6: shared data backref parent 475132116992 count 1 > > > [26062.127041] ref#7: shared data backref parent 471032922112 count 1 > > > [26062.127042] ref#8: shared data backref parent 436112244736 count 1 > > > [26062.127044] ref#9: shared data backref parent 883998720 count 1 > > > [26062.127046] ref#10: shared data backref parent 87015424 count 1 > > > [26062.127048] item 52 key (458659405824 168 1753088) itemoff 12942 itemsize 37 > > > [26062.127050] extent refs 1 gen 2408131 flags 1 > > > [26062.127052] ref#0: shared data backref parent 509278633984 count 1 > > > [26062.127055] item 53 key (458661167104 168 24576) itemoff 12759 itemsize 183 > > > [26062.127056] extent refs 11 gen 1800715 flags 1 > > > [26062.127058] ref#0: extent data backref root 456 objectid 56413614 offset 36765696 count 1 > > > [26062.127059] ref#1: shared data backref parent 508152889344 count 1 > > > [26062.127061] ref#2: shared data backref parent 508031139840 count 1 > > > [26062.127062] ref#3: shared data backref parent 493563478016 count 1 > > > [26062.127065] ref#4: shared data backref parent 493539983360 count 1 > > > [26062.127066] ref#5: shared data backref parent 492674990080 count 1 > > > [26062.127068] ref#6: shared data backref parent 475132166144 count 1 > > > [26062.127070] ref#7: shared data backref parent 471032348672 count 1 > > > [26062.127071] ref#8: shared data backref parent 436111769600 count 1 > > > [26062.127073] ref#9: shared data backref parent 883359744 count 1 > > > [26062.127074] ref#10: shared data backref parent 85082112 count 1 > > > [26062.127077] item 54 key (458661191680 168 524288) itemoff 12667 itemsize 92 > > > [26062.127078] extent refs 4 gen 2419085 flags 1 > > > [26062.127079] ref#0: extent data backref root 456 objectid 75099153 offset 3145728 count 1 > > > [26062.127085] ref#1: shared data backref parent 494075035648 count 1 > > > [26062.127088] ref#2: shared data backref parent 470850240512 count 1 > > > [26062.127089] ref#3: shared data backref parent 437034745856 count 1 > > > [26062.127093] item 55 key (458662014976 168 524288) itemoff 12601 itemsize 66 > > > [26062.127095] extent refs 2 gen 2421893 flags 1 > > > [26062.127096] ref#0: extent data backref root 456 objectid 75220094 offset 1048576 count 1 > > > [26062.127099] ref#1: shared data backref parent 493703069696 count 1 > > > [26062.127102] item 56 key (458663206912 168 786432) itemoff 12548 itemsize 53 > > > [26062.127104] extent refs 1 gen 1444854 flags 1 > > > [26062.127105] ref#0: extent data backref root 456 objectid 50397303 offset 0 count 1 > > > [26062.127109] item 57 key (458664165376 168 77824) itemoff 12482 itemsize 66 > > > [26062.127111] extent refs 2 gen 2368700 flags 1 > > > [26062.127112] ref#0: extent data backref root 456 objectid 72993362 offset 262144 count 1 > > > [26062.127114] ref#1: shared data backref parent 494105804800 count 1 > > > [26062.127118] item 58 key (458664243200 168 1159168) itemoff 12445 itemsize 37 > > > [26062.127120] extent refs 1 gen 1444854 flags 1 > > > [26062.127121] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127125] item 59 key (458665431040 168 753664) itemoff 12408 itemsize 37 > > > [26062.127127] extent refs 1 gen 1444854 flags 1 > > > [26062.127128] ref#0: shared data backref parent 214433792 count 1 > > > [26062.127131] item 60 key (458666274816 168 393216) itemoff 12371 itemsize 37 > > > [26062.127132] extent refs 1 gen 1444854 flags 1 > > > [26062.127133] ref#0: shared data backref parent 214908928 count 1 > > > [26062.127136] item 61 key (458667044864 168 24576) itemoff 12292 itemsize 79 > > > [26062.127138] extent refs 3 gen 1800750 flags 1 > > > [26062.127139] ref#0: extent data backref root 456 objectid 56413614 offset 709386240 count 1 > > > [26062.127141] ref#1: shared data backref parent 509783556096 count 1 > > > [26062.127143] ref#2: shared data backref parent 508544974848 count 1 > > > [26062.127147] item 62 key (458667069440 168 69632) itemoff 12226 itemsize 66 > > > [26062.127148] extent refs 2 gen 2368707 flags 1 > > > [26062.127149] ref#0: extent data backref root 456 objectid 72925787 offset 3932160 count 1 > > > [26062.127151] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127153] item 63 key (458667143168 168 155648) itemoff 12043 itemsize 183 > > > [26062.127155] extent refs 11 gen 1800674 flags 1 > > > [26062.127156] ref#0: extent data backref root 456 objectid 56413614 offset 35766272 count 1 > > > [26062.127157] ref#1: shared data backref parent 508152905728 count 1 > > > [26062.127159] ref#2: shared data backref parent 508031057920 count 1 > > > [26062.127160] ref#3: shared data backref parent 493563166720 count 1 > > > [26062.127161] ref#4: shared data backref parent 493539934208 count 1 > > > [26062.127163] ref#5: shared data backref parent 492674908160 count 1 > > > [26062.127164] ref#6: shared data backref parent 475884208128 count 1 > > > [26062.127165] ref#7: shared data backref parent 471032266752 count 1 > > > [26062.127167] ref#8: shared data backref parent 436111720448 count 1 > > > [26062.127168] ref#9: shared data backref parent 883179520 count 1 > > > [26062.127170] ref#10: shared data backref parent 83017728 count 1 > > > [26062.127172] item 64 key (458667323392 168 393216) itemoff 12006 itemsize 37 > > > [26062.127173] extent refs 1 gen 1444854 flags 1 > > > [26062.127174] ref#0: shared data backref parent 214908928 count 1 > > > [26062.127176] item 65 key (458668109824 168 393216) itemoff 11940 itemsize 66 > > > [26062.127177] extent refs 2 gen 2381729 flags 1 > > > [26062.127178] ref#0: extent data backref root 456 objectid 73362062 offset 131072 count 1 > > > [26062.127180] ref#1: shared data backref parent 437808545792 count 1 > > > [26062.127182] item 66 key (458668605440 168 73728) itemoff 11874 itemsize 66 > > > [26062.127183] extent refs 2 gen 2369098 flags 1 > > > [26062.127184] ref#0: extent data backref root 456 objectid 72925787 offset 4997120 count 1 > > > [26062.127186] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127188] item 67 key (458668683264 168 77824) itemoff 11808 itemsize 66 > > > [26062.127189] extent refs 2 gen 2368700 flags 1 > > > [26062.127190] ref#0: extent data backref root 456 objectid 72993362 offset 393216 count 1 > > > [26062.127191] ref#1: shared data backref parent 494105804800 count 1 > > > [26062.127193] item 68 key (458668761088 168 1167360) itemoff 11771 itemsize 37 > > > [26062.127195] extent refs 1 gen 1444854 flags 1 > > > [26062.127195] ref#0: shared data backref parent 102547456 count 1 > > > [26062.127197] item 69 key (458670264320 168 24576) itemoff 11692 itemsize 79 > > > [26062.127198] extent refs 3 gen 1800750 flags 1 > > > [26062.127199] ref#0: extent data backref root 456 objectid 56413614 offset 717283328 count 1 > > > [26062.127201] ref#1: shared data backref parent 509783408640 count 1 > > > [26062.127202] ref#2: shared data backref parent 508543451136 count 1 > > > [26062.127204] item 70 key (458670465024 168 1814528) itemoff 11655 itemsize 37 > > > [26062.127205] extent refs 1 gen 1444854 flags 1 > > > [26062.127206] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127208] item 71 key (458672279552 168 2256896) itemoff 11618 itemsize 37 > > > [26062.127209] extent refs 1 gen 1444854 flags 1 > > > [26062.127210] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127213] item 72 key (458674536448 168 1024000) itemoff 11565 itemsize 53 > > > [26062.127214] extent refs 1 gen 1444854 flags 1 > > > [26062.127215] ref#0: extent data backref root 456 objectid 50397314 offset 0 count 1 > > > [26062.127217] item 73 key (458675560448 168 524288) itemoff 11473 itemsize 92 > > > [26062.127219] extent refs 4 gen 2419085 flags 1 > > > [26062.127220] ref#0: extent data backref root 456 objectid 75099153 offset 3670016 count 1 > > > [26062.127222] ref#1: shared data backref parent 494075035648 count 1 > > > [26062.127223] ref#2: shared data backref parent 470850240512 count 1 > > > [26062.127225] ref#3: shared data backref parent 437034745856 count 1 > > > [26062.127226] item 74 key (458677182464 168 24576) itemoff 11394 itemsize 79 > > > [26062.127228] extent refs 3 gen 1800750 flags 1 > > > [26062.127229] ref#0: extent data backref root 456 objectid 56413614 offset 717316096 count 1 > > > [26062.127230] ref#1: shared data backref parent 509783408640 count 1 > > > [26062.127232] ref#2: shared data backref parent 508543451136 count 1 > > > [26062.127234] item 75 key (458678693888 168 69632) itemoff 11328 itemsize 66 > > > [26062.127235] extent refs 2 gen 2368731 flags 1 > > > [26062.127236] ref#0: extent data backref root 456 objectid 72925787 offset 3997696 count 1 > > > [26062.127237] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127239] item 76 key (458681303040 168 1789952) itemoff 11291 itemsize 37 > > > [26062.127241] extent refs 1 gen 1444854 flags 1 > > > [26062.127241] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127243] item 77 key (458683092992 168 1941504) itemoff 11254 itemsize 37 > > > [26062.127244] extent refs 1 gen 1444854 flags 1 > > > [26062.127245] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127247] item 78 key (458685034496 168 1355776) itemoff 11217 itemsize 37 > > > [26062.127249] extent refs 1 gen 1444854 flags 1 > > > [26062.127249] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127252] item 79 key (458686390272 168 1409024) itemoff 11180 itemsize 37 > > > [26062.127253] extent refs 1 gen 1444854 flags 1 > > > [26062.127254] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127257] item 80 key (458687799296 168 1748992) itemoff 11143 itemsize 37 > > > [26062.127258] extent refs 1 gen 1444854 flags 1 > > > [26062.127259] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127262] item 81 key (458690596864 168 393216) itemoff 11090 itemsize 53 > > > [26062.127263] extent refs 1 gen 2381741 flags 1 > > > [26062.127264] ref#0: extent data backref root 456 objectid 73362154 offset 131072 count 1 > > > [26062.127266] item 82 key (458691379200 168 69632) itemoff 11024 itemsize 66 > > > [26062.127268] extent refs 2 gen 2368882 flags 1 > > > [26062.127269] ref#0: extent data backref root 456 objectid 72925787 offset 4390912 count 1 > > > [26062.127270] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127272] item 83 key (458692284416 168 73728) itemoff 10958 itemsize 66 > > > [26062.127274] extent refs 2 gen 2368909 flags 1 > > > [26062.127275] ref#0: extent data backref root 456 objectid 72925787 offset 4456448 count 1 > > > [26062.127277] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127279] item 84 key (458695856128 168 69632) itemoff 10892 itemsize 66 > > > [26062.127280] extent refs 2 gen 2369146 flags 1 > > > [26062.127281] ref#0: extent data backref root 456 objectid 72925787 offset 5128192 count 1 > > > [26062.127283] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127285] item 85 key (458699677696 168 53248) itemoff 10855 itemsize 37 > > > [26062.127286] extent refs 1 gen 2369353 flags 1 > > > [26062.127287] ref#0: shared data backref parent 510167498752 count 1 > > > [26062.127289] item 86 key (458699730944 168 8192) itemoff 10776 itemsize 79 > > > [26062.127294] extent refs 3 gen 1800733 flags 1 > > > [26062.127294] ref#0: extent data backref root 456 objectid 56413614 offset 673275904 count 1 > > > [26062.127296] ref#1: shared data backref parent 509783752704 count 1 > > > [26062.127298] ref#2: shared data backref parent 509351313408 count 1 > > > [26062.127299] item 87 key (458699747328 168 155648) itemoff 10593 itemsize 183 > > > [26062.127303] extent refs 22 gen 1800674 flags 1 > > > [26062.127304] ref#0: extent data backref root 456 objectid 56413614 offset 36257792 count 2 > > > [26062.127306] ref#1: shared data backref parent 508152889344 count 2 > > > [26062.127307] ref#2: shared data backref parent 508031139840 count 2 > > > [26062.127308] ref#3: shared data backref parent 493563478016 count 2 > > > [26062.127310] ref#4: shared data backref parent 493539983360 count 2 > > > [26062.127311] ref#5: shared data backref parent 492674990080 count 2 > > > [26062.127312] ref#6: shared data backref parent 475132166144 count 2 > > > [26062.127314] ref#7: shared data backref parent 471032348672 count 2 > > > [26062.127315] ref#8: shared data backref parent 436111769600 count 2 > > > [26062.127317] ref#9: shared data backref parent 883359744 count 2 > > > [26062.127318] ref#10: shared data backref parent 85082112 count 2 > > > [26062.127320] item 88 key (458700173312 168 180224) itemoff 10410 itemsize 183 > > > [26062.127321] extent refs 33 gen 1800674 flags 1 > > > [26062.127322] ref#0: extent data backref root 456 objectid 56413614 offset 36610048 count 3 > > > [26062.127324] ref#1: shared data backref parent 508152889344 count 3 > > > [26062.127325] ref#2: shared data backref parent 508031139840 count 3 > > > [26062.127327] ref#3: shared data backref parent 493563478016 count 3 > > > [26062.127328] ref#4: shared data backref parent 493539983360 count 3 > > > [26062.127329] ref#5: shared data backref parent 492674990080 count 3 > > > [26062.127331] ref#6: shared data backref parent 475132166144 count 3 > > > [26062.127333] ref#7: shared data backref parent 471032348672 count 3 > > > [26062.127334] ref#8: shared data backref parent 436111769600 count 3 > > > [26062.127335] ref#9: shared data backref parent 883359744 count 3 > > > [26062.127337] ref#10: shared data backref parent 85082112 count 3 > > > [26062.127338] item 89 key (458700353536 168 524288) itemoff 10318 itemsize 92 > > > [26062.127340] extent refs 4 gen 2419085 flags 1 > > > [26062.127341] ref#0: extent data backref root 456 objectid 75099153 offset 4194304 count 1 > > > [26062.127342] ref#1: shared data backref parent 494075035648 count 1 > > > [26062.127344] ref#2: shared data backref parent 470850240512 count 1 > > > [26062.127345] ref#3: shared data backref parent 437034745856 count 1 > > > [26062.127347] item 90 key (458701094912 168 69632) itemoff 10252 itemsize 66 > > > [26062.127348] extent refs 2 gen 2368932 flags 1 > > > [26062.127349] ref#0: extent data backref root 456 objectid 72925787 offset 4526080 count 1 > > > [26062.127351] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127353] item 91 key (458701164544 168 57344) itemoff 10215 itemsize 37 > > > [26062.127354] extent refs 1 gen 2369353 flags 1 > > > [26062.127355] ref#0: shared data backref parent 510165155840 count 1 > > > [26062.127356] item 92 key (458701221888 168 73728) itemoff 10149 itemsize 66 > > > [26062.127358] extent refs 2 gen 2369168 flags 1 > > > [26062.127359] ref#0: extent data backref root 456 objectid 72925787 offset 5193728 count 1 > > > [26062.127360] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127362] item 93 key (458701295616 168 61440) itemoff 10096 itemsize 53 > > > [26062.127364] extent refs 1 gen 2369305 flags 1 > > > [26062.127365] ref#0: extent data backref root 456 objectid 73017376 offset 0 count 1 > > > [26062.127367] item 94 key (458702897152 168 524288) itemoff 10004 itemsize 92 > > > [26062.127369] extent refs 4 gen 2419085 flags 1 > > > [26062.127370] ref#0: extent data backref root 456 objectid 75099153 offset 4718592 count 1 > > > [26062.127372] ref#1: shared data backref parent 494075035648 count 1 > > > [26062.127374] ref#2: shared data backref parent 470850240512 count 1 > > > [26062.127375] ref#3: shared data backref parent 437034745856 count 1 > > > [26062.127377] item 95 key (458703454208 168 69632) itemoff 9938 itemsize 66 > > > [26062.127378] extent refs 2 gen 2368955 flags 1 > > > [26062.127379] ref#0: extent data backref root 456 objectid 72925787 offset 4591616 count 1 > > > [26062.127382] ref#1: shared data backref parent 437615230976 count 1 > > > [26062.127384] item 96 key (458703523840 168 294912) itemoff 9901 itemsize 37 > > > [26062.127385] extent refs 1 gen 2369352 flags 1 > > > [26062.127386] ref#0: shared data backref parent 510178295808 count 1 > > > [26062.127388] item 97 key (458703818752 168 81920) itemoff 9864 itemsize 37 > > > [26062.127390] extent refs 1 gen 2369352 flags 1 > > > [26062.127391] ref#0: shared data backref parent 510058790912 count 1 > > > [26062.127393] item 98 key (458703900672 168 131072) itemoff 9827 itemsize 37 > > > [26062.127394] extent refs 1 gen 2369352 flags 1 > > > [26062.127395] ref#0: shared data backref parent 510059184128 count 1 > > > [26062.127398] item 99 key (458704031744 168 524288) itemoff 9774 itemsize 53 > > > [26062.127399] extent refs 1 gen 2378552 flags 1 > > > [26062.127400] ref#0: extent data backref root 456 objectid 73276414 offset 11010048 count 1 > > > [26062.127403] item 100 key (458704556032 168 524288) itemoff 9721 itemsize 53 > > > [26062.127405] extent refs 1 gen 2378552 flags 1 > > > [26062.127406] ref#0: extent data backref root 456 objectid 73276414 offset 11534336 count 1 > > > [26062.127408] item 101 key (458705080320 168 393216) itemoff 9668 itemsize 53 > > > [26062.127409] extent refs 1 gen 2381741 flags 1 > > > [26062.127410] ref#0: extent data backref root 456 objectid 73362155 offset 131072 count 1 > > > [26062.127413] item 102 key (458707353600 168 81920) itemoff 9602 itemsize 66 > > > [26062.127414] extent refs 2 gen 2368700 flags 1 > > > [26062.127415] ref#0: extent data backref root 456 objectid 72993362 offset 131072 count 1 > > > [26062.127416] ref#1: shared data backref parent 494105804800 count 1 > > > [26062.127419] item 103 key (458707435520 168 81920) itemoff 9536 itemsize 66 > > > [26062.127420] extent refs 2 gen 2368700 flags 1 > > > [26062.127421] ref#0: extent data backref root 456 objectid 72993362 offset 524288 count 1 > > > [26062.127423] ref#1: shared data backref parent 494105804800 count 1 > > > [26062.127425] item 104 key (458707517440 168 81920) itemoff 9470 itemsize 66 > > > [26062.127426] extent refs 2 gen 2368700 flags 1 > > > [26062.127428] ref#0: extent data backref root 456 objectid 72993362 offset 655360 count 1 > > > [26062.127430] ref#1: shared data backref parent 494105804800 count 1 > > > [26062.127432] item 105 key (458707738624 168 57344) itemoff 9287 itemsize 183 > > > [26062.127434] extent refs 11 gen 1800699 flags 1 > > > [26062.127435] ref#0: extent data backref root 456 objectid 56413614 offset 39002112 count 1 > > > [26062.127437] ref#1: shared data backref parent 508152987648 count 1 > > > [26062.127439] ref#2: shared data backref parent 508031172608 count 1 > > > [26062.127440] ref#3: shared data backref parent 493563822080 count 1 > > > [26062.127441] ref#4: shared data backref parent 493540130816 count 1 > > > [26062.127443] ref#5: shared data backref parent 492675170304 count 1 > > > [26062.127444] ref#6: shared data backref parent 475884584960 count 1 > > > [26062.127446] ref#7: shared data backref parent 471032496128 count 1 > > > [26062.127447] ref#8: shared data backref parent 436111884288 count 1 > > > [26062.127448] ref#9: shared data backref parent 883703808 count 1 > > > [26062.127450] ref#10: shared data backref parent 85671936 count 1 > > > [26062.127452] item 106 key (458707800064 168 393216) itemoff 9250 itemsize 37 > > > [26062.127453] extent refs 1 gen 1444854 flags 1 > > > [26062.127454] ref#0: shared data backref parent 214908928 count 1 > > > [26062.127457] item 107 key (458708193280 168 11943936) itemoff 9213 itemsize 37 > > > [26062.127458] extent refs 1 gen 2426255 flags 1 > > > [26062.127459] ref#0: shared data backref parent 436271005696 count 1 > > > [26062.127461] item 108 key (458725122048 168 131072) itemoff 9030 itemsize 183 > > > [26062.127463] extent refs 22 gen 1800674 flags 1 > > > [26062.127464] ref#0: extent data backref root 456 objectid 56413614 offset 38928384 count 2 > > > [26062.127465] ref#1: shared data backref parent 508152987648 count 2 > > > [26062.127467] ref#2: shared data backref parent 508031172608 count 2 > > > [26062.127468] ref#3: shared data backref parent 493563822080 count 2 > > > [26062.127470] ref#4: shared data backref parent 493540130816 count 2 > > > [26062.127472] ref#5: shared data backref parent 492675170304 count 2 > > > [26062.127473] ref#6: shared data backref parent 475884584960 count 2 > > > [26062.127475] ref#7: shared data backref parent 471032496128 count 2 > > > [26062.127476] ref#8: shared data backref parent 436111884288 count 2 > > > [26062.127478] ref#9: shared data backref parent 883703808 count 2 > > > [26062.127480] ref#10: shared data backref parent 85671936 count 2 > > > [26062.127483] item 109 key (458725285888 168 1069056) itemoff 8993 itemsize 37 > > > [26062.127486] extent refs 1 gen 1444854 flags 1 > > > [26062.127488] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127491] item 110 key (458726379520 168 3158016) itemoff 8927 itemsize 66 > > > [26062.127494] extent refs 2 gen 1400856 flags 1 > > > [26062.127496] ref#0: extent data backref root 456 objectid 49348086 offset 0 count 1 > > > [26062.127498] ref#1: shared data backref parent 470954196992 count 1 > > > [26062.127501] item 111 key (458729537536 168 2166784) itemoff 8890 itemsize 37 > > > [26062.127503] extent refs 1 gen 1444854 flags 1 > > > [26062.127504] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127507] item 112 key (458731704320 168 393216) itemoff 8837 itemsize 53 > > > [26062.127510] extent refs 1 gen 2381741 flags 1 > > > [26062.127512] ref#0: extent data backref root 456 objectid 73362156 offset 131072 count 1 > > > [26062.127518] item 113 key (458732539904 168 77824) itemoff 8771 itemsize 66 > > > [26062.127520] extent refs 2 gen 2368701 flags 1 > > > [26062.127521] ref#0: extent data backref root 456 objectid 72993032 offset 0 count 1 > > > [26062.127523] ref#1: shared data backref parent 494107131904 count 1 > > > [26062.127526] item 114 key (458732752896 168 196608) itemoff 8588 itemsize 183 > > > [26062.127527] extent refs 44 gen 1800674 flags 1 > > > [26062.127528] ref#0: extent data backref root 456 objectid 56413614 offset 37789696 count 4 > > > [26062.127530] ref#1: shared data backref parent 508152987648 count 4 > > > [26062.127532] ref#2: shared data backref parent 508031172608 count 4 > > > [26062.127533] ref#3: shared data backref parent 493563822080 count 4 > > > [26062.127535] ref#4: shared data backref parent 493540130816 count 4 > > > [26062.127537] ref#5: shared data backref parent 492675170304 count 4 > > > [26062.127539] ref#6: shared data backref parent 475884584960 count 4 > > > [26062.127540] ref#7: shared data backref parent 471032496128 count 4 > > > [26062.127542] ref#8: shared data backref parent 436111884288 count 4 > > > [26062.127543] ref#9: shared data backref parent 883703808 count 4 > > > [26062.127545] ref#10: shared data backref parent 85671936 count 4 > > > [26062.127547] item 115 key (458732949504 168 24576) itemoff 8509 itemsize 79 > > > [26062.127549] extent refs 3 gen 1800750 flags 1 > > > [26062.127552] ref#0: extent data backref root 456 objectid 56413614 offset 738836480 count 1 > > > [26062.127554] ref#1: shared data backref parent 509783130112 count 1 > > > [26062.127557] ref#2: shared data backref parent 508544876544 count 1 > > > [26062.127560] item 116 key (458732978176 168 1351680) itemoff 8472 itemsize 37 > > > [26062.127561] extent refs 1 gen 1444854 flags 1 > > > [26062.127562] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127565] item 117 key (458734329856 168 393216) itemoff 8435 itemsize 37 > > > [26062.127566] extent refs 1 gen 1444854 flags 1 > > > [26062.127568] ref#0: shared data backref parent 214908928 count 1 > > > [26062.127570] item 118 key (458801938432 168 1785856) itemoff 8398 itemsize 37 > > > [26062.127572] extent refs 1 gen 1444854 flags 1 > > > [26062.127573] ref#0: shared data backref parent 93732864 count 1 > > > [26062.127577] item 119 key (458801938432 192 1073741824) itemoff 8374 itemsize 24 > > > [26062.127580] block group used 834060288 chunk_objectid 256 flags 1 > > > [26062.127581] item 120 key (458803724288 168 1409024) itemoff 8337 itemsize 37 > > > [26062.127583] extent refs 1 gen 1444854 flags 1 > > > [26062.127584] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127586] item 121 key (458805133312 168 1183744) itemoff 8300 itemsize 37 > > > [26062.127587] extent refs 1 gen 1444854 flags 1 > > > [26062.127588] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127590] item 122 key (458806317056 168 1323008) itemoff 8263 itemsize 37 > > > [26062.127591] extent refs 1 gen 1444854 flags 1 > > > [26062.127592] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127594] item 123 key (458807640064 168 1589248) itemoff 8226 itemsize 37 > > > [26062.127595] extent refs 1 gen 1444854 flags 1 > > > [26062.127596] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127598] item 124 key (458809229312 168 2727936) itemoff 8189 itemsize 37 > > > [26062.127599] extent refs 1 gen 1444854 flags 1 > > > [26062.127600] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127602] item 125 key (458811957248 168 1978368) itemoff 8152 itemsize 37 > > > [26062.127603] extent refs 1 gen 1444854 flags 1 > > > [26062.127604] ref#0: shared data backref parent 167198720 count 1 > > > [26062.127606] item 126 key (458813943808 168 8192) itemoff 7969 itemsize 183 > > > [26062.127607] extent refs 11 gen 2409230 flags 1 > > > [26062.127608] ref#0: extent data backref root 456 objectid 56413614 offset 58589184 count 1 > > > [26062.127610] ref#1: shared data backref parent 508153675776 count 1 > > > [26062.127611] ref#2: shared data backref parent 493566640128 count 1 > > > [26062.127613] ref#3: shared data backref parent 493540999168 count 1 > > > [26062.127614] ref#4: shared data backref parent 492676497408 count 1 > > > [26062.127616] ref#5: shared data backref parent 475884634112 count 1 > > > [26062.127617] ref#6: shared data backref parent 475691106304 count 1 > > > [26062.127619] ref#7: shared data backref parent 471034494976 count 1 > > > [26062.127620] ref#8: shared data backref parent 436113883136 count 1 > > > [26062.127622] ref#9: shared data backref parent 886407168 count 1 > > > [26062.127623] ref#10: shared data backref parent 89440256 count 1 > > > [26062.127625] item 127 key (458813960192 168 8192) itemoff 7799 itemsize 170 > > > [26062.127626] extent refs 10 gen 2413323 flags 1 > > > [26062.127629] ref#0: extent data backref root 456 objectid 56413614 offset 58580992 count 1 > > > [26062.127631] ref#1: shared data backref parent 508153675776 count 1 > > > [26062.127633] ref#2: shared data backref parent 493566640128 count 1 > > > [26062.127635] ref#3: shared data backref parent 493540999168 count 1 > > > [26062.127636] ref#4: shared data backref parent 492676497408 count 1 > > > [26062.127638] ref#5: shared data backref parent 475691106304 count 1 > > > [26062.127640] ref#6: shared data backref parent 471034494976 count 1 > > > [26062.127642] ref#7: shared data backref parent 436113883136 count 1 > > > [26062.127643] ref#8: shared data backref parent 886407168 count 1 > > > [26062.127645] ref#9: shared data backref parent 89440256 count 1 > > > [26062.127647] item 128 key (458814124032 168 8192) itemoff 7762 itemsize 37 > > > [26062.127648] extent refs 1 gen 2407793 flags 1 > > > [26062.127649] ref#0: shared data backref parent 475884634112 count 1 > > > [26062.127653] item 129 key (458814738432 168 8192) itemoff 7579 itemsize 183 > > > [26062.127655] extent refs 11 gen 1800729 flags 1 > > > [26062.127656] ref#0: extent data backref root 456 objectid 56413614 offset 40116224 count 1 > > > [26062.127658] ref#1: shared data backref parent 508153069568 count 1 > > > [26062.127659] ref#2: shared data backref parent 508031270912 count 1 > > > [26062.127661] ref#3: shared data backref parent 493564215296 count 1 > > > [26062.127663] ref#4: shared data backref parent 493540098048 count 1 > > > [26062.127664] ref#5: shared data backref parent 492676038656 count 1 > > > [26062.127666] ref#6: shared data backref parent 475884142592 count 1 > > > [26062.127668] ref#7: shared data backref parent 471032840192 count 1 > > > [26062.127669] ref#8: shared data backref parent 436112146432 count 1 > > > [26062.127671] ref#9: shared data backref parent 883752960 count 1 > > > [26062.127674] ref#10: shared data backref parent 86654976 count 1 > > > [26062.127678] BTRFS error (device dm-2): unable to find ref byte nr 458640384000 parent 0 root 456 owner 73020573 offset 17039360 > > > [26062.127680] ------------[ cut here ]------------ > > > [26062.127682] BTRFS: Transaction aborted (error -2) > > > [26062.127696] WARNING: CPU: 7 PID: 12394 at fs/btrfs/extent-tree.c:7106 __btrfs_free_extent+0x1e0/0x921 > > > [26062.127698] Modules linked in: msr ccm ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_log_ipv4 nf_log_common xt_LOG iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter rfcomm ax25 bnep pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc nls_utf8 nls_cp437 vfat fat cuse ecryptfs bbswitch(OE) configs input_polldev loop firewire_sbp2 firewire_core crc_itu_t ppdev parport_pc lp parport uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media btusb btrtl btbcm btintel bluetooth ecdh_generic hid_generic usbhid hid joydev arc4 coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic intel_wmi_thunderbolt iTCO_wdt wmi_bmof mei_hdcp iTCO_vendor_support rtsx_pci_sdmmc s nd_hda_intel > > > [26062.127733] snd_hda_codec iwlmvm crct10dif_pclmul snd_hda_core crc32_pclmul mac80211 snd_hwdep thinkpad_acpi snd_pcm ghash_clmulni_intel nvram ledtrig_audio intel_cstate deflate snd_seq efi_pstore iwlwifi snd_seq_device snd_timer intel_rapl_perf psmouse pcspkr efivars wmi hwmon snd ac battery cfg80211 mei_me soundcore xhci_pci xhci_hcd rtsx_pci i2c_i801 rfkill sg nvidiafb intel_pch_thermal usbcore vgastate fb_ddc pcc_cpufreq sata_sil24 r8169 libphy mii fuse fan raid456 multipath mmc_block mmc_core dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_crypt dm_mod async_raid6_recov async_pq async_xor async_memcpy async_tx blowfish_x86_64 blowfish_common crc32c_intel bcache crc64 aesni_intel input_leds i915 aes_x86_64 crypto_simd cryptd ptp glue_helper serio_raw pps_core thermal evdev [last unloaded: e1000e] > > > [26062.127793] CPU: 7 PID: 12394 Comm: btrfs-transacti Tainted: G W OE 5.1.21-amd64-preempt-sysrq-20190816 #5 > > > [26062.127795] Hardware name: LENOVO 20ERCTO1WW/20ERCTO1WW, BIOS N1DET95W (2.21 ) 12/13/2017 > > > [26062.127799] RIP: 0010:__btrfs_free_extent+0x1e0/0x921 > > > [26062.127801] Code: e8 82 d2 fe ff 48 8b 43 50 f0 48 0f ba a8 30 17 00 00 02 0f 92 c0 5a 84 c0 75 11 44 89 ee 48 c7 c7 03 d8 f4 89 e8 33 f1 d8 ff <0f> 0b b9 fe ff ff ff ba c2 1b 00 00 48 c7 c6 e0 c7 c3 89 48 89 df > > > [26062.127803] RSP: 0018:ffffb2d9c46e7c88 EFLAGS: 00010282 > > > [26062.127805] RAX: 0000000000000000 RBX: ffff9abca20884e0 RCX: 0000000000000007 > > > [26062.127807] RDX: 0000000000000000 RSI: ffffb2d9c46e7b64 RDI: ffff9abccf5d6550 > > > [26062.127809] RBP: ffff9ab5a4545460 R08: 0000000000000001 R09: ffffffff8a80c7af > > > [26062.127811] R10: 0000000005f5e100 R11: ffffb2d9c46e7b27 R12: 0000000000000169 > > > [26062.127813] R13: 00000000fffffffe R14: 0000000001040000 R15: 0000006ac918e000 > > > [26062.127815] FS: 0000000000000000(0000) GS:ffff9abccf5c0000(0000) knlGS:0000000000000000 > > > [26062.127817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [26062.127819] CR2: 0000199a9fb4d000 CR3: 000000016c20e006 CR4: 00000000003606e0 > > > [26062.127821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > [26062.127822] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > [26062.127823] Call Trace: > > > [26062.127832] __btrfs_run_delayed_refs+0x750/0xc36 > > > [26062.127837] ? __switch_to_asm+0x41/0x70 > > > [26062.127839] ? __switch_to_asm+0x35/0x70 > > > [26062.127841] ? __switch_to_asm+0x41/0x70 > > > [26062.127848] ? __switch_to+0x13d/0x3d5 > > > [26062.127852] btrfs_run_delayed_refs+0x5d/0x132 > > > [26062.127855] btrfs_commit_transaction+0x55/0x7c8 > > > [26062.127858] ? start_transaction+0x347/0x3cb > > > [26062.127862] transaction_kthread+0xc9/0x135 > > > [26062.127865] ? btrfs_cleanup_transaction+0x403/0x403 > > > [26062.127869] kthread+0xeb/0xf0 > > > [26062.127872] ? kthread_create_worker_on_cpu+0x65/0x65 > > > [26062.127875] ret_from_fork+0x35/0x40 > > > [26062.127879] ---[ end trace 4c1a6b3749a2f651 ]--- > > > [26062.127914] BTRFS: error (device dm-2) in __btrfs_free_extent:7106: errno=-2 No such entry > > > [26062.127917] BTRFS info (device dm-2): forced readonly > > > [26062.127921] BTRFS: error (device dm-2) in btrfs_run_delayed_refs:3008: errno=-2 No such entry -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' @ 2020-05-24 21:30 ` Marc MERLIN 2020-05-25 3:01 ` Marc MERLIN 2020-05-25 16:37 ` Chris Murphy 0 siblings, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2020-05-24 21:30 UTC (permalink / raw) To: Su Yue, linux-btrfs, Qu Wenruo, Josef Bacik; +Cc: Su Yue My data is fine, it's double backed up and the filesystem is still mountable without issues. But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing' Any idea if I can fix them without deleting the filesystem? Current state: saruman:~# btrfs check --repair /dev/mapper/cr enabling repair mode Starting repair. Opening filesystem to check... Checking filesystem on /dev/mapper/cr UUID: 4cb82363-e833-444e-b23e-1597a14a8aab [1/7] checking root items Fixed 0 roots. [2/7] checking extents No device size related problem found [3/7] checking free space cache cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots root 356 inode 14058737 errors 1000, some csum missing root 197362 inode 14058737 errors 1000, some csum missing root 199551 inode 14058737 errors 1000, some csum missing (snip) root 204167 inode 14058737 errors 1000, some csum missing root 204168 inode 14058737 errors 1000, some csum missing ERROR: errors found in fs roots found 112392159232 bytes used, error(s) found total csum bytes: 105763716 total tree bytes: 4036018176 total fs tree bytes: 3666313216 total extent tree bytes: 214728704 btree space waste bytes: 875560453 file data blocks allocated: 207903920128 referenced 206918037504 I can run multiple times, 'some csum missing' does not get fixed. Lowmem does not fix it either: saruman:~# btrfs check --mode=lowmem --repair /dev/mapper/cr Starting repair. WARNING: low-memory mode repair support is only partial Opening filesystem to check... Checking filesystem on /dev/mapper/cr UUID: 4cb82363-e833-444e-b23e-1597a14a8aab [1/7] checking root items Fixed 0 roots. [2/7] checking extents No device size related problem found [3/7] checking free space cache cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots ERROR: root 356 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192 ERROR: root 356 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192 Add a hole [0, 4096] in inode [33037178] Add a hole [0, 4096] in inode [33037180] Add a hole [0, 4096] in inode [33037181] Add a hole [0, 4096] in inode [33037188] Add a hole [0, 4096] in inode [33037189] Add a hole [0, 4096] in inode [33037191] ERROR: root 197362 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192 ERROR: root 197362 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192 Add a hole [0, 4096] in inode [33037178] Add a hole [0, 4096] in inode [33037180] (...) Add a hole [0, 4096] in inode [33037189] Add a hole [0, 4096] in inode [33037191] ERROR: errors found in fs roots found 112392294400 bytes used, error(s) found total csum bytes: 105763716 total tree bytes: 4035985408 total fs tree bytes: 3666313216 total extent tree bytes: 214728704 btree space waste bytes: 875550920 file data blocks allocated: 207903920128 referenced 206918037504 This originally started with: BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112 which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'. Checking filesystem on /dev/mapper/cr UUID: 4cb82363-e833-444e-b23e-1597a14a8aab [1/7] checking root items [2/7] checking extents data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00 incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0 backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0 backpointer mismatch on [2694234112 8192] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache [4/7] checking fs roots root 356 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 203332 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 203669 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204006 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204010 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204024 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204037 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204049 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204167 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 root 204168 inode 33037179 errors 100, file extent discount Found file extent holes: start: 0, len: 8192 ERROR: errors found in fs roots found 111202623488 bytes used, error(s) found total csum bytes: 105763728 total tree bytes: 2846048256 total fs tree bytes: 2485485568 total extent tree bytes: 205619200 btree space waste bytes: 668593131 file data blocks allocated: 149272375296 referenced 140124020736 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-24 21:30 ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN @ 2020-05-25 3:01 ` Marc MERLIN 2020-05-25 16:37 ` Chris Murphy 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2020-05-25 3:01 UTC (permalink / raw) To: Su Yue, linux-btrfs, Qu Wenruo, Josef Bacik; +Cc: Su Yue I forgot to add: I'll keep my system in the broken state for maybe a day or two at most, but given that this is my root filesystem and that I'm booted off a backup media, I can't stay in that state too long. This may not be nothing new and if it's already known that the way out is to wipe and restore backups, let me know. Thnaks, Marc On Sun, May 24, 2020 at 02:30:59PM -0700, Marc MERLIN wrote: > My data is fine, it's double backed up and the filesystem is still mountable without issues. > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing' > > Any idea if I can fix them without deleting the filesystem? > > Current state: > saruman:~# btrfs check --repair /dev/mapper/cr > enabling repair mode > Starting repair. > Opening filesystem to check... > Checking filesystem on /dev/mapper/cr > UUID: 4cb82363-e833-444e-b23e-1597a14a8aab > [1/7] checking root items > Fixed 0 roots. > [2/7] checking extents > No device size related problem found > [3/7] checking free space cache > cache and super generation don't match, space cache will be invalidated > [4/7] checking fs roots > root 356 inode 14058737 errors 1000, some csum missing > root 197362 inode 14058737 errors 1000, some csum missing > root 199551 inode 14058737 errors 1000, some csum missing > (snip) > root 204167 inode 14058737 errors 1000, some csum missing > root 204168 inode 14058737 errors 1000, some csum missing > ERROR: errors found in fs roots > found 112392159232 bytes used, error(s) found > total csum bytes: 105763716 > total tree bytes: 4036018176 > total fs tree bytes: 3666313216 > total extent tree bytes: 214728704 > btree space waste bytes: 875560453 > file data blocks allocated: 207903920128 > referenced 206918037504 > > I can run multiple times, 'some csum missing' does not get fixed. > > Lowmem does not fix it either: > saruman:~# btrfs check --mode=lowmem --repair /dev/mapper/cr > Starting repair. > WARNING: low-memory mode repair support is only partial > Opening filesystem to check... > Checking filesystem on /dev/mapper/cr > UUID: 4cb82363-e833-444e-b23e-1597a14a8aab > [1/7] checking root items > Fixed 0 roots. > [2/7] checking extents > No device size related problem found > [3/7] checking free space cache > cache and super generation don't match, space cache will be invalidated > [4/7] checking fs roots > ERROR: root 356 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192 > ERROR: root 356 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192 > Add a hole [0, 4096] in inode [33037178] > Add a hole [0, 4096] in inode [33037180] > Add a hole [0, 4096] in inode [33037181] > Add a hole [0, 4096] in inode [33037188] > Add a hole [0, 4096] in inode [33037189] > Add a hole [0, 4096] in inode [33037191] > ERROR: root 197362 EXTENT_DATA[14058737 0] csum missing, have: 0, expected: 8192 > ERROR: root 197362 EXTENT_DATA[14058737 0] compressed extent must have csum, but only 0 bytes have, expect 8192 > Add a hole [0, 4096] in inode [33037178] > Add a hole [0, 4096] in inode [33037180] > (...) > Add a hole [0, 4096] in inode [33037189] > Add a hole [0, 4096] in inode [33037191] > ERROR: errors found in fs roots > found 112392294400 bytes used, error(s) found > total csum bytes: 105763716 > total tree bytes: 4035985408 > total fs tree bytes: 3666313216 > total extent tree bytes: 214728704 > btree space waste bytes: 875550920 > file data blocks allocated: 207903920128 > referenced 206918037504 > > > This originally started with: > BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112 > > which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'. > Checking filesystem on /dev/mapper/cr > UUID: 4cb82363-e833-444e-b23e-1597a14a8aab > [1/7] checking root items > [2/7] checking extents > data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree > incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00 > incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0 > backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0 > backpointer mismatch on [2694234112 8192] > ERROR: errors found in extent allocation tree or chunk allocation > [3/7] checking free space cache > [4/7] checking fs roots > root 356 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 203332 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 203669 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204006 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204010 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204024 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204037 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204049 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204167 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 204168 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > ERROR: errors found in fs roots > found 111202623488 bytes used, error(s) found > total csum bytes: 105763728 > total tree bytes: 2846048256 > total fs tree bytes: 2485485568 > total extent tree bytes: 205619200 > btree space waste bytes: 668593131 > file data blocks allocated: 149272375296 > referenced 140124020736 > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-24 21:30 ` 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' Marc MERLIN 2020-05-25 3:01 ` Marc MERLIN @ 2020-05-25 16:37 ` Chris Murphy 2020-05-25 20:16 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Chris Murphy @ 2020-05-25 16:37 UTC (permalink / raw) To: Marc MERLIN; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote: > > My data is fine, it's double backed up and the filesystem is still mountable without issues. > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing' I'm not following the sequence of events. The send|receive failed? Did you try deleting the failed received snapshot? > which I was able to fix with the repair below, but now I'm stuck with the 'some csum missing'. > Checking filesystem on /dev/mapper/cr > UUID: 4cb82363-e833-444e-b23e-1597a14a8aab > [1/7] checking root items > [2/7] checking extents > data backref 2694234112 root 356 owner 14058737 offset 0 num_refs 0 not found in extent tree > incorrect local backref count on 2694234112 root 356 owner 14058737 offset 0 found 1 wanted 0 back 0x55e7383a3a00 > incorrect local backref count on 2694234112 root 2147484004 owner 14058737 offset 0 found 0 wanted 1 back 0x55e733d1c2b0 > backref disk bytenr does not match extent record, bytenr=2694234112, ref bytenr=0 > backpointer mismatch on [2694234112 8192] > ERROR: errors found in extent allocation tree or chunk allocation > [3/7] checking free space cache > [4/7] checking fs roots > root 356 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 > root 203332 inode 33037179 errors 100, file extent discount > Found file extent holes: > start: 0, len: 8192 Is no-holes enabled on either file system? -- Chris Murphy ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 16:37 ` Chris Murphy @ 2020-05-25 20:16 ` Marc MERLIN 2020-05-25 20:24 ` Chris Murphy 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2020-05-25 20:16 UTC (permalink / raw) To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 10:37:58AM -0600, Chris Murphy wrote: > On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote: > > > > My data is fine, it's double backed up and the filesystem is still mountable without issues. > > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing' > > I'm not following the sequence of events. The send|receive failed? Did > you try deleting the failed received snapshot? btrfs send failed because there was an error on the source filesystem. Then I found this in the logs BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112 then I ran the btrfs check repair, with and without lowmem. They fixed some things, but leave me > Is no-holes enabled on either file system? Not intentionally. How do I check? Either way, things happened, not exactly sure what, but I'm just trying to find out if check --repair can/should do a better job, and if I can give anything useful before wiping the filesystem and starting over. As of right now, I have: aruman:~# btrfs check /dev/mapper/cr Opening filesystem to check... Checking filesystem on /dev/mapper/cr UUID: 4cb82363-e833-444e-b23e-1597a14a8aab [1/7] checking root items [2/7] checking extents [3/7] checking free space cache there is no free space entry for 24565465088-24565481472 there is no free space entry for 24565465088-24717033472 cache appears valid but isn't 23643291648 there is no free space entry for 45118177280-45118193664 there is no free space entry for 45118177280-46191869952 cache appears valid but isn't 45118128128 wanted offset 1659663106048, found 1659663089664 wanted offset 1659663106048, found 1659663089664 cache appears valid but isn't 1659663024128 [4/7] checking fs roots root 356 inode 14058737 errors 1000, some csum missing root 197362 inode 14058737 errors 1000, some csum missing root 199551 inode 14058737 errors 1000, some csum missing root 201742 inode 14058737 errors 1000, some csum missing root 202370 inode 14058737 errors 1000, some csum missing root 202683 inode 14058737 errors 1000, some csum missing root 202999 inode 14058737 errors 1000, some csum missing root 203332 inode 14058737 errors 1000, some csum missing root 203669 inode 14058737 errors 1000, some csum missing root 204006 inode 14058737 errors 1000, some csum missing root 204010 inode 14058737 errors 1000, some csum missing root 204024 inode 14058737 errors 1000, some csum missing root 204037 inode 14058737 errors 1000, some csum missing root 204049 inode 14058737 errors 1000, some csum missing root 204063 inode 14058737 errors 1000, some csum missing root 204077 inode 14058737 errors 1000, some csum missing root 204091 inode 14058737 errors 1000, some csum missing root 204105 inode 14058737 errors 1000, some csum missing root 204119 inode 14058737 errors 1000, some csum missing root 204133 inode 14058737 errors 1000, some csum missing root 204146 inode 14058737 errors 1000, some csum missing root 204147 inode 14058737 errors 1000, some csum missing root 204159 inode 14058737 errors 1000, some csum missing root 204160 inode 14058737 errors 1000, some csum missing root 204161 inode 14058737 errors 1000, some csum missing root 204162 inode 14058737 errors 1000, some csum missing root 204163 inode 14058737 errors 1000, some csum missing root 204164 inode 14058737 errors 1000, some csum missing root 204165 inode 14058737 errors 1000, some csum missing root 204166 inode 14058737 errors 1000, some csum missing root 204167 inode 14058737 errors 1000, some csum missing root 204168 inode 14058737 errors 1000, some csum missing ERROR: errors found in fs roots found 112391372800 bytes used, error(s) found total csum bytes: 105763716 total tree bytes: 4036018176 total fs tree bytes: 3666313216 total extent tree bytes: 214728704 btree space waste bytes: 875571363 file data blocks allocated: 207903133696 referenced 20 Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 20:16 ` Marc MERLIN @ 2020-05-25 20:24 ` Chris Murphy 2020-05-25 20:39 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Chris Murphy @ 2020-05-25 20:24 UTC (permalink / raw) To: Marc MERLIN Cc: Chris Murphy, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 2:16 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, May 25, 2020 at 10:37:58AM -0600, Chris Murphy wrote: > > On Sun, May 24, 2020 at 3:31 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > My data is fine, it's double backed up and the filesystem is still mountable without issues. > > > But I had an error that broke btrfs send, and after fixing it with repair, I'm stuck with thses 'csum missing' > > > > I'm not following the sequence of events. The send|receive failed? Did > > you try deleting the failed received snapshot? > > btrfs send failed because there was an error on the source filesystem. > Then I found this in the logs > BTRFS error (device dm-0): did not find backref in send_root. inode=14058737, offset=0, disk_byte=2694234112 found extent=2694234112 > > then I ran the btrfs check repair, with and without lowmem. > They fixed some things, but leave me OK I didn't understand that the problem is with only the sending file system, not the receive file system. And also it sounds like the send did not cause the problem, but it's somehow a pre-existing problem that --repair isn't completely fixing up, or maybe is making different (or worse). So I guess the real question is what happened to this file system before the send, that got it into this weird state. > > > Is no-holes enabled on either file system? > > Not intentionally. How do I check? btrfs insp dump-s It's not yet the default and can't be inadvertently enabled so chances are it's the original holes implementation. -- Chris Murphy ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 20:24 ` Chris Murphy @ 2020-05-25 20:39 ` Marc MERLIN 2020-05-25 22:47 ` Chris Murphy 2020-07-07 3:55 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN 0 siblings, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2020-05-25 20:39 UTC (permalink / raw) To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote: > OK I didn't understand that the problem is with only the sending file > system, not the receive file system. And also it sounds like the send > did not cause the problem, but it's somehow a pre-existing problem > that --repair isn't completely fixing up, or maybe is making different > (or worse). Correct on all points. > So I guess the real question is what happened to this file system > before the send, that got it into this weird state. That too, but honestly there are a lot of variables, and it feels like a bit of wild goose chase. Basically it looked like the issues with the FS are pretty minor (I was able to cp -av the entire data without any file error), but that btrfs check --repair is unable to make it right, which will likely force me to wipe and restore. I know chedk is WIP, and that's why I'm providing feedback :) > > > Is no-holes enabled on either file system? > > > > Not intentionally. How do I check? > > btrfs insp dump-s > > It's not yet the default and can't be inadvertently enabled so chances > are it's the original holes implementation. The filesystem was definitely created a while ago (2-3 years?) I have been recently been playing with bees, but I'm reasonably sure I didn't run in on that filesystem it lists support for "HOLE extents and btrfs no-holes feature" saruman:~# btrfs insp dump-s /dev/mapper/cr superblock: bytenr=65536, device=/dev/mapper/cr --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x34a9cfed [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 4cb82363-e833-444e-b23e-1597a14a8aab metadata_uuid 4cb82363-e833-444e-b23e-1597a14a8aab label btrfs_boot generation 3801933 root 1686707896320 sys_array_size 97 chunk_root_generation 3801931 root_level 1 chunk_root 1679158165504 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 161057079296 bytes_used 112391540736 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x169 ( MIXED_BACKREF | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 3801932 uuid_tree_generation 3801932 dev_item.uuid 924bcb26-9c4c-4106-b3ce-1bb66e66de1f dev_item.fsid 4cb82363-e833-444e-b23e-1597a14a8aab [match] dev_item.type 0 dev_item.total_bytes 161057079296 dev_item.bytes_used 136423931904 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 20:39 ` Marc MERLIN @ 2020-05-25 22:47 ` Chris Murphy 2020-05-25 22:51 ` Chris Murphy 2020-05-26 0:13 ` Marc MERLIN 2020-07-07 3:55 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN 1 sibling, 2 replies; 479+ messages in thread From: Chris Murphy @ 2020-05-25 22:47 UTC (permalink / raw) To: Marc MERLIN Cc: Chris Murphy, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 2:39 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote: > > OK I didn't understand that the problem is with only the sending file > > system, not the receive file system. And also it sounds like the send > > did not cause the problem, but it's somehow a pre-existing problem > > that --repair isn't completely fixing up, or maybe is making different > > (or worse). > > Correct on all points. > > > So I guess the real question is what happened to this file system > > before the send, that got it into this weird state. > > That too, but honestly there are a lot of variables, and it feels like a > bit of wild goose chase. Maybe. The story arc on Btrfs is that check --repair only fixes the things it knows how to fix. It's gotten better but still has the scary warning, and lately has a 10 second delay to really make sure the user meant to use it. And regardless of mode, it's slow and just can't scale. Neither does "wipe and restore from backup". So the problems of inconsistency need to be understood to avoid the problem in the first place. > > Basically it looked like the issues with the FS are pretty minor (I was > able to cp -av the entire data without any file error), but that btrfs > check --repair is unable to make it right, which will likely force me to > wipe and restore. > I know chedk is WIP, and that's why I'm providing feedback :) What about finding inode 14058737 and deleting it? In all of the listed subvolumes? And then unmount and check again? > > > > Is no-holes enabled on either file system? > > > > > > Not intentionally. How do I check? > > > > btrfs insp dump-s > > > > It's not yet the default and can't be inadvertently enabled so chances > > are it's the original holes implementation. > > The filesystem was definitely created a while ago (2-3 years?) > > I have been recently been playing with bees, but I'm reasonably sure I didn't run in on that filesystem > it lists support for "HOLE extents and btrfs no-holes feature" It's default except for LZO. -- Chris Murphy ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 22:47 ` Chris Murphy @ 2020-05-25 22:51 ` Chris Murphy 2020-05-26 0:13 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Chris Murphy @ 2020-05-25 22:51 UTC (permalink / raw) To: Chris Murphy Cc: Marc MERLIN, Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 4:47 PM Chris Murphy <lists@colorremedies.com> wrote: > > On Mon, May 25, 2020 at 2:39 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, May 25, 2020 at 02:24:03PM -0600, Chris Murphy wrote: > > > OK I didn't understand that the problem is with only the sending file > > > system, not the receive file system. And also it sounds like the send > > > did not cause the problem, but it's somehow a pre-existing problem > > > that --repair isn't completely fixing up, or maybe is making different > > > (or worse). > > > > Correct on all points. > > > > > So I guess the real question is what happened to this file system > > > before the send, that got it into this weird state. > > > > That too, but honestly there are a lot of variables, and it feels like a > > bit of wild goose chase. > > Maybe. The story arc on Btrfs is that check --repair only fixes the > things it knows how to fix. It's gotten better but still has the scary > warning, and lately has a 10 second delay to really make sure the user > meant to use it. And regardless of mode, it's slow and just can't > scale. Neither does "wipe and restore from backup". So the problems of > inconsistency need to be understood to avoid the problem in the first > place. > > > > > > Basically it looked like the issues with the FS are pretty minor (I was > > able to cp -av the entire data without any file error), but that btrfs > > check --repair is unable to make it right, which will likely force me to > > wipe and restore. > > I know chedk is WIP, and that's why I'm providing feedback :) > > What about finding inode 14058737 and deleting it? In all of the > listed subvolumes? And then unmount and check again? Before deleting it, can you check if chattr +C is set? Or what kind of file is this? Because there's an old loophole in older kernels where chattr +C could be set, but if defragmented while mounted with a compression option, it would get compressed. And btrfs check complains about it, but it's not actually a problem. Newer kernels don't do this but I'm not certain about the send error's relationship with this. You could try just making a 'cp --reflink=never' copy of the file. And delete the original (and all of its copies in all subvolumes). Now 'btrfs check'. -- Chris Murphy ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.5 kernel and btrfs-progs v5.6 create and cannot fix 'root 204162 inode 14058737 errors 1000, some csum missing' 2020-05-25 22:47 ` Chris Murphy 2020-05-25 22:51 ` Chris Murphy @ 2020-05-26 0:13 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2020-05-26 0:13 UTC (permalink / raw) To: Chris Murphy; +Cc: Su Yue, Btrfs BTRFS, Qu Wenruo, Josef Bacik, Su Yue On Mon, May 25, 2020 at 04:47:31PM -0600, Chris Murphy wrote: > Maybe. The story arc on Btrfs is that check --repair only fixes the > things it knows how to fix. It's gotten better but still has the scary that's always been true indeed :) > warning, and lately has a 10 second delay to really make sure the user > meant to use it. And regardless of mode, it's slow and just can't > scale. Neither does "wipe and restore from backup". So the problems of > inconsistency need to be understood to avoid the problem in the first > place. that's ideal, but honestly it's a multi prong approach. I'm stil hoping that check --repair can improve. > What about finding inode 14058737 and deleting it? In all of the > listed subvolumes? And then unmount and check again? it's in all the subvolumes, including the btrfs send ones. So sure, I can delete my btrfs send snapshots and have to start over, but really btrfs check should fix this better than rm, even if it makes the file not what it was. I can always replace it with a good copy On Mon, May 25, 2020 at 04:51:51PM -0600, Chris Murphy wrote: > Before deleting it, can you check if chattr +C is set? Or what kind of > file is this? Because there's an old loophole in older kernels where saruman:~# lsattr /mnt/mnt/root/usr/share/locale/en_GB/LC_MESSAGES/gstreamer-1.0.mo -------------------- /mnt/mnt/root/usr/share/locale/en_GB/LC_MESSAGES/gstreamer-1.0.mo > You could try just making a 'cp --reflink=never' copy of the file. And > delete the original (and all of its copies in all subvolumes). Now > 'btrfs check'. I can restore the file from backup without issues, it's more that with btrfs send, I'm not allowed to delete the inode until it cycles out, and I can't cycle it out because btrfs send dies when it sees the inconsistent state. I mean, yes, I can do this and delete all the snapshots, I was just hoping to do slighty better as this is close to wiping the filesystem and starting over (actually it's mostly the same to be honest) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed @ 2020-07-07 3:55 ` Marc MERLIN 2020-07-07 14:31 ` Josef Bacik 2020-07-08 3:44 ` Zygo Blaxell 0 siblings, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2020-07-07 3:55 UTC (permalink / raw) To: linux-btrfs I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this before I wipe it (which I assume is the only way out at this point) myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0 Opening filesystem to check... parent transid verify failed on 7325633544192 wanted 359658 found 359661 parent transid verify failed on 7325633544192 wanted 359658 found 359661 parent transid verify failed on 7325633544192 wanted 359658 found 359661 Ignoring transid failure leaf parent key incorrect 7325633544192 ERROR: failed to read block groups: Operation not permitted ERROR: cannot open file system I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean: Opening filesystem to check... Checking filesystem on /dev/mapper/crypt_bcache4 UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d [1/7] checking root items Fixed 0 roots. [2/7] checking extents No device size related problem found [3/7] checking free space cache cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) found 18089211043840 bytes used, no error found total csum bytes: 17580412652 total tree bytes: 82326192128 total fs tree bytes: 56795086848 total extent tree bytes: 5154258944 btree space waste bytes: 13682108904 file data blocks allocated: 24050542804992 I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space btrfs should not corrupt the disk if it's a bit old) myth:/mnt# uname -r 5.6.5-amd64-preempt-sysrq-20190817 Soon after, the copy failed: [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3 [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled [ 2575.931343] BTRFS info (device dm-0): has skinny extents [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations [ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space [ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now [ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space [ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now [ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500 [ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it [ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 [ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 [ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 [ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure [ 3071.661999] BTRFS info (device dm-0): forced readonly [ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 [ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 [ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure Ok, maybe there was an IO failure, although none was shown by the kernel: however, now the FS is mostly dead: [ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3 [ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled [ 3649.106100] BTRFS info (device dm-0): has skinny extents [ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5 [ 3653.002227] BTRFS error (device dm-0): open_ctree failed [ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3 [ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled [ 3876.808195] BTRFS info (device dm-0): has skinny extents [ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5 [ 3880.669435] BTRFS error (device dm-0): open_ctree failed [ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3 [ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled [ 4057.606894] BTRFS info (device dm-0): has skinny extents [ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 4061.503799] BTRFS error (device dm-0): failed to read block g myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0 WARNING: this utility is deprecated, please use 'btrfs rescue zero-log' parent transid verify failed on 7325633544192 wanted 359658 found 359661 parent transid verify failed on 7325633544192 wanted 359658 found 359661 parent transid verify failed on 7325633544192 wanted 359658 found 359661 parent transid verify failed on 7325633544192 wanted 359658 found 359661 Ignoring transid failure leaf parent key incorrect 7325633544192 Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0 myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/ mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so myth:/mnt# dmtail [ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery' [ 6665.975357] BTRFS error (device dm-0): open_ctree failed [ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead [ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time [ 6686.664219] BTRFS info (device dm-0): disabling disk space caching [ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache [ 6686.664232] BTRFS info (device dm-0): has skinny extents [ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 [ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5 [ 6690.556227] BTRFS error (device dm-0): open_ctree failed myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/ Skipping snapshot win_ro.20200615_02:49:02 Skipping snapshot 0Notmachines_ro.20200626_16:09:30 Skipping snapshot 1Appliances_ro.20200626_17:08:56 Skipping snapshot debian32_ro.20200626_17:11:57 Skipping snapshot debian64_ro.20200626_17:18:15 Skipping snapshot ubuntu_ro.20200626_17:18:44 Skipping snapshot win_ro.20200626_18:39:13 Skipping snapshot 0Notmachines_ro.20200629_00:48:39 Skipping snapshot 1Appliances_ro.20200629_01:11:44 Skipping snapshot debian32_ro.20200629_01:11:50 Skipping snapshot debian64_ro.20200629_01:19:52 Skipping snapshot ubuntu_ro.20200629_01:20:01 Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko myth:/mnt# l /mnt/btrfs_bigbackup/ is missing a lot of stuff It's a backup server, I can recreate the data, but it will probably take 2 weeks to copy everything again, and I'd like to know what on earth happened so that I can avoid having this again. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-07 3:55 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN @ 2020-07-07 14:31 ` Josef Bacik 2020-07-07 17:25 ` Marc MERLIN 2020-07-08 3:44 ` Zygo Blaxell 1 sibling, 1 reply; 479+ messages in thread From: Josef Bacik @ 2020-07-07 14:31 UTC (permalink / raw) To: Marc MERLIN, linux-btrfs On 7/6/20 11:55 PM, Marc MERLIN wrote: > I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this > before I wipe it (which I assume is the only way out at this point) > myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0 > Opening filesystem to check... > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > Ignoring transid failure > leaf parent key incorrect 7325633544192 > ERROR: failed to read block groups: Operation not permitted > ERROR: cannot open file system > > > I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean: > > Opening filesystem to check... > Checking filesystem on /dev/mapper/crypt_bcache4 > UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d > [1/7] checking root items > Fixed 0 roots. > [2/7] checking extents > No device size related problem found > [3/7] checking free space cache > cache and super generation don't match, space cache will be invalidated > [4/7] checking fs roots > [5/7] checking only csums items (without verifying data) > [6/7] checking root refs > [7/7] checking quota groups skipped (not enabled on this FS) > found 18089211043840 bytes used, no error found > total csum bytes: 17580412652 > total tree bytes: 82326192128 > total fs tree bytes: 56795086848 > total extent tree bytes: 5154258944 > btree space waste bytes: 13682108904 > file data blocks allocated: 24050542804992 > > > I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake > I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space > btrfs should not corrupt the disk if it's a bit old) > > myth:/mnt# uname -r > 5.6.5-amd64-preempt-sysrq-20190817 > > Soon after, the copy failed: > [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3 > [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled > [ 2575.931343] BTRFS info (device dm-0): has skinny extents > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 You have a corrupt counter here at mount time, does your logs go back far enough to see where those came in? Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-07 14:31 ` Josef Bacik @ 2020-07-07 17:25 ` Marc MERLIN 0 siblings, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2020-07-07 17:25 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Jul 07, 2020 at 10:31:32AM -0400, Josef Bacik wrote: > > Soon after, the copy failed: > > [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3 > > [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled > > [ 2575.931343] BTRFS info (device dm-0): has skinny extents > > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > > You have a corrupt counter here at mount time, does your logs go back far > enough to see where those came in? Thanks, I copied all the logs in the original Email. Here it is all in one swoop: https://pastebin.com/GURufP8w The devices going down and back up is normal, my backup script saw the backup was over (although in that case that's because it failed) and automatically unmounted the filesystem and turned off the drives. I'm not sure how my FS got so corrupted, it's been a while since I've seen damage that bad, especially pretty unexplained like this. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-07 3:55 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN 2020-07-07 14:31 ` Josef Bacik @ 2020-07-08 3:44 ` Zygo Blaxell 2020-07-08 4:10 ` Marc MERLIN 2020-08-12 22:34 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN 1 sibling, 2 replies; 479+ messages in thread From: Zygo Blaxell @ 2020-07-08 3:44 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Jul 06, 2020 at 08:55:30PM -0700, Marc MERLIN wrote: > I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this > before I wipe it (which I assume is the only way out at this point) > myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > Ignoring transid failure > leaf parent key incorrect 7325633544192 > ERROR: failed to read block groups: Operation not permitted > ERROR: cannot open file system > > > I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean: > Opening filesystem to check... > Checking filesystem on /dev/mapper/crypt_bcache4 > UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d > [1/7] checking root items > Fixed 0 roots. > [2/7] checking extents > No device size related problem found > [3/7] checking free space cache > cache and super generation don't match, space cache will be invalidated > [4/7] checking fs roots > [5/7] checking only csums items (without verifying data) > [6/7] checking root refs > [7/7] checking quota groups skipped (not enabled on this FS) > found 18089211043840 bytes used, no error found > total csum bytes: 17580412652 > total tree bytes: 82326192128 > total fs tree bytes: 56795086848 > total extent tree bytes: 5154258944 > btree space waste bytes: 13682108904 > file data blocks allocated: 24050542804992 > > > I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake > I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space > btrfs should not corrupt the disk if it's a bit old) btrfs send has historically had bugs but not filesystem-damaging ones (just relatively harmless kernel crashes and send failures). btrfs receive is almost entirely userspace--it can't corrupt anything that can't be corrupted by normal filesystem operations. > myth:/mnt# uname -r > 5.6.5-amd64-preempt-sysrq-20190817 > > Soon after, the copy failed: > [ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3 > [ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled > [ 2575.931343] BTRFS info (device dm-0): has skinny extents > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 This line does indicate an older problem with the filesystem. It doesn't tell us whether the corruption happened yesterday or a year ago. You will need to look at your older kernel logs for that. > [ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations > [ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space > [ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now > [ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space > [ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now > [ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500 > [ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it Use space_cache=v2, especially on a big filesystem because space_cache=v1 slows down linearly with filesystem size. There is no need for these warnings. They're probably a symptom rather than cause here. > [ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 > [ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 > [ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 > [ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure > [ 3071.661999] BTRFS info (device dm-0): forced readonly > [ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 > [ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661 > [ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure > > Ok, maybe there was an IO failure, although none was shown by the kernel: The "IO failure" mentioned here is the earlier parent transid verify failure. When the verification fails, the caller gets -EIO. > however, now the FS is mostly dead: > [ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3 > [ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled > [ 3649.106100] BTRFS info (device dm-0): has skinny extents > [ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > [ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5 > [ 3653.002227] BTRFS error (device dm-0): open_ctree failed > [ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3 > [ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled > [ 3876.808195] BTRFS info (device dm-0): has skinny extents > [ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > [ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5 > [ 3880.669435] BTRFS error (device dm-0): open_ctree failed > [ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3 > [ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled > [ 4057.606894] BTRFS info (device dm-0): has skinny extents > [ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > [ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 The amount of damage here is small, but it looks like you lost a superblock update or two (btrfs expected an old page and found a new one) so the root of the filesystem now points at an old tree that has since been partially overwritten. Any part of the filesystem on the other side of the missing metadata pages is no longer accessible without a brute force search of the metadata. This might take longer than mkfs+restore with the current btrfs check --repair, especially if you have to run chunk-recover as well. > myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0 > WARNING: this utility is deprecated, please use 'btrfs rescue zero-log' > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > Ignoring transid failure > leaf parent key incorrect 7325633544192 > Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0 Clearing log tree will have no effect on parent transid verify failure. It only helps to work around bugs that occur during log tree replay, which happens at a later stage of mounting the filesystem. > myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/ > mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0, > missing codepage or helper program, or other error > In some cases useful info is found in syslog - try > dmesg | tail or so > > myth:/mnt# dmtail > [ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery' > [ 6665.975357] BTRFS error (device dm-0): open_ctree failed > [ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead > [ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time > [ 6686.664219] BTRFS info (device dm-0): disabling disk space caching > [ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache > [ 6686.664232] BTRFS info (device dm-0): has skinny extents > [ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > [ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > [ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5 > [ 6690.556227] BTRFS error (device dm-0): open_ctree failed > > > myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/ > Skipping snapshot win_ro.20200615_02:49:02 > Skipping snapshot 0Notmachines_ro.20200626_16:09:30 > Skipping snapshot 1Appliances_ro.20200626_17:08:56 > Skipping snapshot debian32_ro.20200626_17:11:57 > Skipping snapshot debian64_ro.20200626_17:18:15 > Skipping snapshot ubuntu_ro.20200626_17:18:44 > Skipping snapshot win_ro.20200626_18:39:13 > Skipping snapshot 0Notmachines_ro.20200629_00:48:39 > Skipping snapshot 1Appliances_ro.20200629_01:11:44 > Skipping snapshot debian32_ro.20200629_01:11:50 > Skipping snapshot debian64_ro.20200629_01:19:52 > Skipping snapshot ubuntu_ro.20200629_01:20:01 > Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko > myth:/mnt# l /mnt/btrfs_bigbackup/ > is missing a lot of stuff > > It's a backup server, I can recreate the data, but it will probably take > 2 weeks to copy everything again, and I'd like to know what on earth > happened so that I can avoid having this again. "parent transid verify failed" is btrfs's gentle way of saying "your devices have busted write ordering, reconfigure or replace them and try again." In a later post in this thread, you posted a log showing your drive model numbers and firmware revisions. White Label repackages drives made by other companies(*) under their own part number (singular, they all use the same part number), but they leave the firmware revision intact, so we can look up the firmware revision and see that you have two different WD models in your md6 array from families with known broken firmware write caching. sdd is running firmware 0957, also found in circa-2014 WD Green. The others are running 01.01RA2 firmware that appears in a model family that includes some broken WD Green and Red models from a few years back (including the venerable datavore 80.00A80). I have a few of the WD branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems. Hopefully your bcache drive is OK, you didn't post any details on that. bcache on a drive with buggy firmware write caching fails *spectacularly*. You can work around buggy write cache firmware with a udev rule like this to disable write cache on all the drives: ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode" Note that in your logs, the kernel reports that 'sdd' has write cache disabled already, maybe due to lack of firmware support or a conservative default setting. That makes it probably the only drive in that array that is working properly. bcache could be losing its mind too, but although I've heard a lot of rumors of bcache bugs, I've yet to catch it having a problem that wasn't directly caused by bad SSD firmware or host configuration. If the bcache was configured in writeback mode and it was separated from the backing device for a while then there could be a consistency issue that would result in something like this, but bcache is pretty good at preventing that. In theory, space_cache=v1 might have consistency issues that lead to some or all of the symptoms above; however, a) btrfs has checks at multiple points to detect or prevent that, and b) the inconsistency would have to be caused by a firmware write cache bug or bcache bug anyway. There are some other questionable things in your setup: you have a mdadm-raid5 with no journal device, so if PPL is also not enabled, and you are running btrfs on top, then this filesystem is vulnerable to instant destruction by mdadm-raid5 write hole after a disk fails. bcache in writeback mode with a single cache pool using multiple physical backing store devices is vulnerable to extra data corruption failures in the event that the SSD(s) go bad. I'm guessing that none of the backing store drives have scterc support, which will complicate error recovery with SATA bus timeouts and resets as disks fail (though most of the problems with that are conveniently also prevented by disabling write cache). None of these issues caused problems today, though, and won't cause a problem until disks start to fail. (*) their product description text says "other companies", but maybe White Label is just a part of WD, hiding their shame as they dispose of unsalable inventory in an unsuspecting market. Don't know, don't care enough to find out. > > Thanks, > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 3:44 ` Zygo Blaxell @ 2020-07-08 4:10 ` Marc MERLIN 2020-07-08 5:49 ` Zygo Blaxell 2022-03-29 17:18 ` Marc MERLIN 2020-08-12 22:34 ` 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed Marc MERLIN 1 sibling, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2020-07-08 4:10 UTC (permalink / raw) To: Zygo Blaxell; +Cc: linux-btrfs Hi Zygo, thanks for the very detailled answer. A few replies: > Hopefully your bcache drive is OK, you didn't post any details on that. > bcache on a drive with buggy firmware write caching fails *spectacularly*. So, I created this as a bcache backing device so that I can elect to add a caching device later (which you can't do after the fact otherwise). There was no caching device here, though. So before we go on, it seems that my FS is damaged beyond easy repair, that we know why (drives with broken write caching) and that unfortunately btrfs gets in states where recovery is hard, at best. If so, I'll just wipe, apply the fixes you recommended and start over. On Tue, Jul 07, 2020 at 11:44:07PM -0400, Zygo Blaxell wrote: > > [ 2575.931343] BTRFS info (device dm-0): has skinny extents > > [ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > > This line does indicate an older problem with the filesystem. It doesn't > tell us whether the corruption happened yesterday or a year ago. > You will need to look at your older kernel logs for that. Yeah, this could have been long time ago, but remember that I ran a 2 day long btrfs check on it and it came back clean. > > [ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it > > Use space_cache=v2, especially on a big filesystem because space_cache=v1 > slows down linearly with filesystem size. There is no need for these > warnings. They're probably a symptom rather than cause here. Thanks for that, was not aware of this. > > Ok, maybe there was an IO failure, although none was shown by the kernel: > > The "IO failure" mentioned here is the earlier parent transid verify > failure. When the verification fails, the caller gets -EIO. Got it, so not a de ice IO failure. > > [ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 > > [ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > [ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > The amount of damage here is small, but it looks like you lost a > superblock update or two (btrfs expected an old page and found a new one) > so the root of the filesystem now points at an old tree that has since > been partially overwritten. > > Any part of the filesystem on the other side of the missing metadata pages > is no longer accessible without a brute force search of the metadata. > This might take longer than mkfs+restore with the current btrfs check > --repair, especially if you have to run chunk-recover as well. unfortunately btrfs check errors early on, so that's not going to work without manual options which seem not very obvious. > > myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0 > > WARNING: this utility is deprecated, please use 'btrfs rescue zero-log' > > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > parent transid verify failed on 7325633544192 wanted 359658 found 359661 > > Ignoring transid failure > > leaf parent key incorrect 7325633544192 > > Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0 > > Clearing log tree will have no effect on parent transid verify failure. > It only helps to work around bugs that occur during log tree replay, > which happens at a later stage of mounting the filesystem. Right, I was just a bit desperate and tried it :) > In a later post in this thread, you posted a log showing your drive model > numbers and firmware revisions. White Label repackages drives made > by other companies(*) under their own part number (singular, they all > use the same part number), but they leave the firmware revision intact, > so we can look up the firmware revision and see that you have two > different WD models in your md6 array from families with known broken > firmware write caching. Dann, I'm impressed by that analysis, thank you. > branded versions of these drives. They are unusable with write cache > enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on > btrfs; on ext4, git and postgresql database corruption. After disabling > write cache, I've used them for years with no problems. Gotcha, I'm very glad you were able to figure that out. As you said, I can disable the write cache. It's a bit sad however that ext4 would have given me something I could have recovered, while with btrfs, it's so much harder to recover if you're not a btrfs FS datastructure expert. > There are some other questionable things in your setup: you have a > mdadm-raid5 with no journal device, so if PPL is also not enabled, Sorry, PPL? > and you are running btrfs on top, then this filesystem is vulnerable > to instant destruction by mdadm-raid5 write hole after a disk fails. wait, if a disk fails, at worst I have a stripe that's half written and hopefully btrfs fails, goes read only and the transaction does not go through, so nothing happens except loss of the last written data? I don't have an external journal because this is an external disk array I can move between machines. Would you suggest I do something else? > (*) their product description text says "other companies", but maybe > White Label is just a part of WD, hiding their shame as they dispose of > unsalable inventory in an unsuspecting market. Don't know, don't care > enough to find out. :) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 4:10 ` Marc MERLIN @ 2020-07-08 5:49 ` Zygo Blaxell 2020-07-08 17:41 ` Marc MERLIN 2022-03-29 17:18 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Zygo Blaxell @ 2020-07-08 5:49 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Jul 07, 2020 at 09:10:41PM -0700, Marc MERLIN wrote: > > branded versions of these drives. They are unusable with write cache > > enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on > > btrfs; on ext4, git and postgresql database corruption. After disabling > > write cache, I've used them for years with no problems. > > Gotcha, I'm very glad you were able to figure that out. As you said, I > can disable the write cache. > It's a bit sad however that ext4 would have given me something I could > have recovered, while with btrfs, it's so much harder to recover if > you're not a btrfs FS datastructure expert. ext2 is pretty much indestructible if you ignore data integrity. If you want to destroy ext2 metadata, you have to overwrite it one bit at a time. The few cases where damage is not proportional to write size (indirect block lists, directories) are limited to single inodes. Every metadata item appears in exactly one location that can be computed, no searching required. Recovering from arbitrarily awful firmware is one of the few things ext2 is really good at. ext4 is significantly more fragile than ext2 if you use the newer features like extent and flex_bg, which make the metadata mobile like btrfs. These can prevent e2fsck from being able to recover a large filesystem. I switched to btrfs back in 2014 after losing a couple of big ext4 filesystems on arrays of WD Greens, _then_ discovered the problem was the disk firmware not the filesystem. > > There are some other questionable things in your setup: you have a > > mdadm-raid5 with no journal device, so if PPL is also not enabled, > > Sorry, PPL? Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm consistency policy, like the journal, but uses reserved metadata space instead of a separate device. > > and you are running btrfs on top, then this filesystem is vulnerable > > to instant destruction by mdadm-raid5 write hole after a disk fails. > > wait, if a disk fails, at worst I have a stripe that's half written and > hopefully btrfs fails, goes read only and the transaction does not go > through, so nothing happens except loss of the last written data? If the array is degraded, and stripe is partially updated, then there is a crash or power failure, parity will be out of sync with data blocks in the stripe, so the missing disk's data cannot be generated from parity. Both old and new data can be damaged by raid5 write hole. The data that is damaged is the block on the missing disk that must be computed using the contents of all other disks. The damage affects old and new data the stripe with equal probability, as the data and parity blocks rotate from one stripe to the next. Damaged data in an uncommitted tree (new data) will be ignored if the transaction is not completed, as no reference to the root of the uncommitted tree will exist after a crash. Damaged data in a committed tree (old data) is already committed, and if it's metadata the damage will also break the filesystem. In other words, only old data can be damaged by the write hole, because any new damaged data will be filtered out by the transaction mechanism. If you have dup metadata in the btrfs then maybe you can recover from the mirror copy in another stripe. Hopefully that's not damaged too, but since both mirrors are updated at roughly the same time on the same disks, damage to both copies is quite likely. mdadm PPL or the journal device finishes partial stripe updates after a crash or power failure, and avoids this failure mode. > I don't have an external journal because this is an external disk array > I can move between machines. Would you suggest I do something else? Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's barely usable and some stuff doesn't work properly, but it can run a backup server, replace a failed disk, and usually self-repair disk corruption too). > > (*) their product description text says "other companies", but maybe > > White Label is just a part of WD, hiding their shame as they dispose of > > unsalable inventory in an unsuspecting market. Don't know, don't care > > enough to find out. > > :) > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ > ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 5:49 ` Zygo Blaxell @ 2020-07-08 17:41 ` Marc MERLIN 2020-07-08 17:44 ` Roman Mamedov 2020-07-08 22:47 ` Zygo Blaxell 0 siblings, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2020-07-08 17:41 UTC (permalink / raw) To: Zygo Blaxell; +Cc: linux-btrfs On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote: > > Sorry, PPL? > > Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm > consistency policy, like the journal, but uses reserved metadata space > instead of a separate device. looks like it's incompatible with --bitmap which I was already using. I'm not sure if --grow was possible to convert from one to another, so I made a new one, and now I have a 4 day wait for the initial scan (external array with Sata PMP, those are slow unfortunately) md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0] 23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_] [>....................] recovery = 0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec Could you confirm that what I did here is ok? gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: /dev/sdq1 appears to be part of a raid array: level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 mdadm: /dev/sdu1 appears to be part of a raid array: level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 mdadm: /dev/sdv1 appears to be part of a raid array: level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 mdadm: /dev/sdw1 appears to be part of a raid array: level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 mdadm: /dev/sdx1 appears to be part of a raid array: level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 mdadm: size set to 5860386816K Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md6 started. gargamel:~# make-bcache -B /dev/md6 UUID: eb898c09-debd-4e86-972f-aecdb59670e2 Set UUID: e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9 version: 1 block_size: 1 data_offset: 16 gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4 gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6 Enter passphrase for /dev/bcache4: gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 > > wait, if a disk fails, at worst I have a stripe that's half written and > > hopefully btrfs fails, goes read only and the transaction does not go > > through, so nothing happens except loss of the last written data? > > If the array is degraded, and stripe is partially updated, then there is > a crash or power failure, parity will be out of sync with data blocks > in the stripe, so the missing disk's data cannot be generated from parity. I thought --bitmap would know this and know to discard the last blocks partiailly written. I guess not? Either way, it seems that losing just a few blocks in the "wrong" place is enough to lose most of a btrfs FS. That's disappointing, I thought it was a bit more fault proof than that. > > I don't have an external journal because this is an external disk array > > I can move between machines. Would you suggest I do something else? > > Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's > barely usable and some stuff doesn't work properly, but it can run > a backup server, replace a failed disk, and usually self-repair disk > corruption too). it's been a while since I've used btrfs raid5, it must have improved since I last did, but I haven't read that it's become production quality yet :) Thanks again for your very helpful answers. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 17:41 ` Marc MERLIN @ 2020-07-08 17:44 ` Roman Mamedov 2020-07-08 22:47 ` Zygo Blaxell 1 sibling, 0 replies; 479+ messages in thread From: Roman Mamedov @ 2020-07-08 17:44 UTC (permalink / raw) To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs On Wed, 8 Jul 2020 10:41:10 -0700 Marc MERLIN <marc@merlins.org> wrote: > On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote: > > > Sorry, PPL? > > > > Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm > > consistency policy, like the journal, but uses reserved metadata space > > instead of a separate device. > > looks like it's incompatible with --bitmap which I was already using. > I'm not sure if --grow was possible to convert from one to another At least you can definitely remove the bitmap with --bitmap=none in the grow mode. Then in the second --grow can probably add PPL (unfamiliar with that). -- With respect, Roman ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 17:41 ` Marc MERLIN 2020-07-08 17:44 ` Roman Mamedov @ 2020-07-08 22:47 ` Zygo Blaxell 1 sibling, 0 replies; 479+ messages in thread From: Zygo Blaxell @ 2020-07-08 22:47 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Jul 08, 2020 at 10:41:10AM -0700, Marc MERLIN wrote: > On Wed, Jul 08, 2020 at 01:49:05AM -0400, Zygo Blaxell wrote: > > > Sorry, PPL? > > > > Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm > > consistency policy, like the journal, but uses reserved metadata space > > instead of a separate device. > > looks like it's incompatible with --bitmap which I was already using. > I'm not sure if --grow was possible to convert from one to another, so I > made a new one, and now I have a 4 day wait for the initial scan > (external array with Sata PMP, those are slow unfortunately) > > md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0] > 23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_] > [>....................] recovery = 0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec > > Could you confirm that what I did here is ok? > > gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1 > mdadm: layout defaults to left-symmetric > mdadm: layout defaults to left-symmetric > mdadm: chunk size defaults to 512K > mdadm: /dev/sdq1 appears to be part of a raid array: > level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 > mdadm: /dev/sdu1 appears to be part of a raid array: > level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 > mdadm: /dev/sdv1 appears to be part of a raid array: > level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 > mdadm: /dev/sdw1 appears to be part of a raid array: > level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 > mdadm: /dev/sdx1 appears to be part of a raid array: > level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016 > mdadm: size set to 5860386816K > Continue creating array? y > mdadm: Defaulting to version 1.2 metadata > mdadm: array /dev/md6 started. > > gargamel:~# make-bcache -B /dev/md6 > UUID: eb898c09-debd-4e86-972f-aecdb59670e2 > Set UUID: e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9 > version: 1 > block_size: 1 > data_offset: 16 > > gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4 > > gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6 > Enter passphrase for /dev/bcache4: > > gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 Looks OK. > > > wait, if a disk fails, at worst I have a stripe that's half written and > > > hopefully btrfs fails, goes read only and the transaction does not go > > > through, so nothing happens except loss of the last written data? > > > > If the array is degraded, and stripe is partially updated, then there is > > a crash or power failure, parity will be out of sync with data blocks > > in the stripe, so the missing disk's data cannot be generated from parity. > > I thought --bitmap would know this and know to discard the last blocks > partiailly written. I guess not? > > Either way, it seems that losing just a few blocks in the "wrong" place > is enough to lose most of a btrfs FS. That's disappointing, I thought it > was a bit more fault proof than that. During a write to a stripe, there's a short window of time when data and parity blocks are out of sync (each disk completes writes at its own speed). If the system is interrupted during this time (crash or power failure), the stripe on disk has data blocks that don't match the parity block. Suppose we have 5 disks. There is always a 1:1 mapping of disks to blocks in every RAID5 stripe, so let's call the data blocks on each disk A, B, C, D, and the parity block P, and ignore the other details. When all the disks are online, before we do a write to D, we have: A ^ B ^ C ^ Dold = Pold (1) After the first write, we have this on disk: A, B, C, Dnew, Pold After the second write, we have this on disk: A, B, C, Dnew, Pnew A ^ B ^ C ^ Dnew = Pnew (2) If we lose drive C, we can get it back from either the first or last state: A ^ B ^ Cmissing ^ Dold = Pold so Cmissing = Pold ^ A ^ B ^ Dold (3) Sub (1) into (3): Cmissing = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dold Sub A ^ A = 0, B ^ B = 0, Dold ^ Dold = 0: Cmissing = C So we recover C correctly. The same can be done with Pnew and Dnew. If there is a power failure or crash and the second write doesn't happen, then we get: Cgarbage = Pold ^ A ^ B ^ Dnew (4) Sub (1) into (4): Cgarbage = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dnew Cancel A ^ A = 0, B ^ B = 0: Cgarbage = C ^ Dold ^ Dnew We didn't get C back, we got C xored with some other stuff. If the writes happen in the other order, we get: Cgarbage2 = Pnew ^ A ^ B ^ Dold (5) Sub (2) into (5): Cgarbage2 = (A ^ B ^ C ^ Dnew) ^ A ^ B ^ Dold Cancel A ^ A = 0, B ^ B = 0: Cgarbage2 = C ^ Dnew ^ Dold We didn't get C back here either. With only 2 disks written, we get the same garbage in both cases. If there are more than 2 disks written then we get multiple possible combinations of writes xored together. If the array is not degraded, then after the crash, mdadm assumes the data blocks are correct, and recomputes the parity block from the data blocks during resync (i.e. it updates Pold to Pnew using Dnew). The filesystem looks only at the data blocks, and all the data blocks are present, so this works. The mdadm bitmaps speed this up by indicating which areas of the disk contain dirty stripes that may require parity recomputation. Areas that have not been written don't need to be resynced. If the array is degraded, some of the data blocks are missing, so we can't recompute Pnew from whichever of A, B, C, and D blocks are present. If we have Pold and Dold it's OK, we can correctly compute A, B, or C if one is missing. If we have Pnew and Dnew it's OK. If P is on the missing disk it's OK, because we have all the data blocks in the stripe intact so we don't need P. If D is missing, we will recompute either Dnew or Dold depending on whether the P block contains Pnew or Pold, but btrfs transactions can handle both results so it's OK no matter what D contains. So far so good, all these cases are OK. PPL ensures we only have these cases. If we have Pold and Dnew or Pnew and Dold, then the computed contents of A, B, and C will be broken when any of those drives are missing. Since we weren't writing to blocks A, B, or C in this transaction, but those are the blocks that are corrupted, we may have broken something btrfs already committed. btrfs will see broken csums and reject the data. If A, B, or C are btrfs metadata blocks, the filesystem is damaged. Either a mirror copy of metadata or a practical fsck is required to repair. btrfs transactions cover only the specific blocks that were written. They don't handle cases where a write to block D corrupts data in other blocks like A, B, or C. raid5 in degraded mode has cases like that, so it doesn't work with btrfs. > > > I don't have an external journal because this is an external disk array > > > I can move between machines. Would you suggest I do something else? > > > > Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's > > barely usable and some stuff doesn't work properly, but it can run > > a backup server, replace a failed disk, and usually self-repair disk > > corruption too). > > it's been a while since I've used btrfs raid5, it must have improved > since I last did, but I haven't read that it's become production quality > yet :) Well, don't try to run a live customer orders database on btrfs raid5. That won't end well. It can hold a backup copy of the customer orders database though, provided you follow the restrictions in https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/ and avoid (or work around) drives with write-caching bugs. > Thanks again for your very helpful answers. > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2020-07-08 4:10 ` Marc MERLIN 2020-07-08 5:49 ` Zygo Blaxell @ 2022-03-29 17:18 ` Marc MERLIN 2022-03-30 5:38 ` Andrei Borzenkov 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-03-29 17:18 UTC (permalink / raw) To: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo Cc: Su Yue Howdy, This is the followup I was hoping I'd never have to send. kernel was 5.7 (long uptime, just upgraded to 5.16). One raid5 drive failed, and as I was replacing it, another one went offline, but not in a way that the md5 array was taken down. I shut the the system down, replaced the bad drive, the 2nd drive that went down wasn't really down, so I broght back the array with a drive missing. mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1 cryptsetup luksOpen /dev/bcache3 dshelf1a btrfs device scan --all-devices mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/ BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 BTRFS error (device dm-17): failed to read chunk tree: -5 BTRFS error (device dm-17): open_ctree failed It's a pretty massive array that will take a *long* time to recover from backup. If there is a reaosnable way to bring it back up, I'd be appreciative. gargamel:~# mount -o ro,recovery /dev/mapper/dshelf1a /mnt/btrfs_pool1/ BTRFS info (device dm-17): flagging fs with big metadata feature BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead BTRFS info (device dm-17): trying to use backup root at mount time BTRFS info (device dm-17): disk space caching is enabled BTRFS info (device dm-17): has skinny extents BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 BTRFS error (device dm-17): failed to read chunk tree: -5 BTRFS error (device dm-17): open_ctree failed Worst case, if it's dead, I'm still happy to try btrfs check lowmem to see if it has gotten better. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2022-03-29 17:18 ` Marc MERLIN @ 2022-03-30 5:38 ` Andrei Borzenkov 2022-03-30 14:39 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Andrei Borzenkov @ 2022-03-30 5:38 UTC (permalink / raw) To: Marc MERLIN, Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo Cc: Su Yue On 29.03.2022 20:18, Marc MERLIN wrote: > Howdy, > > This is the followup I was hoping I'd never have to send. > > kernel was 5.7 (long uptime, just upgraded to 5.16). > > One raid5 drive failed, and as I was replacing it, another one went > offline, but not in a way that the md5 array was taken down. > I shut the the system down, replaced the bad drive, the 2nd drive that > went down wasn't really down, so I broght back the array with a drive > missing. > Sorry, which drive? You had one drive that failed and another drive that went offline. Which drive you replaced? Which drive is missing now? The third one? The one that failed? The one that went offline? > mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1 > cryptsetup luksOpen /dev/bcache3 dshelf1a > btrfs device scan --all-devices > mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/ > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > BTRFS error (device dm-17): failed to read chunk tree: -5 > BTRFS error (device dm-17): open_ctree failed > > It's a pretty massive array that will take a *long* time to recover from > backup. If there is a reaosnable way to bring it back up, I'd be > appreciative. > > gargamel:~# mount -o ro,recovery /dev/mapper/dshelf1a /mnt/btrfs_pool1/ > BTRFS info (device dm-17): flagging fs with big metadata feature > BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead > BTRFS info (device dm-17): trying to use backup root at mount time > BTRFS info (device dm-17): disk space caching is enabled > BTRFS info (device dm-17): has skinny extents > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > BTRFS error (device dm-17): failed to read chunk tree: -5 > BTRFS error (device dm-17): open_ctree failed > > Worst case, if it's dead, I'm still happy to try btrfs check lowmem to > see if it has gotten better. > > Thanks, > Marc ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: 5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed 2022-03-30 5:38 ` Andrei Borzenkov @ 2022-03-30 14:39 ` Marc MERLIN 2022-03-31 17:19 ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-03-30 14:39 UTC (permalink / raw) To: Andrei Borzenkov Cc: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo, Su Yue On Wed, Mar 30, 2022 at 08:38:19AM +0300, Andrei Borzenkov wrote: > On 29.03.2022 20:18, Marc MERLIN wrote: > > Howdy, > > > > This is the followup I was hoping I'd never have to send. > > > > kernel was 5.7 (long uptime, just upgraded to 5.16). > > > > One raid5 drive failed, and as I was replacing it, another one went > > offline, but not in a way that the md5 array was taken down. > > I shut the the system down, replaced the bad drive, the 2nd drive that > > went down wasn't really down, so I broght back the array with a drive > > missing. > > Sorry, which drive? You had one drive that failed and another drive that > went offline. Which drive you replaced? Which drive is missing now? The The drive that failed entirely is the one I removed since clearly I couldn't bring the array back up with it. The one that went offline during shutdown was working again after reboot, so I brought the array back with one drive missing as shown below (4 out of 5) > third one? The one that failed? The one that went offline? > > mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1 > > cryptsetup luksOpen /dev/bcache3 dshelf1a > > btrfs device scan --all-devices > > mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/ > > > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > BTRFS error (device dm-17): failed to read chunk tree: -5 > > BTRFS error (device dm-17): open_ctree failed But clearly the drive that went offline during shutdown, must have caused some corruption, even if it all it did was just refuse all writes. That said, I was somehow hoping that btrfs could unwind the last writes that failed/were incomplete and get back to a proper state. I'm still saddened by how fragile btrfs seems compared to ext4 in those cases (I've had similar issues happen with ext4 in the past, and was always able to repair the filesystem even if I lost a few files in the process) Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-03-30 14:39 ` Marc MERLIN @ 2022-03-31 17:19 ` Marc MERLIN 2022-04-03 23:33 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-03-31 17:19 UTC (permalink / raw) To: Andrei Borzenkov Cc: Zygo Blaxell, Josef Bacik, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo I'm going to wait a few more days before I give up and restore from backup and will consider whether if I should go back to ext4 which for me has clearly been more resilient of disk misbehaviours like I get sometimes (of course, nothing will recover from a real double risk failure in raid5, but over the last 20 years, it's not been uncommon for me to see more than one drive kicked out of an array due to SCSI or sata issues, or a drive having a weird shutdown and being able to come back after a power cycle with all data except the last blocks written to it). If I go back with btrfs (despite it being non resilient to any problem described above, btrfs send is great for backups of course), what are today's best recommended practices? Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger files (100MB to 10GB). 1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach gargamel:/dev# cat /sys/block/md7/bcache/cache_mode [writethrough] writeback writearound none 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache64 4) cryptsetup luksOpen /dev/bcache64 dshelf1 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1 Any other btrfs options I should set for format to improve reliability first and performance second? I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ? Anything else you recommend given that I indeed I have layers in the middle. I'm actually considering dropping bcache for this filesystem given that it could be another cause for corruption that btrfs can't deal with. Thanks Marc > > > mdadm --assemble --run --force /dev/md7 /dev/sd[gijk]1 > > > cryptsetup luksOpen /dev/bcache3 dshelf1a > > > btrfs device scan --all-devices > > > mount /dev/mapper/dshelf1a /mnt/btrfs_pool1/ > > > > > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > BTRFS error (device dm-17): failed to read chunk tree: -5 > > > BTRFS error (device dm-17): open_ctree failed > > But clearly the drive that went offline during shutdown, must have > caused some corruption, even if it all it did was just refuse all > writes. > That said, I was somehow hoping that btrfs could unwind the last writes > that failed/were incomplete and get back to a proper state. I'm still > saddened by how fragile btrfs seems compared to ext4 in those cases > (I've had similar issues happen with ext4 in the past, and was always > able to repair the filesystem even if I lost a few files in the process) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-03-31 17:19 ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN @ 2022-04-03 23:33 ` Josef Bacik 2022-04-04 1:01 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-03 23:33 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo On Thu, Mar 31, 2022 at 10:19:27AM -0700, Marc MERLIN wrote: > I'm going to wait a few more days before I give up and restore from > backup and will consider whether if I should go back to ext4 which for > me has clearly been more resilient of disk misbehaviours like I get > sometimes (of course, nothing will recover from a real double risk > failure in raid5, but over the last 20 years, it's not been uncommon for > me to see more than one drive kicked out of an array due to SCSI or sata > issues, or a drive having a weird shutdown and being able to come back > after a power cycle with all data except the last blocks written to it). > > If I go back with btrfs (despite it being non resilient to any problem > described above, btrfs send is great for backups of course), what are > today's best recommended practices? > > Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger > files (100MB to 10GB). > > 1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal > 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 > /sys/block/bcache64/bcache/attach > gargamel:/dev# cat /sys/block/md7/bcache/cache_mode > [writethrough] writeback writearound none > 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache64 > 4) cryptsetup luksOpen /dev/bcache64 dshelf1 > 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1 > > Any other btrfs options I should set for format to improve reliability > first and performance second? > I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ? > > Anything else you recommend given that I indeed I have layers in the > middle. I'm actually considering dropping bcache for this filesystem > given that it could be another cause for corruption that btrfs can't > deal with. > Sorry shit went real wrong this week. Can you do btrfs inspect-internal dump-super -f /dev/whatever This is going to spit out the backup roots for everything. I was looking at the code and realized the backup root code doesn't actually work for the chunk root, since it's special. Can you then do btrfs check --chunk-root <bytenr> /dev/whatever from whatever the newest backup root is, and then keep rolling back until you find one that works. If you find one that does work hooray, I can write something to swap out the chunk bytenr in your super block and we can carry on. If not we can go with btrfs rescue chunk-recover /dev/whatever and see how that works out. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-03 23:33 ` Josef Bacik @ 2022-04-04 1:01 ` Marc MERLIN 2022-04-04 15:08 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 1:01 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Su Yue, Qu Wenruo On Sun, Apr 03, 2022 at 07:33:27PM -0400, Josef Bacik wrote: > Sorry shit went real wrong this week. Can you do > > btrfs inspect-internal dump-super -f /dev/whatever Thanks for the answer. superblock: bytenr=65536, device=/dev/mapper/dshelf1a --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xd3d00183 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 96539b8c-ccc9-47bf-9e6c-29305890941e metadata_uuid 96539b8c-ccc9-47bf-9e6c-29305890941e label dshelf1 generation 1602089 root 13577814573056 sys_array_size 129 chunk_root_generation 1600938 root_level 1 chunk_root 21069824 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 24004156973056 bytes_used 15113376952320 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x169 ( MIXED_BACKREF | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) cache_generation 1602089 uuid_tree_generation 1602089 dev_item.uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30 dev_item.fsid 96539b8c-ccc9-47bf-9e6c-29305890941e [match] dev_item.type 0 dev_item.total_bytes 24004156973056 dev_item.bytes_used 15178439589888 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 0 stripe 0 devid 1 offset 20971520 dev_uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30 stripe 1 devid 1 offset 29360128 dev_uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30 backup_roots[4]: backup 0: backup_tree_root: 13577814573056 gen: 1602089 level: 1 backup_chunk_root: 21069824 gen: 1600938 level: 1 backup_extent_root: 13577824387072 gen: 1602089 level: 2 backup_fs_root: 13577799401472 gen: 1602089 level: 2 backup_dev_root: 15645196861440 gen: 1600938 level: 1 backup_csum_root: 13577868050432 gen: 1602089 level: 3 backup_total_bytes: 24004156973056 backup_bytes_used: 15113376952320 backup_num_devices: 1 backup 1: backup_tree_root: 13577775284224 gen: 1602086 level: 1 backup_chunk_root: 21069824 gen: 1600938 level: 1 backup_extent_root: 13577801826304 gen: 1602086 level: 2 backup_fs_root: 13577770450944 gen: 1602086 level: 2 backup_dev_root: 15645196861440 gen: 1600938 level: 1 backup_csum_root: 13577819963392 gen: 1602086 level: 3 backup_total_bytes: 24004156973056 backup_bytes_used: 15113376985088 backup_num_devices: 1 backup 2: backup_tree_root: 13577821667328 gen: 1602087 level: 1 backup_chunk_root: 21069824 gen: 1600938 level: 1 backup_extent_root: 13577825026048 gen: 1602087 level: 2 backup_fs_root: 13577810673664 gen: 1602087 level: 2 backup_dev_root: 15645196861440 gen: 1600938 level: 1 backup_csum_root: 13577829089280 gen: 1602087 level: 3 backup_total_bytes: 24004156973056 backup_bytes_used: 15113377001472 backup_num_devices: 1 backup 3: backup_tree_root: 13577779511296 gen: 1602088 level: 1 backup_chunk_root: 21069824 gen: 1600938 level: 1 backup_extent_root: 13577802170368 gen: 1602088 level: 2 backup_fs_root: 13577773039616 gen: 1602088 level: 2 backup_dev_root: 15645196861440 gen: 1600938 level: 1 backup_csum_root: 13577819963392 gen: 1602088 level: 3 backup_total_bytes: 24004156973056 backup_bytes_used: 15113377017856 backup_num_devices: 1 > This is going to spit out the backup roots for everything. I was looking at the > code and realized the backup root code doesn't actually work for the chunk root, > since it's special. Can you then do > > btrfs check --chunk-root <bytenr> /dev/whatever > from whatever the newest backup root is, and then keep rolling back until you > find one that works. If you find one that does work hooray, I can write > something to swap out the chunk bytenr in your super block and we can carry on. All four seem to be generation 1600938, backup_chunk_root:21069824 gargamel:/var/cache/apt# btrfs check --chunk-root 21069824 /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 ERROR: could not setup extent tree ERROR: cannot open file system > If not we can go with > btrfs rescue chunk-recover /dev/whatever > and see how that works out. Thanks, gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a Scanning: 1509736448 in dev0 I'll let this run, looks like it might take a while. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 1:01 ` Marc MERLIN @ 2022-04-04 15:08 ` Marc MERLIN 2022-04-04 17:18 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 15:08 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote: > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > Scanning: 1509736448 in dev0 > > I'll let this run, looks like it might take a while. 14h later, still going on. I'll update when it's done, it may take a few days. gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a Scanning: 7049296592896 in dev0 Please let me know about the backup chunk roots that all seemed to be the same and were apparently all pointing to the same corrupted version? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 15:08 ` Marc MERLIN @ 2022-04-04 17:18 ` Josef Bacik 2022-04-04 17:43 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 17:18 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote: > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > > Scanning: 1509736448 in dev0 > > > > I'll let this run, looks like it might take a while. > > 14h later, still going on. I'll update when it's done, it may take a few > days. > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > Scanning: 7049296592896 in dev0 > > Please let me know about the backup chunk roots that all seemed to be > the same and were apparently all pointing to the same corrupted version? Ok cool apparently we have code that does this already, can you do btrfs-find-root -o 3 /dev/whatever and then use the bytenr's it finds like you did with the backup roots btrfs check --chunk-bytenr <bytenr> /dev/whatever and see if one of them works. You can do this without stopping the chunk recovery if you want to let that keep going. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 17:18 ` Josef Bacik @ 2022-04-04 17:43 ` Marc MERLIN 2022-04-04 17:53 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 17:43 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 01:18:48PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote: > > > > On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote: > > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > > > Scanning: 1509736448 in dev0 > > > > > > I'll let this run, looks like it might take a while. > > > > 14h later, still going on. I'll update when it's done, it may take a few > > days. > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > > Scanning: 7049296592896 in dev0 > > > > Please let me know about the backup chunk roots that all seemed to be > > the same and were apparently all pointing to the same corrupted version? > > Ok cool apparently we have code that does this already, can you do > > btrfs-find-root -o 3 /dev/whatever > and then use the bytenr's it finds like you did with the backup roots gargamel:~# btrfs-find-root -o 3 /dev/mapper/dshelf1a parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 WARNING: could not setup extent tree, skipping it parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 Couldn't setup device tree Superblock thinks the generation is 1600938 Superblock thinks the level is 1 Found tree root at 21069824 gen 1600938 level 1 Well block 21053440(gen: 1822 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Well block 21004288(gen: 1692 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Well block 20987904(gen: 1443 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Well block 20971520(gen: 1293 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Well block 21037056(gen: 1170 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Well block 21020672(gen: 773 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 Not a good sign I assume? > btrfs check --chunk-bytenr <bytenr> /dev/whatever > > and see if one of them works. You can do this without stopping the > chunk recovery if you want to let that keep going. Thanks, > > Josef > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 17:43 ` Marc MERLIN @ 2022-04-04 17:53 ` Josef Bacik 2022-04-04 18:10 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 17:53 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 1:44 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 01:18:48PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 11:09 AM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Sun, Apr 03, 2022 at 06:01:01PM -0700, Marc MERLIN wrote: > > > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > > > > Scanning: 1509736448 in dev0 > > > > > > > > I'll let this run, looks like it might take a while. > > > > > > 14h later, still going on. I'll update when it's done, it may take a few > > > days. > > > gargamel:~# btrfs rescue chunk-recover /dev/mapper/dshelf1a > > > Scanning: 7049296592896 in dev0 > > > > > > Please let me know about the backup chunk roots that all seemed to be > > > the same and were apparently all pointing to the same corrupted version? > > > > Ok cool apparently we have code that does this already, can you do > > > > btrfs-find-root -o 3 /dev/whatever > > and then use the bytenr's it finds like you did with the backup roots > > gargamel:~# btrfs-find-root -o 3 /dev/mapper/dshelf1a > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > leaf parent key incorrect 13577821667328 > WARNING: could not setup extent tree, skipping it > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > leaf parent key incorrect 13577821667328 > Couldn't setup device tree > Superblock thinks the generation is 1600938 > Superblock thinks the level is 1 > Found tree root at 21069824 gen 1600938 level 1 > Well block 21053440(gen: 1822 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > Well block 21004288(gen: 1692 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > Well block 20987904(gen: 1443 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > Well block 20971520(gen: 1293 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > Well block 21037056(gen: 1170 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > Well block 21020672(gen: 773 level: 0) seems good, but generation/level doesn't match, want gen: 1600938 level: 1 > > Not a good sign I assume? > Can you do btrfs check --chunk-bytenr 21053440 /dev/whatever also I can't seem to find output from a normal btrfs check run, can I see that as well? I'm curious if we could just tell the super block the right transid for this and everything would be ok. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 17:53 ` Josef Bacik @ 2022-04-04 18:10 ` Marc MERLIN 2022-04-04 18:46 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 18:10 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 01:53:56PM -0400, Josef Bacik wrote: > Can you do > > btrfs check --chunk-bytenr 21053440 /dev/whatever I assume you meant chunk-root gargamel:~# btrfs check --chunk-root 21053440 /dev/mapper/dshelf1a Opening filesystem to check... No mapping for 13577814573056-13577814589440 Couldn't map the block 13577814573056 Couldn't map the block 13577814573056 bad tree block 13577814573056, bytenr mismatch, want=13577814573056, have=0 Couldn't read tree root ERROR: cannot open file system gargamel:~# btrfs check --chunk-root 21069824 /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 ERROR: could not setup extent tree ERROR: cannot open file system > also I can't seem to find output from a normal btrfs check run, can I > see that as well? I'm curious if we could just tell the super block > the right transid for this and everything would be ok. Thanks, gargamel:~# btrfs check /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 ERROR: could not setup extent tree ERROR: cannot open file system Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 18:10 ` Marc MERLIN @ 2022-04-04 18:46 ` Josef Bacik 2022-04-04 19:04 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 18:46 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 2:10 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 01:53:56PM -0400, Josef Bacik wrote: > > Can you do > > > > btrfs check --chunk-bytenr 21053440 /dev/whatever > > I assume you meant chunk-root > I did, my bad. > > gargamel:~# btrfs check /dev/mapper/dshelf1a > Opening filesystem to check... > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > leaf parent key incorrect 13577821667328 > ERROR: could not setup extent tree > ERROR: cannot open file system > Ok now that I've learned how to read, the real problem is the extent tree is fucked. The chunk tree isn't happy, but it appears to be sane enough to proceed, which is good because that could be unfun to restore. Good news is the extent tree is easily re-buildable, lets use btrfs check --init-extent-tree /dev/whatever and let that baby do its thing. Hopefully that rebuilds everything and you'll be good to go. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 18:46 ` Josef Bacik @ 2022-04-04 19:04 ` Marc MERLIN 2022-04-04 19:52 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 19:04 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 02:46:57PM -0400, Josef Bacik wrote: > Ok now that I've learned how to read, the real problem is the extent > tree is fucked. The chunk tree isn't happy, but it appears to be sane > enough to proceed, which is good because that could be unfun to > restore. > > Good news is the extent tree is easily re-buildable, lets use > > btrfs check --init-extent-tree /dev/whatever > > and let that baby do its thing. Hopefully that rebuilds everything > and you'll be good to go. Thanks, Thanks for the suggestion, sadly it's not that simple: gargamel:~# btrfs check --init-extent-tree /dev/mapper/dshelf1a WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 WARNING: could not setup extent tree, skipping it parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf parent key incorrect 13577821667328 Couldn't setup device tree ERROR: cannot open file system gargamel:~# btrfs --version btrfs-progs v5.10.1 Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 19:04 ` Marc MERLIN @ 2022-04-04 19:52 ` Josef Bacik 2022-04-04 20:33 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 19:52 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 3:04 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 02:46:57PM -0400, Josef Bacik wrote: > > Ok now that I've learned how to read, the real problem is the extent > > tree is fucked. The chunk tree isn't happy, but it appears to be sane > > enough to proceed, which is good because that could be unfun to > > restore. > > > > Good news is the extent tree is easily re-buildable, lets use > > > > btrfs check --init-extent-tree /dev/whatever > > > > and let that baby do its thing. Hopefully that rebuilds everything > > and you'll be good to go. Thanks, > > Thanks for the suggestion, sadly it's not that simple: > > gargamel:~# btrfs check --init-extent-tree /dev/mapper/dshelf1a > WARNING: > > Do not use --repair unless you are advised to do so by a developer > or an experienced user, and then only after having accepted that no > fsck can successfully repair all types of filesystem corruption. Eg. > some software or hardware bugs can fatally damage a volume. > The operation will start in 10 seconds. > Use Ctrl-C to stop it. > 10 9 8 7 6 5 4 3 2 1 > Starting repair. > Opening filesystem to check... > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > leaf parent key incorrect 13577821667328 > WARNING: could not setup extent tree, skipping it > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > leaf parent key incorrect 13577821667328 > Couldn't setup device tree > ERROR: cannot open file system > > gargamel:~# btrfs --version > btrfs-progs v5.10.1 > Can you build a recent btrfs-progs from git? We chucked that error apparently and I can't figure out where it's complaining. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 19:52 ` Josef Bacik @ 2022-04-04 20:33 ` Marc MERLIN 2022-04-04 21:04 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 20:33 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote: > Can you build a recent btrfs-progs from git? We chucked that error > apparently and I can't figure out where it's complaining. Thanks, Sure, here's git master gargamel:/var/local/src/btrfs-progs# ./btrfs --version btrfs-progs v5.16.2 gargamel:/var/local/src/btrfs-progs# ./btrfs check --init-extent-tree /dev/mapper/dshelf1a WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't setup device tree ERROR: cannot open file system Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 20:33 ` Marc MERLIN @ 2022-04-04 21:04 ` Josef Bacik 2022-04-04 21:29 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 21:04 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote: > > Can you build a recent btrfs-progs from git? We chucked that error > > apparently and I can't figure out where it's complaining. Thanks, > > Sure, here's git master > Alright we've entered the "Josef throws code at the problem" portion of the event. git clone https://github.com/josefbacik/btrfs-progs.git git checkout for-marc <build> re-run fsck I wonder if the tree root is fucked and that's why it's not finding the device tree. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 21:04 ` Josef Bacik @ 2022-04-04 21:29 ` Marc MERLIN 2022-04-04 21:40 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 21:29 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 05:04:52PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote: > > > Can you build a recent btrfs-progs from git? We chucked that error > > > apparently and I can't figure out where it's complaining. Thanks, > > > > Sure, here's git master > > > > Alright we've entered the "Josef throws code at the problem" portion > of the event. thanks :) > git clone https://github.com/josefbacik/btrfs-progs.git > git checkout for-marc > <build> > re-run fsck > > I wonder if the tree root is fucked and that's why it's not finding > the device tree. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# git remote -v origin https://github.com/josefbacik/btrfs-progs.git (fetch) origin https://github.com/josefbacik/btrfs-progs.git (push) gargamel:/var/local/src/btrfs-progs-josefbacik# git branch * for-marc master gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs --version btrfs-progs v5.16.2 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree /dev/mapper/dshelf1a WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree ERROR: cannot open file system -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 21:29 ` Marc MERLIN @ 2022-04-04 21:40 ` Josef Bacik 2022-04-04 22:09 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 21:40 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 5:29 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 05:04:52PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 4:33 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Mon, Apr 04, 2022 at 03:52:15PM -0400, Josef Bacik wrote: > > > > Can you build a recent btrfs-progs from git? We chucked that error > > > > apparently and I can't figure out where it's complaining. Thanks, > > > > > > Sure, here's git master > > > > > > > Alright we've entered the "Josef throws code at the problem" portion > > of the event. > > thanks :) > > > git clone https://github.com/josefbacik/btrfs-progs.git > > git checkout for-marc > > <build> > > re-run fsck > > > > I wonder if the tree root is fucked and that's why it's not finding > > the device tree. Thanks, > > gargamel:/var/local/src/btrfs-progs-josefbacik# git remote -v > origin https://github.com/josefbacik/btrfs-progs.git (fetch) > origin https://github.com/josefbacik/btrfs-progs.git (push) > gargamel:/var/local/src/btrfs-progs-josefbacik# git branch > * for-marc > master > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs --version > btrfs-progs v5.16.2 Ok cool, lets do ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a and see what those say, it looks like the tree root is stale and can't find the root pointers for anything. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 21:40 ` Josef Bacik @ 2022-04-04 22:09 ` Marc MERLIN 2022-04-04 22:34 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 22:09 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 05:40:14PM -0400, Josef Bacik wrote: > Ok cool, lets do > > ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a > ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a > ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a > > and see what those say, it looks like the tree root is stale and can't > find the root pointers for anything. Thanks, Sure thing, the last one is slightly different (found 1602088) gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577779511296 wanted 1602089 found 1602242 parent transid verify failed on 13577779511296 wanted 1602089 found 1602242 parent transid verify failed on 13577779511296 wanted 1602089 found 1602242 Ignoring transid failure ERROR: root [1 0] level 0 does not match 1 Couldn't read tree root ERROR: cannot open file system gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure ERROR: root [1 0] level 0 does not match 1 Couldn't read tree root ERROR: cannot open file system gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577775284224 wanted 1602089 found 1602088 parent transid verify failed on 13577775284224 wanted 1602089 found 1602242 parent transid verify failed on 13577775284224 wanted 1602089 found 1602242 Ignoring transid failure ERROR: root [1 0] level 0 does not match 1 Couldn't read tree root ERROR: cannot open file system -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 22:09 ` Marc MERLIN @ 2022-04-04 22:34 ` Josef Bacik 2022-04-04 22:45 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 22:34 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 6:09 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 05:40:14PM -0400, Josef Bacik wrote: > > Ok cool, lets do > > > > ./btrfs check -r 13577779511296 /dev/mapper/dshelf1a > > ./btrfs check -r 13577821667328 /dev/mapper/dshelf1a > > ./btrfs check -r 13577775284224 /dev/mapper/dshelf1a > > > > and see what those say, it looks like the tree root is stale and can't > > find the root pointers for anything. Thanks, > > Sure thing, the last one is slightly different (found 1602088) > Alright, lets see how fucked this thing is ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 22:34 ` Josef Bacik @ 2022-04-04 22:45 ` Marc MERLIN 2022-04-04 22:52 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 22:45 UTC (permalink / raw) To: Josef Bacik Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote: > Alright, lets see how fucked this thing is > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Couldn't read chunk tree ERROR: unable to open /dev/mapper/dshelf1a -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 22:45 ` Marc MERLIN @ 2022-04-04 22:52 ` Josef Bacik 2022-04-04 23:18 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 22:52 UTC (permalink / raw) To: Marc MERLIN Cc: Andrei Borzenkov, Zygo Blaxell, linux-btrfs, Chris Murphy, Qu Wenruo On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote: > > Alright, lets see how fucked this thing is > > > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal > dump-tree -t ROOT /dev/mapper/dshelf1a > btrfs-progs v5.16.2 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Couldn't read chunk tree > ERROR: unable to open /dev/mapper/dshelf1a > idk what I expected there, you can pull my tree again, rebuild, re-run, it should work this time. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 22:52 ` Josef Bacik @ 2022-04-04 23:18 ` Marc MERLIN 2022-04-04 23:24 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 23:18 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs [removed Cc to others] On Mon, Apr 04, 2022 at 06:52:58PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote: > > > Alright, lets see how fucked this thing is > > > > > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal > > dump-tree -t ROOT /dev/mapper/dshelf1a > > btrfs-progs v5.16.2 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > Couldn't read chunk tree > > ERROR: unable to open /dev/mapper/dshelf1a > > idk what I expected there, you can pull my tree again, rebuild, > re-run, it should work this time. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree ERROR: unable to open /dev/mapper/dshelf1a -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 23:18 ` Marc MERLIN @ 2022-04-04 23:24 ` Josef Bacik 2022-04-04 23:42 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-04 23:24 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 7:18 PM Marc MERLIN <marc@merlins.org> wrote: > > [removed Cc to others] > > On Mon, Apr 04, 2022 at 06:52:58PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 6:45 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Mon, Apr 04, 2022 at 06:34:39PM -0400, Josef Bacik wrote: > > > > Alright, lets see how fucked this thing is > > > > > > > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > > > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal > > > dump-tree -t ROOT /dev/mapper/dshelf1a > > > btrfs-progs v5.16.2 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > Couldn't read chunk tree > > > ERROR: unable to open /dev/mapper/dshelf1a > > > > idk what I expected there, you can pull my tree again, rebuild, > > re-run, it should work this time. Thanks, > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > btrfs-progs v5.16.2 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > Couldn't find the last root for 4 > Couldn't setup device tree > ERROR: unable to open /dev/mapper/dshelf1a > Re-pull and try again please. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 23:24 ` Josef Bacik @ 2022-04-04 23:42 ` Marc MERLIN 2022-04-05 0:08 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-04 23:42 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 07:24:10PM -0400, Josef Bacik wrote: > Re-pull and try again please. Thanks, Looks the same? gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree ERROR: unable to open /dev/mapper/dshelf1a gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12 commit d58725072227f6bd41633cee71695b6bdc17d67f Author: Josef Bacik <josef@toxicpanda.com> Date: Mon Apr 4 19:23:40 2022 -0400 try harder to open a corrupt fs commit 1f5227d8de1dbff1b465fdff0177f833ac9df000 Author: Josef Bacik <josef@toxicpanda.com> Date: Mon Apr 4 18:52:05 2022 -0400 allow dump-tree to ignore transid mismatches -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-04 23:42 ` Marc MERLIN @ 2022-04-05 0:08 ` Josef Bacik 2022-04-05 0:13 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 0:08 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 7:42 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 07:24:10PM -0400, Josef Bacik wrote: > > Re-pull and try again please. Thanks, > > Looks the same? > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > btrfs-progs v5.16.2 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > Couldn't find the last root for 4 > Couldn't setup device tree > ERROR: unable to open /dev/mapper/dshelf1a > Fuck it, I'll just comment it out. Try it again please. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:08 ` Josef Bacik @ 2022-04-05 0:13 ` Marc MERLIN 2022-04-05 0:15 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 0:13 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 08:08:53PM -0400, Josef Bacik wrote: > Fuck it, I'll just comment it out. Try it again please. Thanks, same output :-/ gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree ERROR: unable to open /dev/mapper/dshelf1a gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12 commit 25024ece18db0a9ef554892098125f2ac5e27f97 Author: Josef Bacik <josef@toxicpanda.com> Date: Mon Apr 4 20:08:05 2022 -0400 mel:/var/local/src/btrfs-progs-josefbacik# l btrfs -rwxr-xr-x 1 root staff 6216888 Apr 4 17:11 btrfs -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:13 ` Marc MERLIN @ 2022-04-05 0:15 ` Josef Bacik 2022-04-05 0:18 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 0:15 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 8:13 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 08:08:53PM -0400, Josef Bacik wrote: > > Fuck it, I'll just comment it out. Try it again please. Thanks, > > same output :-/ > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > btrfs-progs v5.16.2 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > Couldn't find the last root for 4 > Couldn't setup device tree > ERROR: unable to open /dev/mapper/dshelf1a > gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -12 > commit 25024ece18db0a9ef554892098125f2ac5e27f97 > Author: Josef Bacik <josef@toxicpanda.com> > Date: Mon Apr 4 20:08:05 2022 -0400 > mel:/var/local/src/btrfs-progs-josefbacik# l btrfs > -rwxr-xr-x 1 root staff 6216888 Apr 4 17:11 btrfs > Hmm I wonder if the dependencies aren't done right, can you make clean make then rerun? Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:15 ` Josef Bacik @ 2022-04-05 0:18 ` Marc MERLIN 2022-04-05 0:24 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 0:18 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 08:15:45PM -0400, Josef Bacik wrote: > Hmm I wonder if the dependencies aren't done right, can you > > make clean > make > sauron [mc]# mv btrfs btrfs.orig sauron [mc]# make clean; make (...) sauron [mc]# diff btrfs btrfs.orig sauron [mc]# l btrfs btrfs.orig -rwxr-xr-x 1 root staff 6216888 Apr 4 17:17 btrfs* -rwxr-xr-x 1 root staff 6216888 Apr 4 17:11 btrfs.orig* Binary identical after rebuild. -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:18 ` Marc MERLIN @ 2022-04-05 0:24 ` Josef Bacik 2022-04-05 0:28 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 0:24 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 8:18 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 08:15:45PM -0400, Josef Bacik wrote: > > Hmm I wonder if the dependencies aren't done right, can you > > > > make clean > > make > > > sauron [mc]# mv btrfs btrfs.orig > sauron [mc]# make clean; make > (...) > sauron [mc]# diff btrfs btrfs.orig > sauron [mc]# l btrfs btrfs.orig > -rwxr-xr-x 1 root staff 6216888 Apr 4 17:17 btrfs* > -rwxr-xr-x 1 root staff 6216888 Apr 4 17:11 btrfs.orig* > > Binary identical after rebuild. Sigh time for printf sanity checks, thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:24 ` Josef Bacik @ 2022-04-05 0:28 ` Marc MERLIN 2022-04-05 0:39 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 0:28 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > Binary identical after rebuild. > > Sigh time for printf sanity checks, thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -3 commit 7839d7519ffb3b5d1d102536eb768c25a69c4c05 Author: Josef Bacik <josef@toxicpanda.com> Date: Mon Apr 4 20:24:18 2022 -0400 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55ea174c4830 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55ea174c4830 ERROR: unable to open /dev/mapper/dshelf1a -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:28 ` Marc MERLIN @ 2022-04-05 0:39 ` Josef Bacik 2022-04-05 0:58 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 0:39 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > > Binary identical after rebuild. > > > > Sigh time for printf sanity checks, thanks, > I'm dumb, try again please, thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:39 ` Josef Bacik @ 2022-04-05 0:58 ` Marc MERLIN 2022-04-05 1:06 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 0:58 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > > > Binary identical after rebuild. > > > > > > Sigh time for printf sanity checks, thanks, > > > > I'm dumb, try again please, thanks, progress :) parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x563be446b430 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x563be446b430 btrfs-progs v5.16.2 root tree node 13577814573056 level 1 items 322 free space 171 generation 1602089 owner ROOT_TREE node 13577814573056 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a key (EXTENT_TREE ROOT_ITEM 0) block 13577821667328 gen 1602089 key (278 INODE_ITEM 0) block 364978405376 gen 533 key (306 INODE_ITEM 0) block 13577300590592 gen 1601950 key (366 EXTENT_DATA 0) block 11822267351040 gen 1591706 key (428 INODE_ITEM 0) block 11970963865600 gen 1449566 key (489 EXTENT_DATA 0) block 13576971354112 gen 1601940 key (541 INODE_ITEM 0) block 364866551808 gen 1453942 key (602 EXTENT_DATA 0) block 11970666151936 gen 1601669 key (664 INODE_ITEM 0) block 11651970449408 gen 1592655 key (725 EXTENT_DATA 0) block 12511619350528 gen 1449892 key (787 INODE_ITEM 0) block 15645177659392 gen 1551626 key (848 EXTENT_DATA 0) block 11651970482176 gen 1592655 key (910 INODE_ITEM 0) block 13577663184896 gen 1602069 key (971 INODE_ITEM 0) block 15645836460032 gen 1592383 key (1032 INODE_ITEM 0) block 15645836836864 gen 1592383 key (1093 EXTENT_DATA 0) block 13577870934016 gen 1602089 key (1153 INODE_ITEM 0) block 10194600820736 gen 1601442 key (1214 INODE_ITEM 0) block 15645373874176 gen 1593692 key (1275 INODE_ITEM 0) block 13577130164224 gen 1589778 key (1336 EXTENT_DATA 0) block 15645703028736 gen 1589924 key (1398 INODE_ITEM 0) block 11652218060800 gen 1599077 key (1459 EXTENT_DATA 0) block 15645720608768 gen 1597377 key (1521 INODE_ITEM 0) block 15645373956096 gen 1593692 key (1582 EXTENT_DATA 0) block 11652218290176 gen 1599077 key (1644 INODE_ITEM 0) block 11652218503168 gen 1599077 key (1705 EXTENT_DATA 0) block 15645124706304 gen 1600922 key (1767 INODE_ITEM 0) block 15645067100160 gen 1600014 key (1828 EXTENT_DATA 0) block 13576966029312 gen 1601940 key (1890 INODE_ITEM 0) block 782933999616 gen 1599367 key (1951 EXTENT_DATA 0) block 11822337982464 gen 1599620 key (2013 INODE_ITEM 0) block 13577131016192 gen 1589778 key (2074 EXTENT_DATA 0) block 13577010249728 gen 1601941 key (2136 INODE_ITEM 0) block 10679167303680 gen 1589365 key (2197 EXTENT_DATA 0) block 13576983691264 gen 1601940 key (2259 INODE_ITEM 0) block 15645786914816 gen 1601203 key (2320 EXTENT_DATA 0) block 15645786996736 gen 1601203 key (2382 INODE_ITEM 0) block 10194603294720 gen 1601442 key (2444 INODE_ITEM 0) block 15645125050368 gen 1600922 key (2505 INODE_ITEM 0) block 15645125869568 gen 1600922 key (2566 EXTENT_DATA 0) block 13576928788480 gen 1601938 key (2628 INODE_ITEM 0) block 15645126426624 gen 1600922 key (2689 EXTENT_DATA 0) block 15645459152896 gen 1593949 key (2751 INODE_ITEM 0) block 11652048273408 gen 1601540 key (2812 EXTENT_DATA 0) block 13577100820480 gen 1600621 key (2874 INODE_ITEM 0) block 15646063067136 gen 1590067 key (2935 EXTENT_DATA 0) block 13577101197312 gen 1600621 key (2997 INODE_ITEM 0) block 13577103032320 gen 1599331 key (3058 EXTENT_DATA 0) block 13577101639680 gen 1600621 key (3120 INODE_ITEM 0) block 13577300656128 gen 1601950 key (3181 EXTENT_DATA 0) block 13577102114816 gen 1600621 key (3243 INODE_ITEM 0) block 15645910220800 gen 1600395 key (3305 INODE_ITEM 0) block 13577104310272 gen 1599331 key (3366 INODE_ITEM 0) block 15645703143424 gen 1589924 key (3427 EXTENT_DATA 0) block 13576956723200 gen 1601938 key (3489 INODE_ITEM 0) block 10194606997504 gen 1601442 key (3550 EXTENT_DATA 0) block 13577102557184 gen 1600621 key (3612 INODE_ITEM 0) block 11651868557312 gen 1600466 key (3673 EXTENT_DATA 0) block 15645099851776 gen 1594356 key (3735 INODE_ITEM 0) block 15645620404224 gen 1594485 key (3796 EXTENT_DATA 0) block 15645911203840 gen 1600394 key (3858 INODE_ITEM 0) block 11971147825152 gen 1599202 key (3919 EXTENT_DATA 0) block 13576956887040 gen 1601938 key (3981 INODE_ITEM 0) block 11971147857920 gen 1599202 key (4042 EXTENT_DATA 0) block 11971147907072 gen 1599202 key (4104 INODE_ITEM 0) block 11971147923456 gen 1599202 key (4165 EXTENT_DATA 0) block 15645947117568 gen 1594618 key (4227 INODE_ITEM 0) block 15645830856704 gen 1594078 key (4288 EXTENT_DATA 0) block 11971148087296 gen 1599202 key (4350 INODE_ITEM 0) block 11822017118208 gen 1596840 key (4411 EXTENT_DATA 0) block 11971148169216 gen 1599202 key (4473 INODE_ITEM 0) block 13577871097856 gen 1602089 key (4534 EXTENT_DATA 0) block 15645494771712 gen 1594885 key (4596 INODE_ITEM 0) block 15646063476736 gen 1594749 key (4657 EXTENT_DATA 0) block 12512415301632 gen 1601928 key (4719 INODE_ITEM 0) block 15646063509504 gen 1594749 key (4780 INODE_ITEM 0) block 15645325754368 gen 1595408 key (4842 INODE_ITEM 0) block 13577397714944 gen 1601962 key (4903 EXTENT_DATA 0) block 15645716168704 gen 1600306 key (4965 INODE_ITEM 0) block 15645668786176 gen 1601187 key (5026 EXTENT_DATA 0) block 15645827448832 gen 1596971 key (5088 INODE_ITEM 0) block 12512415940608 gen 1601928 key (5149 EXTENT_DATA 0) block 15645650518016 gen 1596339 key (5211 INODE_ITEM 0) block 15645050585088 gen 1597228 key (5272 EXTENT_DATA 0) block 15645067165696 gen 1600014 key (5334 INODE_ITEM 0) block 13577220980736 gen 1601949 key (5395 EXTENT_DATA 0) block 15645133520896 gen 1595328 key (5457 INODE_ITEM 0) block 13577497264128 gen 1595015 key (5518 EXTENT_DATA 0) block 10194258903040 gen 1596209 key (5580 INODE_ITEM 0) block 15645917265920 gen 1595143 key (5641 EXTENT_DATA 0) block 13577668739072 gen 1602069 key (5703 INODE_ITEM 0) block 15646063820800 gen 1594749 key (5764 EXTENT_DATA 0) block 15646063837184 gen 1594749 key (5826 INODE_ITEM 0) block 13577353576448 gen 1595270 key (5887 EXTENT_DATA 0) block 15646063853568 gen 1594749 key (5949 INODE_ITEM 0) block 15646063869952 gen 1594749 key (6010 EXTENT_DATA 0) block 15645831118848 gen 1594078 key (6072 INODE_ITEM 0) block 13577353658368 gen 1595270 key (6133 EXTENT_DATA 0) block 13577497919488 gen 1595015 key (6195 INODE_ITEM 0) block 13577235742720 gen 1601949 key (6256 EXTENT_DATA 0) block 10679320313856 gen 1600457 key (6318 INODE_ITEM 0) block 13577561620480 gen 1596098 key (6379 EXTENT_DATA 0) block 13577353805824 gen 1595270 key (6441 INODE_ITEM 0) block 15645132898304 gen 1596112 key (6502 EXTENT_DATA 0) block 15645650731008 gen 1596339 key (6564 INODE_ITEM 0) block 15645948575744 gen 1596187 key (6625 EXTENT_DATA 0) block 15645454974976 gen 1595753 key (6687 INODE_ITEM 0) block 13577190424576 gen 1597100 key (6749 INODE_ITEM 0) block 13577221177344 gen 1601949 key (6810 INODE_ITEM 0) block 15645668818944 gen 1601187 key (6871 EXTENT_DATA 0) block 12512416022528 gen 1601928 key (6933 INODE_ITEM 0) block 12512416071680 gen 1601928 key (6994 EXTENT_DATA 0) block 15645668868096 gen 1601187 key (7056 INODE_ITEM 0) block 13577221226496 gen 1601949 key (7117 EXTENT_DATA 0) block 15645548511232 gen 1595772 key (7179 INODE_ITEM 0) block 12512416268288 gen 1601928 key (7240 EXTENT_DATA 0) block 13577104080896 gen 1600621 key (7302 INODE_ITEM 0) block 15645326917632 gen 1595408 key (7363 EXTENT_DATA 0) block 15645326934016 gen 1595408 key (7425 INODE_ITEM 0) block 15645831577600 gen 1595539 key (7486 EXTENT_DATA 0) block 15645831610368 gen 1595539 key (7548 INODE_ITEM 0) block 13577575661568 gen 1602037 key (7609 EXTENT_DATA 0) block 15645831725056 gen 1595539 key (7671 INODE_ITEM 0) block 15645831741440 gen 1595539 key (7732 EXTENT_DATA 0) block 15645831790592 gen 1595539 key (7794 INODE_ITEM 0) block 13577104392192 gen 1600621 key (7855 EXTENT_DATA 0) block 15645831856128 gen 1595539 key (7917 INODE_ITEM 0) block 15645545578496 gen 1596008 key (7978 EXTENT_DATA 0) block 15645669097472 gen 1601187 key (8040 INODE_ITEM 0) block 13577122562048 gen 1601944 key (8101 EXTENT_DATA 0) block 15645771317248 gen 1595811 key (8163 INODE_ITEM 0) block 15645146808320 gen 1595947 key (8224 EXTENT_DATA 0) block 15645830512640 gen 1596971 key (8286 INODE_ITEM 0) block 15645146841088 gen 1595947 key (8348 INODE_ITEM 0) block 13577355280384 gen 1595270 key (8409 INODE_ITEM 0) block 15645410263040 gen 1601053 key (8470 EXTENT_DATA 0) block 12512416858112 gen 1601928 key (8532 INODE_ITEM 0) block 13577301131264 gen 1601950 key (8593 EXTENT_DATA 0) block 15645651288064 gen 1596339 key (8655 INODE_ITEM 0) block 12511560712192 gen 1600489 key (8717 INODE_ITEM 0) block 12511560925184 gen 1600489 key (8778 INODE_ITEM 0) block 15645410279424 gen 1601053 key (8839 EXTENT_DATA 0) block 13577668902912 gen 1602069 key (8901 INODE_ITEM 0) block 13577106718720 gen 1600621 key (8962 EXTENT_DATA 0) block 12511562842112 gen 1600489 key (9024 INODE_ITEM 0) block 15645410328576 gen 1601053 key (9085 EXTENT_DATA 0) block 15645669228544 gen 1601187 key (9147 INODE_ITEM 0) block 13577107308544 gen 1600621 key (9209 INODE_ITEM 0) block 15645651615744 gen 1596339 key (9270 INODE_ITEM 0) block 15645651632128 gen 1596339 key (9331 EXTENT_DATA 0) block 13577122971648 gen 1601944 key (9393 INODE_ITEM 0) block 15645051731968 gen 1597228 key (9454 EXTENT_DATA 0) block 12511564906496 gen 1600489 key (9516 INODE_ITEM 0) block 15645058105344 gen 1597228 key (9578 INODE_ITEM 0) block 15645061201920 gen 1597228 key (9639 INODE_ITEM 0) block 12511565053952 gen 1600489 key (9700 EXTENT_DATA 0) block 15645669359616 gen 1601187 key (9762 INODE_ITEM 0) block 13577550823424 gen 1602036 key (9823 EXTENT_DATA 0) block 15645410738176 gen 1601053 key (9885 INODE_ITEM 0) block 13577107865600 gen 1600621 key (9947 INODE_ITEM 0) block 13577873506304 gen 1602089 key (10008 INODE_ITEM 0) block 13577108291584 gen 1600621 key (10070 INODE_ITEM 0) block 13577108307968 gen 1600621 key (10131 INODE_ITEM 0) block 12511565660160 gen 1600489 key (10193 INODE_ITEM 0) block 12511567151104 gen 1600489 key (10254 INODE_ITEM 0) block 13577397747712 gen 1601962 key (10315 EXTENT_DATA 0) block 12511570444288 gen 1600489 key (10377 INODE_ITEM 0) block 12511570493440 gen 1600489 key (10439 INODE_ITEM 0) block 12511570657280 gen 1600489 key (10500 INODE_ITEM 0) block 15645746413568 gen 1596616 key (10561 EXTENT_DATA 0) block 15645669736448 gen 1601187 key (10623 INODE_ITEM 0) block 15645669867520 gen 1601187 key (10685 INODE_ITEM 0) block 13577669722112 gen 1602069 key (10746 INODE_ITEM 0) block 15645317120000 gen 1600752 key (10807 EXTENT_DATA 0) block 15645128458240 gen 1600922 key (10869 INODE_ITEM 0) block 15645410836480 gen 1601053 key (10931 INODE_ITEM 0) block 15645410869248 gen 1601053 key (10992 INODE_ITEM 0) block 15645669982208 gen 1601187 key (11086 INODE_ITEM 0) block 15645410934784 gen 1601053 key (11115 INODE_ITEM 0) block 13577810395136 gen 1602084 key (11177 INODE_ITEM 0) block 13577849110528 gen 1602088 key (11232 INODE_ITEM 0) block 13577828564992 gen 1602085 key (11322 ROOT_ITEM 4223) block 13577837477888 gen 1602086 key (13004 INODE_ITEM 0) block 137504833536 gen 1601371 key (14024 INODE_ITEM 0) block 13577810739200 gen 1602084 key (17642 INODE_ITEM 0) block 46358790144 gen 1601370 key (22853 INODE_ITEM 0) block 15646046289920 gen 1601329 key (23051 INODE_ITEM 0) block 12512432324608 gen 1601928 key (23201 INODE_ITEM 0) block 432390144 gen 1601369 key (23410 INODE_ITEM 0) block 15645432397824 gen 1600181 key (24093 INODE_ITEM 0) block 13577314729984 gen 1597100 key (24359 INODE_ITEM 0) block 12512246775808 gen 1592687 key (24517 INODE_ITEM 0) block 15645536681984 gen 1596008 key (25179 INODE_ITEM 0) block 15645545480192 gen 1597267 key (25524 INODE_ITEM 0) block 15645953802240 gen 1596032 key (25745 INODE_ITEM 0) block 13577867067392 gen 1602089 key (27417 INODE_ITEM 0) block 10679248257024 gen 1560751 key (27478 EXTENT_DATA 0) block 11821933953024 gen 1590261 key (31836 INODE_ITEM 0) block 15645953900544 gen 1596032 key (33174 INODE_ITEM 0) block 15645501882368 gen 1564185 key (33195 INODE_ITEM 0) block 15645501898752 gen 1564185 key (33256 EXTENT_DATA 0) block 15645501964288 gen 1564185 key (33314 INODE_ITEM 0) block 13576879210496 gen 1581661 key (33376 INODE_ITEM 0) block 13576879259648 gen 1581661 key (36142 INODE_ITEM 0) block 15645954031616 gen 1596032 key (38085 INODE_ITEM 0) block 15645954080768 gen 1596032 key (41794 INODE_ITEM 0) block 15645954129920 gen 1596032 key (47471 INODE_ITEM 0) block 15645954195456 gen 1596032 key (49726 INODE_ITEM 0) block 7283718029312 gen 1600445 key (53503 INODE_ITEM 0) block 15645536829440 gen 1596008 key (54250 INODE_ITEM 0) block 6781551640576 gen 1595865 key (56850 INODE_ITEM 0) block 11970759065600 gen 1590188 key (57663 INODE_ITEM 0) block 15645399777280 gen 1600155 key (61323 INODE_ITEM 0) block 15646041735168 gen 1601329 key (61922 INODE_ITEM 0) block 13577866543104 gen 1602089 key (72604 INODE_ITEM 0) block 15645308272640 gen 1589881 key (75479 INODE_ITEM 0) block 15645106438144 gen 1589878 key (77348 INODE_ITEM 0) block 15645308370944 gen 1589881 key (77976 INODE_ITEM 0) block 15645106585600 gen 1589878 key (78251 INODE_ITEM 0) block 11822462091264 gen 1590186 key (78355 INODE_ITEM 0) block 15645188128768 gen 1589879 key (78627 INODE_ITEM 0) block 15645566943232 gen 1559929 key (78956 INODE_ITEM 0) block 15646041817088 gen 1601329 key (80698 EXTENT_DATA 0) block 15646041882624 gen 1601329 key (84137 INODE_ITEM 0) block 15645831938048 gen 1560193 key (84476 INODE_ITEM 0) block 15646041899008 gen 1601329 key (87291 EXTENT_DATA 0) block 15645349134336 gen 1589884 key (90310 INODE_ITEM 0) block 15646041915392 gen 1601329 key (90750 INODE_ITEM 0) block 15645264510976 gen 1589880 key (91087 INODE_ITEM 0) block 15645264543744 gen 1589880 key (91252 INODE_ITEM 0) block 15646041948160 gen 1601329 key (92792 INODE_ITEM 0) block 15645308403712 gen 1589881 key (99516 INODE_ITEM 0) block 15645262626816 gen 1600937 key (111330 INODE_ITEM 0) block 15646041964544 gen 1601329 key (111397 INODE_ITEM 0) block 15646042046464 gen 1601329 key (111461 EXTENT_DATA 0) block 15646042062848 gen 1601329 key (118793 INODE_ITEM 0) block 15646042112000 gen 1601329 key (118910 INODE_ITEM 0) block 15645997793280 gen 1551263 key (119431 INODE_ITEM 0) block 15645997891584 gen 1551263 key (119738 INODE_ITEM 0) block 15646042161152 gen 1601329 key (119962 INODE_ITEM 0) block 15645661495296 gen 1557907 key (120299 INODE_ITEM 0) block 12512162103296 gen 1595662 key (120534 INODE_ITEM 0) block 6781699784704 gen 1581636 key (120854 INODE_ITEM 0) block 15645598416896 gen 1579668 key (121029 INODE_ITEM 0) block 15646053974016 gen 1600424 key (121535 INODE_ITEM 0) block 10194495127552 gen 1600453 key (125999 INODE_ITEM 0) block 15645262692352 gen 1600937 key (128782 INODE_ITEM 0) block 11160603590656 gen 1538300 key (128903 INODE_ITEM 0) block 15645277224960 gen 1553667 key (129646 INODE_ITEM 0) block 15646079418368 gen 1600430 key (130451 INODE_ITEM 0) block 15645476159488 gen 1548451 key (131414 INODE_ITEM 0) block 15645412081664 gen 1548443 key (131754 INODE_ITEM 0) block 782966816768 gen 1568473 key (132223 INODE_ITEM 0) block 13577811099648 gen 1602084 key (132420 INODE_ITEM 0) block 15645772152832 gen 1601202 key (132530 INODE_ITEM 0) block 8733501063168 gen 1595867 key (137958 INODE_ITEM 0) block 15645772201984 gen 1601202 key (157591 EXTENT_DATA 0) block 15645772234752 gen 1601202 key (159785 ROOT_ITEM 1591389) block 13577238380544 gen 1601950 key (162267 INODE_ITEM 0) block 15645772251136 gen 1601202 key (163298 ROOT_ITEM 1597234) block 11160516886528 gen 1601457 key (164620 ROOT_ITEM 1601059) block 13577821503488 gen 1602089 key (165390 ROOT_ITEM 1602066) block 13577832382464 gen 1602088 key (FREE_SPACE UNTYPED 41905291264) block 13577305997312 gen 1601950 key (FREE_SPACE UNTYPED 304435167232) block 13577236987904 gen 1601949 key (FREE_SPACE UNTYPED 568038785024) block 13577676505088 gen 1602069 key (FREE_SPACE UNTYPED 831642402816) block 13577873948672 gen 1602089 key (FREE_SPACE UNTYPED 1095246020608) block 11821928955904 gen 1599077 key (FREE_SPACE UNTYPED 1359386509312) block 13576985133056 gen 1601940 key (FREE_SPACE UNTYPED 1622990127104) block 15645538811904 gen 1600234 key (FREE_SPACE UNTYPED 1887130615808) block 13577019703296 gen 1601941 key (FREE_SPACE UNTYPED 2150734233600) block 13576865955840 gen 1601938 key (FREE_SPACE UNTYPED 2414337851392) block 15645142892544 gen 1600922 key (FREE_SPACE UNTYPED 2678478340096) block 13577306046464 gen 1601950 key (FREE_SPACE UNTYPED 2942081957888) block 13577178562560 gen 1600621 key (FREE_SPACE UNTYPED 3206222446592) block 13576867692544 gen 1601938 key (FREE_SPACE UNTYPED 3469826064384) block 10194651906048 gen 1601442 key (FREE_SPACE UNTYPED 3733966553088) block 13576867708928 gen 1601938 key (FREE_SPACE UNTYPED 3997570170880) block 11971165716480 gen 1599202 key (FREE_SPACE UNTYPED 4261710659584) block 13577148350464 gen 1601944 key (FREE_SPACE UNTYPED 4525314277376) block 13577873686528 gen 1602089 key (FREE_SPACE UNTYPED 4789454766080) block 13577399156736 gen 1601962 key (FREE_SPACE UNTYPED 5053058383872) block 12512437698560 gen 1601928 key (FREE_SPACE UNTYPED 5317198872576) block 13577237069824 gen 1601949 key (FREE_SPACE UNTYPED 5580802490368) block 13577677029376 gen 1602069 key (FREE_SPACE UNTYPED 5844406108160) block 13577503686656 gen 1595270 key (FREE_SPACE UNTYPED 6108546596864) block 13577237561344 gen 1601949 key (FREE_SPACE UNTYPED 6372150214656) block 10679368450048 gen 1600457 key (FREE_SPACE UNTYPED 6636290703360) block 13577237102592 gen 1601949 key (FREE_SPACE UNTYPED 6899894321152) block 12512438042624 gen 1601928 key (FREE_SPACE UNTYPED 7164034809856) block 13577237135360 gen 1601949 key (FREE_SPACE UNTYPED 7427638427648) block 12512439009280 gen 1601928 key (FREE_SPACE UNTYPED 7691778916352) block 13577580838912 gen 1602037 key (FREE_SPACE UNTYPED 7955382534144) block 13577178939392 gen 1600621 key (FREE_SPACE UNTYPED 8219523022848) block 13577149349888 gen 1601944 key (FREE_SPACE UNTYPED 8483126640640) block 13577306488832 gen 1601950 key (FREE_SPACE UNTYPED 8746730258432) block 12512439271424 gen 1601928 key (FREE_SPACE UNTYPED 9010870747136) block 13577677389824 gen 1602069 key (FREE_SPACE UNTYPED 9274474364928) block 15645677338624 gen 1601187 key (FREE_SPACE UNTYPED 9542909820928) block 13577149857792 gen 1601944 key (FREE_SPACE UNTYPED 9802218471424) block 12511725666304 gen 1600489 key (FREE_SPACE UNTYPED 10066358960128) block 13577561161728 gen 1602036 key (FREE_SPACE UNTYPED 10329962577920) block 13577874063360 gen 1602089 key (FREE_SPACE UNTYPED 10594103066624) block 13577399468032 gen 1601962 key (FREE_SPACE UNTYPED 10857706684416) block 15645677436928 gen 1601187 key (FREE_SPACE UNTYPED 11121847173120) block 13577677438976 gen 1602069 key (FREE_SPACE UNTYPED 11385450790912) block 15645781377024 gen 1601202 key (FREE_SPACE UNTYPED 11649591279616) block 13577820651520 gen 1602084 key (FREE_SPACE UNTYPED 11935206604800) block 13577820913664 gen 1602084 key (FREE_SPACE UNTYPED 12209614749696) block 12512439549952 gen 1601928 key (FREE_SPACE UNTYPED 12473788792832) block 13577873932288 gen 1602089 key (FREE_SPACE UNTYPED 12755142705152) block 15645976985600 gen 1596032 key (FREE_SPACE UNTYPED 13023611715584) block 8239004696576 gen 1600445 key (FREE_SPACE UNTYPED 13350197002240) block 13577871638528 gen 1602089 key (FREE_SPACE UNTYPED 13660642607104) block 15646046699520 gen 1601329 key (FREE_SPACE UNTYPED 13938775293952) block 15646046715904 gen 1601329 key (FREE_SPACE UNTYPED 14247006306304) block 15646046732288 gen 1601329 key (FREE_SPACE UNTYPED 14520810471424) block 15646046748672 gen 1601329 key (FREE_SPACE UNTYPED 14831121858560) block 15646046765056 gen 1601329 key (FREE_SPACE UNTYPED 15109220990976) block 15645276061696 gen 1600937 key (FREE_SPACE UNTYPED 15359402835968) block 13577819045888 gen 1602084 key (FREE_SPACE UNTYPED 15666492997632) block 13577823518720 gen 1602089 key (FREE_SPACE UNTYPED 15954255806464) block 13577821749248 gen 1602089 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure leaf 13577821667328 items 197 free space 4857 generation 1602242 owner EXTENT_TREE leaf 13577821667328 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (8733701406720 METADATA_ITEM 0) itemoff 16250 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 1 key (8733701423104 METADATA_ITEM 0) itemoff 16217 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 2 key (8733701439488 METADATA_ITEM 0) itemoff 16184 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 3 key (8733701455872 METADATA_ITEM 0) itemoff 16151 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 4 key (8733701472256 METADATA_ITEM 0) itemoff 16118 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 5 key (8733701488640 METADATA_ITEM 0) itemoff 16085 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 6 key (8733701505024 METADATA_ITEM 0) itemoff 16052 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 7 key (8733701521408 METADATA_ITEM 0) itemoff 16019 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 8 key (8733701537792 METADATA_ITEM 0) itemoff 15986 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 9 key (8733701554176 METADATA_ITEM 0) itemoff 15953 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 10 key (8733701570560 METADATA_ITEM 0) itemoff 15920 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 11 key (8733701586944 METADATA_ITEM 0) itemoff 15887 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 12 key (8733701603328 METADATA_ITEM 0) itemoff 15854 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 13 key (8733701619712 METADATA_ITEM 0) itemoff 15821 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 14 key (8733701636096 METADATA_ITEM 0) itemoff 15788 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 15 key (8733701652480 METADATA_ITEM 0) itemoff 15755 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 16 key (8733701668864 METADATA_ITEM 0) itemoff 15722 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 17 key (8733701685248 METADATA_ITEM 0) itemoff 15689 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 18 key (8733701701632 METADATA_ITEM 0) itemoff 15656 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 19 key (8733701718016 METADATA_ITEM 0) itemoff 15623 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 20 key (8733701734400 METADATA_ITEM 0) itemoff 15590 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 21 key (8733701750784 METADATA_ITEM 0) itemoff 15557 itemsize 33 refs 1 gen 1584113 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 22 key (8733701783552 METADATA_ITEM 0) itemoff 15524 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 23 key (8733701799936 METADATA_ITEM 0) itemoff 15491 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 24 key (8733701816320 METADATA_ITEM 0) itemoff 15458 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 25 key (8733701832704 METADATA_ITEM 0) itemoff 15425 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 26 key (8733701865472 METADATA_ITEM 0) itemoff 15392 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 27 key (8733701881856 METADATA_ITEM 0) itemoff 15359 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 28 key (8733701898240 METADATA_ITEM 1) itemoff 15326 itemsize 33 refs 1 gen 1474318 flags TREE_BLOCK tree block skinny level 1 tree block backref root CSUM_TREE item 29 key (8733701914624 METADATA_ITEM 0) itemoff 15293 itemsize 33 refs 1 gen 1474318 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 30 key (8733701931008 METADATA_ITEM 0) itemoff 15260 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 31 key (8733701947392 METADATA_ITEM 0) itemoff 15227 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 32 key (8733701963776 METADATA_ITEM 0) itemoff 15194 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 33 key (8733701980160 METADATA_ITEM 0) itemoff 15161 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 34 key (8733701996544 METADATA_ITEM 0) itemoff 15128 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 35 key (8733702012928 METADATA_ITEM 0) itemoff 15095 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 36 key (8733702029312 METADATA_ITEM 0) itemoff 15062 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 37 key (8733702045696 METADATA_ITEM 0) itemoff 15029 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 38 key (8733702062080 METADATA_ITEM 0) itemoff 14996 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 39 key (8733702078464 METADATA_ITEM 0) itemoff 14963 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 40 key (8733702094848 METADATA_ITEM 0) itemoff 14930 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 41 key (8733702111232 METADATA_ITEM 0) itemoff 14897 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 42 key (8733702127616 METADATA_ITEM 0) itemoff 14864 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 43 key (8733702144000 METADATA_ITEM 0) itemoff 14831 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 44 key (8733702176768 METADATA_ITEM 0) itemoff 14798 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 45 key (8733702193152 METADATA_ITEM 0) itemoff 14765 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 46 key (8733702209536 METADATA_ITEM 0) itemoff 14732 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 47 key (8733702225920 METADATA_ITEM 0) itemoff 14699 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 48 key (8733702242304 METADATA_ITEM 0) itemoff 14666 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 49 key (8733702258688 METADATA_ITEM 0) itemoff 14633 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 50 key (8733702275072 METADATA_ITEM 0) itemoff 14600 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 51 key (8733702291456 METADATA_ITEM 0) itemoff 14567 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 52 key (8733702307840 METADATA_ITEM 0) itemoff 14534 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 53 key (8733702324224 METADATA_ITEM 0) itemoff 14501 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 54 key (8733702340608 METADATA_ITEM 0) itemoff 14468 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 55 key (8733702356992 METADATA_ITEM 0) itemoff 14435 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 56 key (8733702389760 METADATA_ITEM 0) itemoff 14402 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 57 key (8733702406144 METADATA_ITEM 0) itemoff 14369 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 58 key (8733702422528 METADATA_ITEM 0) itemoff 14336 itemsize 33 refs 1 gen 838551 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 59 key (8733702438912 METADATA_ITEM 0) itemoff 14303 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 60 key (8733702455296 METADATA_ITEM 0) itemoff 14270 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 61 key (8733702504448 METADATA_ITEM 0) itemoff 14237 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 62 key (8733702520832 METADATA_ITEM 0) itemoff 14204 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 63 key (8733702537216 METADATA_ITEM 0) itemoff 14171 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 64 key (8733702553600 METADATA_ITEM 0) itemoff 14138 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 65 key (8733702569984 METADATA_ITEM 0) itemoff 14105 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 66 key (8733702586368 METADATA_ITEM 0) itemoff 14072 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 67 key (8733702602752 METADATA_ITEM 0) itemoff 14039 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 68 key (8733702619136 METADATA_ITEM 0) itemoff 14006 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 69 key (8733702635520 METADATA_ITEM 0) itemoff 13973 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 70 key (8733702651904 METADATA_ITEM 0) itemoff 13940 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 71 key (8733702668288 METADATA_ITEM 0) itemoff 13907 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 72 key (8733702684672 METADATA_ITEM 0) itemoff 13874 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 73 key (8733702701056 METADATA_ITEM 0) itemoff 13841 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 74 key (8733702717440 METADATA_ITEM 0) itemoff 13808 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 75 key (8733702733824 METADATA_ITEM 0) itemoff 13775 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 76 key (8733702750208 METADATA_ITEM 0) itemoff 13742 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 77 key (8733702766592 METADATA_ITEM 0) itemoff 13709 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 78 key (8733702782976 METADATA_ITEM 0) itemoff 13676 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 79 key (8733702799360 METADATA_ITEM 0) itemoff 13643 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 80 key (8733702815744 METADATA_ITEM 0) itemoff 13610 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 81 key (8733702832128 METADATA_ITEM 0) itemoff 13577 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 82 key (8733702848512 METADATA_ITEM 0) itemoff 13544 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 83 key (8733702864896 METADATA_ITEM 0) itemoff 13511 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 84 key (8733702881280 METADATA_ITEM 0) itemoff 13478 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 85 key (8733702897664 METADATA_ITEM 0) itemoff 13445 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 86 key (8733702914048 METADATA_ITEM 0) itemoff 13412 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 87 key (8733702930432 METADATA_ITEM 0) itemoff 13379 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 88 key (8733702946816 METADATA_ITEM 0) itemoff 13346 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 89 key (8733702963200 METADATA_ITEM 0) itemoff 13313 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 90 key (8733702979584 METADATA_ITEM 0) itemoff 13280 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 91 key (8733702995968 METADATA_ITEM 0) itemoff 13247 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 92 key (8733703012352 METADATA_ITEM 0) itemoff 13214 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 93 key (8733703028736 METADATA_ITEM 0) itemoff 13181 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 94 key (8733703045120 METADATA_ITEM 0) itemoff 13148 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 95 key (8733703061504 METADATA_ITEM 0) itemoff 13115 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 96 key (8733703077888 METADATA_ITEM 0) itemoff 13082 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 97 key (8733703094272 METADATA_ITEM 0) itemoff 13049 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 98 key (8733703110656 METADATA_ITEM 0) itemoff 13016 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 99 key (8733703127040 METADATA_ITEM 0) itemoff 12983 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 100 key (8733703143424 METADATA_ITEM 0) itemoff 12950 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 101 key (8733703159808 METADATA_ITEM 0) itemoff 12917 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 102 key (8733703176192 METADATA_ITEM 0) itemoff 12884 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 103 key (8733703192576 METADATA_ITEM 0) itemoff 12851 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 104 key (8733703208960 METADATA_ITEM 0) itemoff 12818 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 105 key (8733703225344 METADATA_ITEM 0) itemoff 12785 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 106 key (8733703241728 METADATA_ITEM 0) itemoff 12752 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 107 key (8733703258112 METADATA_ITEM 0) itemoff 12719 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 108 key (8733703274496 METADATA_ITEM 0) itemoff 12686 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 109 key (8733703290880 METADATA_ITEM 0) itemoff 12653 itemsize 33 refs 1 gen 1584112 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 110 key (8733703323648 METADATA_ITEM 0) itemoff 12620 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 111 key (8733703340032 METADATA_ITEM 0) itemoff 12587 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 112 key (8733703454720 METADATA_ITEM 0) itemoff 12554 itemsize 33 refs 1 gen 1535773 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 113 key (8733703471104 METADATA_ITEM 0) itemoff 12521 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 114 key (8733703487488 METADATA_ITEM 0) itemoff 12488 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 115 key (8733703503872 METADATA_ITEM 0) itemoff 12455 itemsize 33 refs 1 gen 519869 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 116 key (8733703520256 METADATA_ITEM 0) itemoff 12422 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 117 key (8733703536640 METADATA_ITEM 0) itemoff 12389 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 118 key (8733703553024 METADATA_ITEM 0) itemoff 12356 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 119 key (8733703569408 METADATA_ITEM 0) itemoff 12323 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 120 key (8733703602176 METADATA_ITEM 0) itemoff 12290 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 121 key (8733703651328 METADATA_ITEM 0) itemoff 12257 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 122 key (8733703733248 METADATA_ITEM 0) itemoff 12224 itemsize 33 refs 1 gen 1510262 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 123 key (8733703749632 METADATA_ITEM 0) itemoff 12191 itemsize 33 refs 1 gen 1601423 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 124 key (8733703766016 METADATA_ITEM 0) itemoff 12158 itemsize 33 refs 1 gen 1509558 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 125 key (8733703798784 METADATA_ITEM 0) itemoff 12125 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 126 key (8733703815168 METADATA_ITEM 0) itemoff 12092 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 127 key (8733703831552 METADATA_ITEM 0) itemoff 12059 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 128 key (8733703847936 METADATA_ITEM 0) itemoff 12026 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 129 key (8733703864320 METADATA_ITEM 0) itemoff 11993 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 130 key (8733703880704 METADATA_ITEM 0) itemoff 11960 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 131 key (8733703897088 METADATA_ITEM 0) itemoff 11927 itemsize 33 refs 1 gen 1518676 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 132 key (8733703913472 METADATA_ITEM 0) itemoff 11894 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 133 key (8733703929856 METADATA_ITEM 0) itemoff 11861 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 134 key (8733703946240 METADATA_ITEM 0) itemoff 11828 itemsize 33 refs 1 gen 1590258 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11223 item 135 key (8733703979008 METADATA_ITEM 0) itemoff 11795 itemsize 33 refs 1 gen 749275 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 136 key (8733704060928 METADATA_ITEM 0) itemoff 11762 itemsize 33 refs 1 gen 1509163 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 137 key (8733704126464 METADATA_ITEM 0) itemoff 11729 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 138 key (8733704142848 METADATA_ITEM 0) itemoff 11696 itemsize 33 refs 1 gen 1496691 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 139 key (8733704159232 METADATA_ITEM 0) itemoff 11663 itemsize 33 refs 1 gen 1496691 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 140 key (8733704241152 METADATA_ITEM 0) itemoff 11630 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 141 key (8733704257536 METADATA_ITEM 0) itemoff 11597 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 142 key (8733704273920 METADATA_ITEM 0) itemoff 11564 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 143 key (8733704290304 METADATA_ITEM 0) itemoff 11531 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 144 key (8733704306688 METADATA_ITEM 0) itemoff 11498 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 145 key (8733704323072 METADATA_ITEM 0) itemoff 11465 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 146 key (8733704355840 METADATA_ITEM 0) itemoff 11432 itemsize 33 refs 1 gen 1526001 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 147 key (8733704404992 METADATA_ITEM 0) itemoff 11399 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 148 key (8733704421376 METADATA_ITEM 0) itemoff 11366 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 149 key (8733704437760 METADATA_ITEM 0) itemoff 11333 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 150 key (8733704454144 METADATA_ITEM 0) itemoff 11300 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 151 key (8733704470528 METADATA_ITEM 0) itemoff 11267 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 152 key (8733704519680 METADATA_ITEM 0) itemoff 11234 itemsize 33 refs 1 gen 1497583 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 153 key (8733704536064 METADATA_ITEM 0) itemoff 11201 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 154 key (8733704617984 METADATA_ITEM 0) itemoff 11168 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 155 key (8733704667136 METADATA_ITEM 0) itemoff 11135 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 156 key (8733704683520 METADATA_ITEM 0) itemoff 11102 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 157 key (8733704716288 METADATA_ITEM 0) itemoff 11069 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 158 key (8733704765440 METADATA_ITEM 0) itemoff 11036 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 159 key (8733704798208 METADATA_ITEM 0) itemoff 11003 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 160 key (8733704847360 METADATA_ITEM 0) itemoff 10970 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 161 key (8733704863744 METADATA_ITEM 0) itemoff 10937 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 162 key (8733704880128 METADATA_ITEM 0) itemoff 10904 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 163 key (8733704912896 METADATA_ITEM 0) itemoff 10871 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 164 key (8733704945664 METADATA_ITEM 0) itemoff 10838 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 165 key (8733704962048 METADATA_ITEM 0) itemoff 10805 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 166 key (8733704978432 METADATA_ITEM 0) itemoff 10772 itemsize 33 refs 1 gen 736618 flags TREE_BLOCK tree block skinny level 0 shared block backref parent 10679198629888 item 167 key (8733705043968 METADATA_ITEM 0) itemoff 10739 itemsize 33 refs 1 gen 1554576 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 168 key (8733705093120 METADATA_ITEM 0) itemoff 10706 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 169 key (8733705109504 METADATA_ITEM 0) itemoff 10673 itemsize 33 refs 1 gen 1554576 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 170 key (8733705125888 METADATA_ITEM 0) itemoff 10640 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 171 key (8733705142272 METADATA_ITEM 0) itemoff 10607 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 172 key (8733705175040 METADATA_ITEM 0) itemoff 10574 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 173 key (8733705191424 METADATA_ITEM 0) itemoff 10541 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 174 key (8733705207808 METADATA_ITEM 0) itemoff 10508 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 175 key (8733705224192 METADATA_ITEM 0) itemoff 10475 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 176 key (8733705240576 METADATA_ITEM 0) itemoff 10442 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 177 key (8733705256960 METADATA_ITEM 0) itemoff 10409 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 178 key (8733705273344 METADATA_ITEM 0) itemoff 10376 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 179 key (8733705289728 METADATA_ITEM 0) itemoff 10343 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 180 key (8733705306112 METADATA_ITEM 0) itemoff 10310 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 181 key (8733705322496 METADATA_ITEM 0) itemoff 10277 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 182 key (8733705338880 METADATA_ITEM 0) itemoff 10244 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 183 key (8733705453568 METADATA_ITEM 0) itemoff 10211 itemsize 33 refs 1 gen 68949 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 184 key (8733705469952 METADATA_ITEM 0) itemoff 10178 itemsize 33 refs 1 gen 904828 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 185 key (8733705502720 METADATA_ITEM 0) itemoff 10145 itemsize 33 refs 1 gen 1530500 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 186 key (8733705535488 METADATA_ITEM 0) itemoff 10112 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 187 key (8733705551872 METADATA_ITEM 0) itemoff 10079 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 188 key (8733705568256 METADATA_ITEM 0) itemoff 10046 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 189 key (8733705584640 METADATA_ITEM 0) itemoff 10013 itemsize 33 refs 1 gen 1530500 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 190 key (8733705601024 METADATA_ITEM 0) itemoff 9980 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 191 key (8733705633792 METADATA_ITEM 0) itemoff 9947 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 192 key (8733705650176 METADATA_ITEM 0) itemoff 9914 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 193 key (8733705666560 METADATA_ITEM 0) itemoff 9881 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 194 key (8733705682944 METADATA_ITEM 0) itemoff 9848 itemsize 33 refs 1 gen 904828 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 195 key (8733705699328 METADATA_ITEM 0) itemoff 9815 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 196 key (8733705715712 METADATA_ITEM 0) itemoff 9782 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE leaf 364978405376 items 56 free space 8919 generation 533 owner ROOT_TREE leaf 364978405376 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (278 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 38 transid 38 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740280.173040226 (2018-02-15 16:18:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 1 key (278 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 38 type 1 (regular) extent data disk byte 26621038592 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (279 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 38 transid 38 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740280.173040226 (2018-02-15 16:18:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 3 key (279 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 38 type 1 (regular) extent data disk byte 26621300736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (280 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 38 transid 38 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740280.173040226 (2018-02-15 16:18:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 5 key (280 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 38 type 1 (regular) extent data disk byte 26621562880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (281 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 40 transid 40 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740347.756116870 (2018-02-15 16:19:07) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 7 key (281 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 40 type 1 (regular) extent data disk byte 30292549632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (282 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 41 transid 41 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740380.519669272 (2018-02-15 16:19:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 9 key (282 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 41 type 1 (regular) extent data disk byte 32163766272 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (283 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 41 transid 41 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740380.519669272 (2018-02-15 16:19:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 11 key (283 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 41 type 1 (regular) extent data disk byte 32164028416 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (284 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 41 transid 41 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740380.519669272 (2018-02-15 16:19:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 13 key (284 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 41 type 1 (regular) extent data disk byte 32164290560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (285 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 42 transid 42 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740419.539136240 (2018-02-15 16:20:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 15 key (285 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 42 type 1 (regular) extent data disk byte 34247188480 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (286 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 43 transid 43 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740452.214689894 (2018-02-15 16:20:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 17 key (286 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 43 type 1 (regular) extent data disk byte 35423563776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (287 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 42 transid 42 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740419.543136185 (2018-02-15 16:20:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 19 key (287 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 42 type 1 (regular) extent data disk byte 34247712768 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (288 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 44 transid 44 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740485.94240785 (2018-02-15 16:21:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 21 key (288 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 44 type 1 (regular) extent data disk byte 37536821248 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (289 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 45 transid 45 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740521.829739034 (2018-02-15 16:22:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 23 key (289 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 45 type 1 (regular) extent data disk byte 39703162880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (290 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 44 transid 44 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740485.94240785 (2018-02-15 16:21:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 25 key (290 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 44 type 1 (regular) extent data disk byte 37537345536 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (291 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 46 transid 46 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740560.741207592 (2018-02-15 16:22:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 27 key (291 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 46 type 1 (regular) extent data disk byte 41816502272 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (292 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 47 transid 47 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740603.752620191 (2018-02-15 16:23:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 29 key (292 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 47 type 1 (regular) extent data disk byte 44754206720 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (293 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 47 transid 47 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740603.752620191 (2018-02-15 16:23:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 31 key (293 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 47 type 1 (regular) extent data disk byte 44754468864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (294 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 46 transid 46 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740560.741207592 (2018-02-15 16:22:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 33 key (294 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 46 type 1 (regular) extent data disk byte 41817288704 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (295 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 48 transid 48 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740636.584171840 (2018-02-15 16:23:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 35 key (295 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 48 type 1 (regular) extent data disk byte 45620813824 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (296 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 47 transid 47 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740603.756620136 (2018-02-15 16:23:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 37 key (296 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 47 type 1 (regular) extent data disk byte 44754993152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (297 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 47 transid 47 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740603.756620136 (2018-02-15 16:23:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 39 key (297 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 47 type 1 (regular) extent data disk byte 44755255296 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (298 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 49 transid 49 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740669.327724712 (2018-02-15 16:24:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 41 key (298 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 49 type 1 (regular) extent data disk byte 46119051264 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (299 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 51 transid 51 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740736.874802394 (2018-02-15 16:25:36) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 43 key (299 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 51 type 1 (regular) extent data disk byte 47770337280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (300 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 53 transid 53 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740812.653767782 (2018-02-15 16:26:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 45 key (300 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 53 type 1 (regular) extent data disk byte 52100096000 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (301 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 52 transid 52 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740777.838243105 (2018-02-15 16:26:17) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 47 key (301 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 52 type 1 (regular) extent data disk byte 50697764864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (302 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 52 transid 52 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740777.838243105 (2018-02-15 16:26:17) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 49 key (302 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 52 type 1 (regular) extent data disk byte 50698027008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (303 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 54 transid 54 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740843.369348452 (2018-02-15 16:27:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 51 key (303 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 54 type 1 (regular) extent data disk byte 53655924736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (304 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 55 transid 55 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740874.100928922 (2018-02-15 16:27:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 53 key (304 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 55 type 1 (regular) extent data disk byte 55110717440 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (305 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 54 transid 54 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740843.369348452 (2018-02-15 16:27:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 55 key (305 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 54 type 1 (regular) extent data disk byte 53656453120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) parent transid verify failed on 13577300590592 wanted 1601950 found 1602229 leaf 13577300590592 items 119 free space 581 generation 1601950 owner ROOT_TREE leaf 13577300590592 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (306 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 56 transid 56 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740904.920508209 (2018-02-15 16:28:24) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 1 key (306 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 56 type 1 (regular) extent data disk byte 55728295936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (307 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 57 transid 57 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740937.620061853 (2018-02-15 16:28:57) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 3 key (307 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 57 type 1 (regular) extent data disk byte 56279109632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (308 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 59 transid 59 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518740999.155221937 (2018-02-15 16:29:59) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 5 key (308 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 59 type 1 (regular) extent data disk byte 58089660416 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (309 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 60 transid 60 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741031.798776402 (2018-02-15 16:30:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 7 key (309 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 60 type 1 (regular) extent data disk byte 59047624704 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (310 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 61 transid 61 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741064.558329304 (2018-02-15 16:31:04) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 9 key (310 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 61 type 1 (regular) extent data disk byte 60360396800 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (311 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 1601940 transid 1601940 size 131072 nbytes 59050164224 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 450517 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1648366373.835066464 (2022-03-27 00:32:53) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 11 key (311 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 1601940 type 1 (regular) extent data disk byte 4552318590976 nr 131072 extent data offset 0 nr 131072 ram 131072 extent compression 0 (none) item 12 key (312 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 62 transid 62 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741097.333882006 (2018-02-15 16:31:37) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 13 key (312 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 62 type 1 (regular) extent data disk byte 61363699712 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (313 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 63 transid 63 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741132.157406780 (2018-02-15 16:32:12) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 15 key (313 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 63 type 1 (regular) extent data disk byte 62069760000 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (314 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 64 transid 64 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741164.920959686 (2018-02-15 16:32:44) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 17 key (314 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 64 type 1 (regular) extent data disk byte 62734540800 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (315 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 69 transid 69 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741339.2584470 (2018-02-15 16:35:39) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 19 key (315 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 69 type 1 (regular) extent data disk byte 69593169920 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (316 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 65 transid 65 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741197.684512612 (2018-02-15 16:33:17) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 21 key (316 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 65 type 1 (regular) extent data disk byte 63950589952 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (317 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 69 transid 69 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741339.2584470 (2018-02-15 16:35:39) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 23 key (317 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 69 type 1 (regular) extent data disk byte 69593956352 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (318 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 67 transid 67 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741265.271590412 (2018-02-15 16:34:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 25 key (318 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 67 type 1 (regular) extent data disk byte 66273447936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (319 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 69 transid 69 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741339.2584470 (2018-02-15 16:35:39) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 27 key (319 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 69 type 1 (regular) extent data disk byte 69594218496 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (320 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 87 transid 87 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741855.83545862 (2018-02-15 16:44:15) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 29 key (320 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 87 type 1 (regular) extent data disk byte 79485554688 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (321 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 69 transid 69 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741339.2584470 (2018-02-15 16:35:39) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 31 key (321 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 69 type 1 (regular) extent data disk byte 69593694208 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (322 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 86 transid 86 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741823.343978624 (2018-02-15 16:43:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 33 key (322 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 86 type 1 (regular) extent data disk byte 76264198144 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (323 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 86 transid 86 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741823.347978570 (2018-02-15 16:43:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 35 key (323 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 86 type 1 (regular) extent data disk byte 76264591360 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (324 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 76 transid 76 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741568.379455572 (2018-02-15 16:39:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 37 key (324 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 76 type 1 (regular) extent data disk byte 73456918528 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (325 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 86 transid 86 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741823.347978570 (2018-02-15 16:43:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 39 key (325 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 86 type 1 (regular) extent data disk byte 77337407488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (326 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 1601950 transid 1601950 size 131072 nbytes 54084239360 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 412630 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1648367018.70270714 (2022-03-27 00:43:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 41 key (326 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 1601950 type 1 (regular) extent data disk byte 937112211456 nr 131072 extent data offset 0 nr 131072 ram 131072 extent compression 0 (none) item 42 key (327 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 86 transid 86 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741823.347978570 (2018-02-15 16:43:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 43 key (327 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 86 type 1 (regular) extent data disk byte 77337669632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (328 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 1448946 transid 1448946 size 262144 nbytes 28573696 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 109 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699040.284833350 (2020-07-13 20:57:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 45 key (328 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 1448946 type 1 (regular) extent data disk byte 385896763392 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (329 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 88 transid 88 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741889.903071124 (2018-02-15 16:44:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 47 key (329 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 88 type 1 (regular) extent data disk byte 81740009472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (331 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 87 transid 87 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741855.83545862 (2018-02-15 16:44:15) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 49 key (331 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 87 type 1 (regular) extent data disk byte 80142901248 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (332 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 89 transid 89 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741922.666624432 (2018-02-15 16:45:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 51 key (332 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 89 type 1 (regular) extent data disk byte 83344203776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (333 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 90 transid 90 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741957.478149836 (2018-02-15 16:45:57) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 53 key (333 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 90 type 1 (regular) extent data disk byte 84745506816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (334 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 1420 transid 1420 size 262144 nbytes 17301504 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 66 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518797932.463632141 (2018-02-16 08:18:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 55 key (334 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 1420 type 1 (regular) extent data disk byte 156876902400 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (335 INODE_ITEM 0) itemoff 10159 itemsize 160 generation 90 transid 90 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741957.482149782 (2018-02-15 16:45:57) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 57 key (335 EXTENT_DATA 0) itemoff 10106 itemsize 53 generation 90 type 1 (regular) extent data disk byte 84746031104 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (336 INODE_ITEM 0) itemoff 9946 itemsize 160 generation 91 transid 91 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518741992.301675095 (2018-02-15 16:46:32) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 59 key (336 EXTENT_DATA 0) itemoff 9893 itemsize 53 generation 91 type 1 (regular) extent data disk byte 85901545472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 60 key (337 INODE_ITEM 0) itemoff 9733 itemsize 160 generation 101 transid 101 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742342.504901802 (2018-02-15 16:52:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 61 key (337 EXTENT_DATA 0) itemoff 9680 itemsize 53 generation 101 type 1 (regular) extent data disk byte 85902106624 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 62 key (338 INODE_ITEM 0) itemoff 9520 itemsize 160 generation 1475 transid 1475 size 262144 nbytes 13369344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518800228.376440697 (2018-02-16 08:57:08) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 63 key (338 EXTENT_DATA 0) itemoff 9467 itemsize 53 generation 1475 type 1 (regular) extent data disk byte 1706741551104 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 64 key (339 INODE_ITEM 0) itemoff 9307 itemsize 160 generation 104 transid 104 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.807562229 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 65 key (339 EXTENT_DATA 0) itemoff 9254 itemsize 53 generation 104 type 1 (regular) extent data disk byte 95592022016 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 66 key (340 INODE_ITEM 0) itemoff 9094 itemsize 160 generation 104 transid 104 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.811562175 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 67 key (340 EXTENT_DATA 0) itemoff 9041 itemsize 53 generation 104 type 1 (regular) extent data disk byte 96665616384 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 68 key (341 INODE_ITEM 0) itemoff 8881 itemsize 160 generation 104 transid 104 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.811562175 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 69 key (341 EXTENT_DATA 0) itemoff 8828 itemsize 53 generation 104 type 1 (regular) extent data disk byte 97739059200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 70 key (342 INODE_ITEM 0) itemoff 8668 itemsize 160 generation 104 transid 104 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.811562175 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 71 key (342 EXTENT_DATA 0) itemoff 8615 itemsize 53 generation 104 type 1 (regular) extent data disk byte 97739583488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 72 key (343 INODE_ITEM 0) itemoff 8455 itemsize 160 generation 105 transid 105 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742481.771004057 (2018-02-15 16:54:41) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 73 key (343 EXTENT_DATA 0) itemoff 8402 itemsize 53 generation 105 type 1 (regular) extent data disk byte 109406760960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 74 key (344 INODE_ITEM 0) itemoff 8242 itemsize 160 generation 104 transid 104 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.811562175 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 75 key (344 EXTENT_DATA 0) itemoff 8189 itemsize 53 generation 104 type 1 (regular) extent data disk byte 99886931968 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 76 key (345 INODE_ITEM 0) itemoff 8029 itemsize 160 generation 104 transid 104 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742440.811562175 (2018-02-15 16:54:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 77 key (345 EXTENT_DATA 0) itemoff 7976 itemsize 53 generation 104 type 1 (regular) extent data disk byte 100959903744 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 78 key (346 INODE_ITEM 0) itemoff 7816 itemsize 160 generation 106 transid 106 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742514.534557633 (2018-02-15 16:55:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 79 key (346 EXTENT_DATA 0) itemoff 7763 itemsize 53 generation 106 type 1 (regular) extent data disk byte 113845682176 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 80 key (347 INODE_ITEM 0) itemoff 7603 itemsize 160 generation 106 transid 106 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742514.538557578 (2018-02-15 16:55:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 81 key (347 EXTENT_DATA 0) itemoff 7550 itemsize 53 generation 106 type 1 (regular) extent data disk byte 115284467712 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 82 key (348 INODE_ITEM 0) itemoff 7390 itemsize 160 generation 106 transid 106 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742514.538557578 (2018-02-15 16:55:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 83 key (348 EXTENT_DATA 0) itemoff 7337 itemsize 53 generation 106 type 1 (regular) extent data disk byte 115284729856 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 84 key (349 INODE_ITEM 0) itemoff 7177 itemsize 160 generation 107 transid 107 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 85 key (349 EXTENT_DATA 0) itemoff 7124 itemsize 53 generation 107 type 1 (regular) extent data disk byte 113621704704 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 86 key (350 INODE_ITEM 0) itemoff 6964 itemsize 160 generation 107 transid 107 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 87 key (350 EXTENT_DATA 0) itemoff 6911 itemsize 53 generation 107 type 1 (regular) extent data disk byte 113623695360 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 88 key (351 INODE_ITEM 0) itemoff 6751 itemsize 160 generation 106 transid 106 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742514.538557578 (2018-02-15 16:55:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 89 key (351 EXTENT_DATA 0) itemoff 6698 itemsize 53 generation 106 type 1 (regular) extent data disk byte 115285516288 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 90 key (352 INODE_ITEM 0) itemoff 6538 itemsize 160 generation 106 transid 106 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742514.538557578 (2018-02-15 16:55:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 91 key (352 EXTENT_DATA 0) itemoff 6485 itemsize 53 generation 106 type 1 (regular) extent data disk byte 115285778432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 92 key (353 INODE_ITEM 0) itemoff 6325 itemsize 160 generation 107 transid 107 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 93 key (353 EXTENT_DATA 0) itemoff 6272 itemsize 53 generation 107 type 1 (regular) extent data disk byte 115993206784 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 94 key (354 INODE_ITEM 0) itemoff 6112 itemsize 160 generation 108 transid 108 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742582.121636759 (2018-02-15 16:56:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 95 key (354 EXTENT_DATA 0) itemoff 6059 itemsize 53 generation 108 type 1 (regular) extent data disk byte 118523154432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 96 key (355 INODE_ITEM 0) itemoff 5899 itemsize 160 generation 103 transid 103 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742408.48008631 (2018-02-15 16:53:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 97 key (355 EXTENT_DATA 0) itemoff 5846 itemsize 53 generation 103 type 1 (regular) extent data disk byte 109407629312 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 98 key (356 INODE_ITEM 0) itemoff 5686 itemsize 160 generation 107 transid 107 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 99 key (356 EXTENT_DATA 0) itemoff 5633 itemsize 53 generation 107 type 1 (regular) extent data disk byte 116802277376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 100 key (357 INODE_ITEM 0) itemoff 5473 itemsize 160 generation 102 transid 102 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742375.276455209 (2018-02-15 16:52:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 101 key (357 EXTENT_DATA 0) itemoff 5420 itemsize 53 generation 102 type 1 (regular) extent data disk byte 90223198208 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 102 key (358 INODE_ITEM 0) itemoff 5260 itemsize 160 generation 107 transid 107 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 103 key (358 EXTENT_DATA 0) itemoff 5207 itemsize 53 generation 107 type 1 (regular) extent data disk byte 116802539520 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 104 key (359 INODE_ITEM 0) itemoff 5047 itemsize 160 generation 107 transid 107 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742549.354083210 (2018-02-15 16:55:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 105 key (359 EXTENT_DATA 0) itemoff 4994 itemsize 53 generation 107 type 1 (regular) extent data disk byte 116802801664 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 106 key (360 INODE_ITEM 0) itemoff 4834 itemsize 160 generation 1453981 transid 1453981 size 262144 nbytes 8388608 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 32 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594951634.324611316 (2020-07-16 19:07:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 107 key (360 EXTENT_DATA 0) itemoff 4781 itemsize 53 generation 1453981 type 1 (regular) extent data disk byte 749642706944 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 108 key (361 INODE_ITEM 0) itemoff 4621 itemsize 160 generation 109 transid 109 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742616.937162420 (2018-02-15 16:56:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 109 key (361 EXTENT_DATA 0) itemoff 4568 itemsize 53 generation 109 type 1 (regular) extent data disk byte 119953424384 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 110 key (362 INODE_ITEM 0) itemoff 4408 itemsize 160 generation 108 transid 108 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742582.125636704 (2018-02-15 16:56:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 111 key (362 EXTENT_DATA 0) itemoff 4355 itemsize 53 generation 108 type 1 (regular) extent data disk byte 118523940864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 112 key (363 INODE_ITEM 0) itemoff 4195 itemsize 160 generation 108 transid 108 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742582.125636704 (2018-02-15 16:56:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 113 key (363 EXTENT_DATA 0) itemoff 4142 itemsize 53 generation 108 type 1 (regular) extent data disk byte 118524203008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 114 key (364 INODE_ITEM 0) itemoff 3982 itemsize 160 generation 109 transid 109 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742616.937162420 (2018-02-15 16:56:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 115 key (364 EXTENT_DATA 0) itemoff 3929 itemsize 53 generation 109 type 1 (regular) extent data disk byte 119953686528 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 116 key (365 INODE_ITEM 0) itemoff 3769 itemsize 160 generation 108 transid 108 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742582.125636704 (2018-02-15 16:56:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 117 key (365 EXTENT_DATA 0) itemoff 3716 itemsize 53 generation 108 type 1 (regular) extent data disk byte 118524727296 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 118 key (366 INODE_ITEM 0) itemoff 3556 itemsize 160 generation 110 transid 110 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742647.652743952 (2018-02-15 16:57:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) leaf 11822267351040 items 123 free space 162 generation 1591706 owner ROOT_TREE leaf 11822267351040 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (366 EXTENT_DATA 0) itemoff 16230 itemsize 53 generation 110 type 1 (regular) extent data disk byte 121487896576 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 1 key (367 INODE_ITEM 0) itemoff 16070 itemsize 160 generation 109 transid 109 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742616.937162420 (2018-02-15 16:56:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 2 key (367 EXTENT_DATA 0) itemoff 16017 itemsize 53 generation 109 type 1 (regular) extent data disk byte 119954210816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 3 key (368 INODE_ITEM 0) itemoff 15857 itemsize 160 generation 109 transid 109 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742616.941162365 (2018-02-15 16:56:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 4 key (368 EXTENT_DATA 0) itemoff 15804 itemsize 53 generation 109 type 1 (regular) extent data disk byte 119954472960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 5 key (369 INODE_ITEM 0) itemoff 15644 itemsize 160 generation 1475 transid 1475 size 262144 nbytes 14155776 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 54 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518800228.384440588 (2018-02-16 08:57:08) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 6 key (369 EXTENT_DATA 0) itemoff 15591 itemsize 53 generation 1475 type 1 (regular) extent data disk byte 1728215785472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 7 key (370 INODE_ITEM 0) itemoff 15431 itemsize 160 generation 111 transid 111 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742678.376325386 (2018-02-15 16:57:58) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 8 key (370 EXTENT_DATA 0) itemoff 15378 itemsize 53 generation 111 type 1 (regular) extent data disk byte 123032813568 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 9 key (371 INODE_ITEM 0) itemoff 15218 itemsize 160 generation 112 transid 112 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742711.143878988 (2018-02-15 16:58:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 10 key (371 EXTENT_DATA 0) itemoff 15165 itemsize 53 generation 112 type 1 (regular) extent data disk byte 124644073472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 11 key (372 INODE_ITEM 0) itemoff 15005 itemsize 160 generation 112 transid 112 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742711.147878934 (2018-02-15 16:58:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 12 key (372 EXTENT_DATA 0) itemoff 14952 itemsize 53 generation 112 type 1 (regular) extent data disk byte 124644335616 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 13 key (373 INODE_ITEM 0) itemoff 14792 itemsize 160 generation 1448951 transid 1448951 size 262144 nbytes 16252928 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 62 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699242.6790619 (2020-07-13 21:00:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 14 key (373 EXTENT_DATA 0) itemoff 14739 itemsize 53 generation 1448951 type 1 (regular) extent data disk byte 385906561024 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 15 key (374 INODE_ITEM 0) itemoff 14579 itemsize 160 generation 112 transid 112 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742711.147878934 (2018-02-15 16:58:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 16 key (374 EXTENT_DATA 0) itemoff 14526 itemsize 53 generation 112 type 1 (regular) extent data disk byte 124644859904 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 17 key (375 INODE_ITEM 0) itemoff 14366 itemsize 160 generation 114 transid 114 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742776.678986230 (2018-02-15 16:59:36) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 18 key (375 EXTENT_DATA 0) itemoff 14313 itemsize 53 generation 114 type 1 (regular) extent data disk byte 127791153152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 19 key (376 INODE_ITEM 0) itemoff 14153 itemsize 160 generation 1448958 transid 1448958 size 262144 nbytes 4718592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699388.446423495 (2020-07-13 21:03:08) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 20 key (376 EXTENT_DATA 0) itemoff 14100 itemsize 53 generation 1448958 type 1 (regular) extent data disk byte 385917460480 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 21 key (377 INODE_ITEM 0) itemoff 13940 itemsize 160 generation 116 transid 116 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742840.170121365 (2018-02-15 17:00:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 22 key (377 EXTENT_DATA 0) itemoff 13887 itemsize 53 generation 116 type 1 (regular) extent data disk byte 130674802688 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 23 key (378 INODE_ITEM 0) itemoff 13727 itemsize 160 generation 1448946 transid 1448946 size 262144 nbytes 18612224 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 71 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699040.284833350 (2020-07-13 20:57:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 24 key (378 EXTENT_DATA 0) itemoff 13674 itemsize 53 generation 1448946 type 1 (regular) extent data disk byte 385897598976 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 25 key (379 INODE_ITEM 0) itemoff 13514 itemsize 160 generation 117 transid 117 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742870.889702925 (2018-02-15 17:01:10) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 26 key (379 EXTENT_DATA 0) itemoff 13461 itemsize 53 generation 117 type 1 (regular) extent data disk byte 132128493568 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 27 key (380 INODE_ITEM 0) itemoff 13301 itemsize 160 generation 118 transid 118 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742901.605284551 (2018-02-15 17:01:41) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 28 key (380 EXTENT_DATA 0) itemoff 13248 itemsize 53 generation 118 type 1 (regular) extent data disk byte 133475274752 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 29 key (381 INODE_ITEM 0) itemoff 13088 itemsize 160 generation 119 transid 119 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742934.376838184 (2018-02-15 17:02:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 30 key (381 EXTENT_DATA 0) itemoff 13035 itemsize 53 generation 119 type 1 (regular) extent data disk byte 134998917120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 31 key (382 INODE_ITEM 0) itemoff 12875 itemsize 160 generation 1448948 transid 1448948 size 262144 nbytes 17825792 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 68 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699122.230374758 (2020-07-13 20:58:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 32 key (382 EXTENT_DATA 0) itemoff 12822 itemsize 53 generation 1448948 type 1 (regular) extent data disk byte 385902223360 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 33 key (383 INODE_ITEM 0) itemoff 12662 itemsize 160 generation 121 transid 121 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742995.820001329 (2018-02-15 17:03:15) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 34 key (383 EXTENT_DATA 0) itemoff 12609 itemsize 53 generation 121 type 1 (regular) extent data disk byte 138572734464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 35 key (384 INODE_ITEM 0) itemoff 12449 itemsize 160 generation 1448962 transid 1448962 size 262144 nbytes 6291456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699552.281556333 (2020-07-13 21:05:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 36 key (384 EXTENT_DATA 0) itemoff 12396 itemsize 53 generation 1448962 type 1 (regular) extent data disk byte 385924395008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 37 key (385 INODE_ITEM 0) itemoff 12236 itemsize 160 generation 122 transid 122 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743030.635527161 (2018-02-15 17:03:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 38 key (385 EXTENT_DATA 0) itemoff 12183 itemsize 53 generation 122 type 1 (regular) extent data disk byte 140095905792 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 39 key (386 INODE_ITEM 0) itemoff 12023 itemsize 160 generation 121 transid 121 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518742995.820001329 (2018-02-15 17:03:15) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 40 key (386 EXTENT_DATA 0) itemoff 11970 itemsize 53 generation 121 type 1 (regular) extent data disk byte 138573520896 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 41 key (387 INODE_ITEM 0) itemoff 11810 itemsize 160 generation 1448948 transid 1448948 size 262144 nbytes 5242880 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 20 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699122.230374758 (2020-07-13 20:58:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 42 key (387 EXTENT_DATA 0) itemoff 11757 itemsize 53 generation 1448948 type 1 (regular) extent data disk byte 385902485504 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 43 key (388 INODE_ITEM 0) itemoff 11597 itemsize 160 generation 124 transid 124 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743092.70690480 (2018-02-15 17:04:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 44 key (388 EXTENT_DATA 0) itemoff 11544 itemsize 53 generation 124 type 1 (regular) extent data disk byte 143136792576 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 45 key (389 INODE_ITEM 0) itemoff 11384 itemsize 160 generation 125 transid 125 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743122.790272128 (2018-02-15 17:05:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 46 key (389 EXTENT_DATA 0) itemoff 11331 itemsize 53 generation 125 type 1 (regular) extent data disk byte 144674807808 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 47 key (390 INODE_ITEM 0) itemoff 11171 itemsize 160 generation 1449887 transid 1449887 size 262144 nbytes 5505024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594782921.334009642 (2020-07-14 20:15:21) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 48 key (390 EXTENT_DATA 0) itemoff 11118 itemsize 53 generation 1449887 type 1 (regular) extent data disk byte 721578090496 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 49 key (391 INODE_ITEM 0) itemoff 10958 itemsize 160 generation 126 transid 126 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743155.585825517 (2018-02-15 17:05:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 50 key (391 EXTENT_DATA 0) itemoff 10905 itemsize 53 generation 126 type 1 (regular) extent data disk byte 146200510464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 51 key (392 INODE_ITEM 0) itemoff 10745 itemsize 160 generation 127 transid 127 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743186.281407513 (2018-02-15 17:06:26) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 52 key (392 EXTENT_DATA 0) itemoff 10692 itemsize 53 generation 127 type 1 (regular) extent data disk byte 147668541440 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 53 key (393 INODE_ITEM 0) itemoff 10532 itemsize 160 generation 1448949 transid 1448949 size 262144 nbytes 4194304 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699168.280995440 (2020-07-13 20:59:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 54 key (393 EXTENT_DATA 0) itemoff 10479 itemsize 53 generation 1448949 type 1 (regular) extent data disk byte 385891045376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 55 key (394 INODE_ITEM 0) itemoff 10319 itemsize 160 generation 128 transid 128 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743216.996989248 (2018-02-15 17:06:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 56 key (394 EXTENT_DATA 0) itemoff 10266 itemsize 53 generation 128 type 1 (regular) extent data disk byte 149261901824 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 57 key (395 INODE_ITEM 0) itemoff 10106 itemsize 160 generation 129 transid 129 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743247.720570883 (2018-02-15 17:07:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 58 key (395 EXTENT_DATA 0) itemoff 10053 itemsize 53 generation 129 type 1 (regular) extent data disk byte 150720843776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 59 key (396 INODE_ITEM 0) itemoff 9893 itemsize 160 generation 1448965 transid 1448965 size 262144 nbytes 5505024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699675.185917664 (2020-07-13 21:07:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 60 key (396 EXTENT_DATA 0) itemoff 9840 itemsize 53 generation 1448965 type 1 (regular) extent data disk byte 385926844416 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 61 key (397 INODE_ITEM 0) itemoff 9680 itemsize 160 generation 130 transid 130 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743278.444152528 (2018-02-15 17:07:58) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 62 key (397 EXTENT_DATA 0) itemoff 9627 itemsize 53 generation 130 type 1 (regular) extent data disk byte 151962271744 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 63 key (398 INODE_ITEM 0) itemoff 9467 itemsize 160 generation 1448956 transid 1448956 size 262144 nbytes 4718592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699294.481223871 (2020-07-13 21:01:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 64 key (398 EXTENT_DATA 0) itemoff 9414 itemsize 53 generation 1448956 type 1 (regular) extent data disk byte 385908002816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 65 key (399 INODE_ITEM 0) itemoff 9254 itemsize 160 generation 132 transid 132 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743343.975260243 (2018-02-15 17:09:03) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 66 key (399 EXTENT_DATA 0) itemoff 9201 itemsize 53 generation 132 type 1 (regular) extent data disk byte 155183456256 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 67 key (400 INODE_ITEM 0) itemoff 9041 itemsize 160 generation 1448967 transid 1448967 size 262144 nbytes 7864320 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 30 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699747.867770777 (2020-07-13 21:09:07) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 68 key (400 EXTENT_DATA 0) itemoff 8988 itemsize 53 generation 1448967 type 1 (regular) extent data disk byte 385927790592 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 69 key (401 INODE_ITEM 0) itemoff 8828 itemsize 160 generation 134 transid 134 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743409.510367947 (2018-02-15 17:10:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 70 key (401 EXTENT_DATA 0) itemoff 8775 itemsize 53 generation 134 type 1 (regular) extent data disk byte 158500700160 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 71 key (402 INODE_ITEM 0) itemoff 8615 itemsize 160 generation 134 transid 134 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743409.514367892 (2018-02-15 17:10:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 72 key (402 EXTENT_DATA 0) itemoff 8562 itemsize 53 generation 134 type 1 (regular) extent data disk byte 158500962304 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 73 key (403 INODE_ITEM 0) itemoff 8402 itemsize 160 generation 135 transid 135 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743442.277921815 (2018-02-15 17:10:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 74 key (403 EXTENT_DATA 0) itemoff 8349 itemsize 53 generation 135 type 1 (regular) extent data disk byte 160101437440 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 75 key (404 INODE_ITEM 0) itemoff 8189 itemsize 160 generation 1420 transid 1420 size 262144 nbytes 3407872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518797935.239594272 (2018-02-16 08:18:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 76 key (404 EXTENT_DATA 0) itemoff 8136 itemsize 53 generation 1420 type 1 (regular) extent data disk byte 2149658714112 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 77 key (405 INODE_ITEM 0) itemoff 7976 itemsize 160 generation 134 transid 134 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743409.514367892 (2018-02-15 17:10:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 78 key (405 EXTENT_DATA 0) itemoff 7923 itemsize 53 generation 134 type 1 (regular) extent data disk byte 158501748736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 79 key (406 INODE_ITEM 0) itemoff 7763 itemsize 160 generation 136 transid 136 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743475.49475641 (2018-02-15 17:11:15) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 80 key (406 EXTENT_DATA 0) itemoff 7710 itemsize 53 generation 136 type 1 (regular) extent data disk byte 161595084800 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 81 key (407 INODE_ITEM 0) itemoff 7550 itemsize 160 generation 2368 transid 2368 size 262144 nbytes 53215232 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 203 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518841610.511576836 (2018-02-16 20:26:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 82 key (407 EXTENT_DATA 0) itemoff 7497 itemsize 53 generation 2368 type 1 (regular) extent data disk byte 1580039909376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 83 key (408 INODE_ITEM 0) itemoff 7337 itemsize 160 generation 138 transid 138 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743540.584583433 (2018-02-15 17:12:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 84 key (408 EXTENT_DATA 0) itemoff 7284 itemsize 53 generation 138 type 1 (regular) extent data disk byte 164798357504 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 85 key (409 INODE_ITEM 0) itemoff 7124 itemsize 160 generation 137 transid 137 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743507.817029532 (2018-02-15 17:11:47) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 86 key (409 EXTENT_DATA 0) itemoff 7071 itemsize 53 generation 137 type 1 (regular) extent data disk byte 163291897856 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 87 key (410 INODE_ITEM 0) itemoff 6911 itemsize 160 generation 1448951 transid 1448951 size 262144 nbytes 6553600 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699242.6790619 (2020-07-13 21:00:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 88 key (410 EXTENT_DATA 0) itemoff 6858 itemsize 53 generation 1448951 type 1 (regular) extent data disk byte 385906823168 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 89 key (411 INODE_ITEM 0) itemoff 6698 itemsize 160 generation 140 transid 140 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743606.123691212 (2018-02-15 17:13:26) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 90 key (411 EXTENT_DATA 0) itemoff 6645 itemsize 53 generation 140 type 1 (regular) extent data disk byte 166995783680 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 91 key (412 INODE_ITEM 0) itemoff 6485 itemsize 160 generation 141 transid 141 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743640.935217321 (2018-02-15 17:14:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 92 key (412 EXTENT_DATA 0) itemoff 6432 itemsize 53 generation 141 type 1 (regular) extent data disk byte 169143394304 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 93 key (413 INODE_ITEM 0) itemoff 6272 itemsize 160 generation 1448963 transid 1448963 size 262144 nbytes 10485760 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 40 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699593.240342546 (2020-07-13 21:06:33) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 94 key (413 EXTENT_DATA 0) itemoff 6219 itemsize 53 generation 1448963 type 1 (regular) extent data disk byte 385925726208 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 95 key (414 INODE_ITEM 0) itemoff 6059 itemsize 160 generation 142 transid 142 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743671.654799143 (2018-02-15 17:14:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 96 key (414 EXTENT_DATA 0) itemoff 6006 itemsize 53 generation 142 type 1 (regular) extent data disk byte 171165818880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 97 key (415 INODE_ITEM 0) itemoff 5846 itemsize 160 generation 143 transid 143 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743704.426353042 (2018-02-15 17:15:04) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 98 key (415 EXTENT_DATA 0) itemoff 5793 itemsize 53 generation 143 type 1 (regular) extent data disk byte 171290898432 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 99 key (416 INODE_ITEM 0) itemoff 5633 itemsize 160 generation 143 transid 143 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743704.426353042 (2018-02-15 17:15:04) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 100 key (416 EXTENT_DATA 0) itemoff 5580 itemsize 53 generation 143 type 1 (regular) extent data disk byte 172363669504 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 101 key (417 INODE_ITEM 0) itemoff 5420 itemsize 160 generation 1448949 transid 1448949 size 262144 nbytes 20447232 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 78 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699168.280995440 (2020-07-13 20:59:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 102 key (417 EXTENT_DATA 0) itemoff 5367 itemsize 53 generation 1448949 type 1 (regular) extent data disk byte 385892741120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 103 key (418 INODE_ITEM 0) itemoff 5207 itemsize 160 generation 145 transid 145 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743765.865516732 (2018-02-15 17:16:05) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 104 key (418 EXTENT_DATA 0) itemoff 5154 itemsize 53 generation 145 type 1 (regular) extent data disk byte 175585861632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 105 key (419 INODE_ITEM 0) itemoff 4994 itemsize 160 generation 144 transid 144 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743735.145934883 (2018-02-15 17:15:35) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 106 key (419 EXTENT_DATA 0) itemoff 4941 itemsize 53 generation 144 type 1 (regular) extent data disk byte 174177488896 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 107 key (420 INODE_ITEM 0) itemoff 4781 itemsize 160 generation 146 transid 146 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743796.589098537 (2018-02-15 17:16:36) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 108 key (420 EXTENT_DATA 0) itemoff 4728 itemsize 53 generation 146 type 1 (regular) extent data disk byte 177198120960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 109 key (421 INODE_ITEM 0) itemoff 4568 itemsize 160 generation 1448965 transid 1448965 size 262144 nbytes 7340032 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 28 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699675.185917664 (2020-07-13 21:07:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 110 key (421 EXTENT_DATA 0) itemoff 4515 itemsize 53 generation 1448965 type 1 (regular) extent data disk byte 385927106560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 111 key (422 INODE_ITEM 0) itemoff 4355 itemsize 160 generation 147 transid 147 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743829.348652638 (2018-02-15 17:17:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 112 key (422 EXTENT_DATA 0) itemoff 4302 itemsize 53 generation 147 type 1 (regular) extent data disk byte 178797867008 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 113 key (423 INODE_ITEM 0) itemoff 4142 itemsize 160 generation 148 transid 148 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743862.124206531 (2018-02-15 17:17:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 114 key (423 EXTENT_DATA 0) itemoff 4089 itemsize 53 generation 148 type 1 (regular) extent data disk byte 179880357888 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 115 key (424 INODE_ITEM 0) itemoff 3929 itemsize 160 generation 1591706 transid 1591706 size 262144 nbytes 91750400 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 350 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1643596316.171356577 (2022-01-30 18:31:56) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 116 key (424 EXTENT_DATA 0) itemoff 3876 itemsize 53 generation 1591706 type 1 (regular) extent data disk byte 749647687680 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 117 key (425 INODE_ITEM 0) itemoff 3716 itemsize 160 generation 150 transid 150 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743927.659314565 (2018-02-15 17:18:47) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 118 key (425 EXTENT_DATA 0) itemoff 3663 itemsize 53 generation 150 type 1 (regular) extent data disk byte 183102033920 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 119 key (426 INODE_ITEM 0) itemoff 3503 itemsize 160 generation 1453661 transid 1453661 size 262144 nbytes 6815744 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 26 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594926629.456877205 (2020-07-16 12:10:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 120 key (426 EXTENT_DATA 0) itemoff 3450 itemsize 53 generation 1453661 type 1 (regular) extent data disk byte 747309912064 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 121 key (427 INODE_ITEM 0) itemoff 3290 itemsize 160 generation 151 transid 151 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743960.426868596 (2018-02-15 17:19:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 122 key (427 EXTENT_DATA 0) itemoff 3237 itemsize 53 generation 151 type 1 (regular) extent data disk byte 185019625472 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) leaf 11970963865600 items 123 free space 55 generation 1449566 owner ROOT_TREE leaf 11970963865600 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (428 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 152 transid 152 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743991.142450563 (2018-02-15 17:19:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 1 key (428 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 152 type 1 (regular) extent data disk byte 185249386496 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (429 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 152 transid 152 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518743991.142450563 (2018-02-15 17:19:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 3 key (429 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 152 type 1 (regular) extent data disk byte 186323259392 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (430 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 1448949 transid 1448949 size 262144 nbytes 7077888 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 27 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699168.280995440 (2020-07-13 20:59:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 5 key (430 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 1448949 type 1 (regular) extent data disk byte 385895763968 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (431 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 153 transid 153 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744025.957976741 (2018-02-15 17:20:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 7 key (431 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 153 type 1 (regular) extent data disk byte 188171296768 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (432 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 155 transid 155 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744089.449112687 (2018-02-15 17:21:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 9 key (432 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 155 type 1 (regular) extent data disk byte 190618001408 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (433 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 155 transid 155 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744089.449112687 (2018-02-15 17:21:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 11 key (433 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 155 type 1 (regular) extent data disk byte 190618263552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (434 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 155 transid 155 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744089.449112687 (2018-02-15 17:21:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 13 key (434 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 155 type 1 (regular) extent data disk byte 191040020480 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (435 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 1448966 transid 1448966 size 262144 nbytes 5505024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699711.4859189 (2020-07-13 21:08:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 15 key (435 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 1448966 type 1 (regular) extent data disk byte 385915817984 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (436 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 158 transid 158 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744187.751774948 (2018-02-15 17:23:07) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 17 key (436 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 158 type 1 (regular) extent data disk byte 195433910272 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (437 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 158 transid 158 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744187.751774948 (2018-02-15 17:23:07) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 19 key (437 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 158 type 1 (regular) extent data disk byte 195434172416 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (438 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 1448965 transid 1448965 size 262144 nbytes 6291456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699675.185917664 (2020-07-13 21:07:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 21 key (438 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 1448965 type 1 (regular) extent data disk byte 385927368704 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (439 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 160 transid 160 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744251.242910981 (2018-02-15 17:24:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 23 key (439 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 160 type 1 (regular) extent data disk byte 198565240832 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (440 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 160 transid 160 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744251.242910981 (2018-02-15 17:24:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 25 key (440 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 160 type 1 (regular) extent data disk byte 198565502976 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (441 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 1448966 transid 1448966 size 262144 nbytes 7602176 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 29 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699711.16858834 (2020-07-13 21:08:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 27 key (441 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 1448966 type 1 (regular) extent data disk byte 385924657152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (442 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 160 transid 160 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744251.242910981 (2018-02-15 17:24:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 29 key (442 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 160 type 1 (regular) extent data disk byte 198566027264 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (443 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 163 transid 163 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744349.541573431 (2018-02-15 17:25:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 31 key (443 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 163 type 1 (regular) extent data disk byte 203380895744 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (444 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 1449566 transid 1449566 size 262144 nbytes 5505024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594721866.262712109 (2020-07-14 03:17:46) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 33 key (444 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 1449566 type 1 (regular) extent data disk byte 386341617664 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (445 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 162 transid 162 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744314.734047048 (2018-02-15 17:25:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 35 key (445 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 162 type 1 (regular) extent data disk byte 201780236288 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (446 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 164 transid 164 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744382.309127580 (2018-02-15 17:26:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 37 key (446 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 164 type 1 (regular) extent data disk byte 204576649216 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (447 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 1448971 transid 1448971 size 262144 nbytes 6029312 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 23 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699900.442400193 (2020-07-13 21:11:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 39 key (447 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 1448971 type 1 (regular) extent data disk byte 385932079104 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (448 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 166 transid 166 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744447.852235796 (2018-02-15 17:27:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 41 key (448 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 166 type 1 (regular) extent data disk byte 208198201344 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (449 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 166 transid 166 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744447.852235796 (2018-02-15 17:27:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 43 key (449 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 166 type 1 (regular) extent data disk byte 208198463488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (450 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 1519 transid 1519 size 262144 nbytes 7864320 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 30 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518802433.118441774 (2018-02-16 09:33:53) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 45 key (450 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 1519 type 1 (regular) extent data disk byte 1145710120960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (451 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 168 transid 168 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744513.387344154 (2018-02-15 17:28:33) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 47 key (451 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 168 type 1 (regular) extent data disk byte 211231223808 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (452 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 168 transid 168 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744513.387344154 (2018-02-15 17:28:33) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 49 key (452 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 168 type 1 (regular) extent data disk byte 211231485952 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (453 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 1498 transid 1498 size 262144 nbytes 6553600 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 25 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518801363.816988131 (2018-02-16 09:16:03) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 51 key (453 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 1498 type 1 (regular) extent data disk byte 946510630912 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (454 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 170 transid 170 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744574.934476684 (2018-02-15 17:29:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 53 key (454 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 170 type 1 (regular) extent data disk byte 214238515200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (455 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 170 transid 170 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744574.934476684 (2018-02-15 17:29:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 55 key (455 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 170 type 1 (regular) extent data disk byte 214238777344 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (456 INODE_ITEM 0) itemoff 10159 itemsize 160 generation 2685 transid 2685 size 262144 nbytes 17563648 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 67 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518855339.205406132 (2018-02-17 00:15:39) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 57 key (456 EXTENT_DATA 0) itemoff 10106 itemsize 53 generation 2685 type 1 (regular) extent data disk byte 1105982980096 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (457 INODE_ITEM 0) itemoff 9946 itemsize 160 generation 172 transid 172 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744638.509573221 (2018-02-15 17:30:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 59 key (457 EXTENT_DATA 0) itemoff 9893 itemsize 53 generation 172 type 1 (regular) extent data disk byte 217410400256 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 60 key (458 INODE_ITEM 0) itemoff 9733 itemsize 160 generation 173 transid 173 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744671.85110480 (2018-02-15 17:31:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 61 key (458 EXTENT_DATA 0) itemoff 9680 itemsize 53 generation 173 type 1 (regular) extent data disk byte 218879705088 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 62 key (459 INODE_ITEM 0) itemoff 9520 itemsize 160 generation 173 transid 173 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744671.85110480 (2018-02-15 17:31:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 63 key (459 EXTENT_DATA 0) itemoff 9467 itemsize 53 generation 173 type 1 (regular) extent data disk byte 218880229376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 64 key (460 INODE_ITEM 0) itemoff 9307 itemsize 160 generation 1448949 transid 1448949 size 262144 nbytes 15466496 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 59 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699168.280995440 (2020-07-13 20:59:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 65 key (460 EXTENT_DATA 0) itemoff 9254 itemsize 53 generation 1448949 type 1 (regular) extent data disk byte 385898123264 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 66 key (461 INODE_ITEM 0) itemoff 9094 itemsize 160 generation 173 transid 173 size 262144 nbytes 262144 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744671.85110480 (2018-02-15 17:31:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 67 key (461 EXTENT_DATA 0) itemoff 9041 itemsize 53 generation 173 type 1 (regular) extent data disk byte 218880491520 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 68 key (462 INODE_ITEM 0) itemoff 8881 itemsize 160 generation 175 transid 175 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744732.520238131 (2018-02-15 17:32:12) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 69 key (462 EXTENT_DATA 0) itemoff 8828 itemsize 53 generation 175 type 1 (regular) extent data disk byte 221741686784 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 70 key (463 INODE_ITEM 0) itemoff 8668 itemsize 160 generation 1448971 transid 1448971 size 262144 nbytes 56623104 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 216 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699900.442400193 (2020-07-13 21:11:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 71 key (463 EXTENT_DATA 0) itemoff 8615 itemsize 53 generation 1448971 type 1 (regular) extent data disk byte 385932341248 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 72 key (464 INODE_ITEM 0) itemoff 8455 itemsize 160 generation 179 transid 179 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744859.842431625 (2018-02-15 17:34:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 73 key (464 EXTENT_DATA 0) itemoff 8402 itemsize 53 generation 179 type 1 (regular) extent data disk byte 224975945728 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 74 key (465 INODE_ITEM 0) itemoff 8242 itemsize 160 generation 179 transid 179 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744859.842431625 (2018-02-15 17:34:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 75 key (465 EXTENT_DATA 0) itemoff 8189 itemsize 53 generation 179 type 1 (regular) extent data disk byte 224976207872 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 76 key (466 INODE_ITEM 0) itemoff 8029 itemsize 160 generation 180 transid 180 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744892.261971940 (2018-02-15 17:34:52) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 77 key (466 EXTENT_DATA 0) itemoff 7976 itemsize 53 generation 180 type 1 (regular) extent data disk byte 226577911808 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 78 key (467 INODE_ITEM 0) itemoff 7816 itemsize 160 generation 181 transid 181 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744925.33507385 (2018-02-15 17:35:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 79 key (467 EXTENT_DATA 0) itemoff 7763 itemsize 53 generation 181 type 1 (regular) extent data disk byte 228198100992 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 80 key (468 INODE_ITEM 0) itemoff 7603 itemsize 160 generation 181 transid 181 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518744925.33507385 (2018-02-15 17:35:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 81 key (468 EXTENT_DATA 0) itemoff 7550 itemsize 53 generation 181 type 1 (regular) extent data disk byte 228198363136 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 82 key (469 INODE_ITEM 0) itemoff 7390 itemsize 160 generation 1448969 transid 1448969 size 262144 nbytes 45613056 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 174 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699831.833295032 (2020-07-13 21:10:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 83 key (469 EXTENT_DATA 0) itemoff 7337 itemsize 53 generation 1448969 type 1 (regular) extent data disk byte 385929334784 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 84 key (470 INODE_ITEM 0) itemoff 7177 itemsize 160 generation 186 transid 186 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745090.967119882 (2018-02-15 17:38:10) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 85 key (470 EXTENT_DATA 0) itemoff 7124 itemsize 53 generation 186 type 1 (regular) extent data disk byte 232493420544 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 86 key (471 INODE_ITEM 0) itemoff 6964 itemsize 160 generation 190 transid 190 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745219.945264147 (2018-02-15 17:40:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 87 key (471 EXTENT_DATA 0) itemoff 6911 itemsize 53 generation 190 type 1 (regular) extent data disk byte 233566826496 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 88 key (472 INODE_ITEM 0) itemoff 6751 itemsize 160 generation 1448949 transid 1448949 size 262144 nbytes 11272192 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699168.280995440 (2020-07-13 20:59:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 89 key (472 EXTENT_DATA 0) itemoff 6698 itemsize 53 generation 1448949 type 1 (regular) extent data disk byte 385899433984 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 90 key (473 INODE_ITEM 0) itemoff 6538 itemsize 160 generation 189 transid 189 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745189.229705836 (2018-02-15 17:39:49) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 91 key (473 EXTENT_DATA 0) itemoff 6485 itemsize 53 generation 189 type 1 (regular) extent data disk byte 233567481856 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 92 key (474 INODE_ITEM 0) itemoff 6325 itemsize 160 generation 192 transid 192 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745287.536292726 (2018-02-15 17:41:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 93 key (474 EXTENT_DATA 0) itemoff 6272 itemsize 53 generation 192 type 1 (regular) extent data disk byte 235713855488 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 94 key (475 INODE_ITEM 0) itemoff 6112 itemsize 160 generation 1448952 transid 1448952 size 262144 nbytes 8650752 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 33 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699261.674203156 (2020-07-13 21:01:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 95 key (475 EXTENT_DATA 0) itemoff 6059 itemsize 53 generation 1448952 type 1 (regular) extent data disk byte 385905856512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 96 key (476 INODE_ITEM 0) itemoff 5899 itemsize 160 generation 202 transid 202 size 262144 nbytes 3407872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745621.355491718 (2018-02-15 17:47:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 97 key (476 EXTENT_DATA 0) itemoff 5846 itemsize 53 generation 202 type 1 (regular) extent data disk byte 247526436864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 98 key (477 INODE_ITEM 0) itemoff 5686 itemsize 160 generation 196 transid 196 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745416.550440527 (2018-02-15 17:43:36) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 99 key (477 EXTENT_DATA 0) itemoff 5633 itemsize 53 generation 196 type 1 (regular) extent data disk byte 241084006400 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 100 key (478 INODE_ITEM 0) itemoff 5473 itemsize 160 generation 197 transid 197 size 262144 nbytes 2621440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745451.389940795 (2018-02-15 17:44:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 101 key (478 EXTENT_DATA 0) itemoff 5420 itemsize 53 generation 197 type 1 (regular) extent data disk byte 240008871936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 102 key (479 INODE_ITEM 0) itemoff 5260 itemsize 160 generation 196 transid 196 size 262144 nbytes 1572864 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745416.550440527 (2018-02-15 17:43:36) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 103 key (479 EXTENT_DATA 0) itemoff 5207 itemsize 53 generation 196 type 1 (regular) extent data disk byte 242157457408 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 104 key (480 INODE_ITEM 0) itemoff 5047 itemsize 160 generation 2370 transid 2370 size 262144 nbytes 22544384 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 86 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518841689.246621552 (2018-02-16 20:28:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 105 key (480 EXTENT_DATA 0) itemoff 4994 itemsize 53 generation 2370 type 1 (regular) extent data disk byte 1247717027840 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 106 key (481 INODE_ITEM 0) itemoff 4834 itemsize 160 generation 1519 transid 1519 size 262144 nbytes 12845056 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 49 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518802433.118441774 (2018-02-16 09:33:53) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 107 key (481 EXTENT_DATA 0) itemoff 4781 itemsize 53 generation 1519 type 1 (regular) extent data disk byte 1145711169536 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 108 key (482 INODE_ITEM 0) itemoff 4621 itemsize 160 generation 199 transid 199 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745520.996942914 (2018-02-15 17:45:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 109 key (482 EXTENT_DATA 0) itemoff 4568 itemsize 53 generation 199 type 1 (regular) extent data disk byte 248600330240 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 110 key (483 INODE_ITEM 0) itemoff 4408 itemsize 160 generation 201 transid 201 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745588.671966038 (2018-02-15 17:46:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 111 key (483 EXTENT_DATA 0) itemoff 4355 itemsize 53 generation 201 type 1 (regular) extent data disk byte 256116416512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 112 key (484 INODE_ITEM 0) itemoff 4195 itemsize 160 generation 201 transid 201 size 262144 nbytes 2621440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745588.671966038 (2018-02-15 17:46:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 113 key (484 EXTENT_DATA 0) itemoff 4142 itemsize 53 generation 201 type 1 (regular) extent data disk byte 257190248448 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 114 key (485 INODE_ITEM 0) itemoff 3982 itemsize 160 generation 202 transid 202 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745621.355491718 (2018-02-15 17:47:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 115 key (485 EXTENT_DATA 0) itemoff 3929 itemsize 53 generation 202 type 1 (regular) extent data disk byte 251821535232 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 116 key (486 INODE_ITEM 0) itemoff 3769 itemsize 160 generation 202 transid 202 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745621.355491718 (2018-02-15 17:47:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 117 key (486 EXTENT_DATA 0) itemoff 3716 itemsize 53 generation 202 type 1 (regular) extent data disk byte 248599805952 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 118 key (487 INODE_ITEM 0) itemoff 3556 itemsize 160 generation 202 transid 202 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745621.355491718 (2018-02-15 17:47:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 119 key (487 EXTENT_DATA 0) itemoff 3503 itemsize 53 generation 202 type 1 (regular) extent data disk byte 252894867456 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 120 key (488 INODE_ITEM 0) itemoff 3343 itemsize 160 generation 204 transid 204 size 262144 nbytes 3407872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745688.938511563 (2018-02-15 17:48:08) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 121 key (488 EXTENT_DATA 0) itemoff 3290 itemsize 53 generation 204 type 1 (regular) extent data disk byte 269818867712 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 122 key (489 INODE_ITEM 0) itemoff 3130 itemsize 160 generation 204 transid 204 size 262144 nbytes 3407872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 13 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518745688.938511563 (2018-02-15 17:48:08) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) parent transid verify failed on 13576971354112 wanted 1601940 found 1602224 parent transid verify failed on 13576971354112 wanted 1601940 found 1602224 parent transid verify failed on 13576971354112 wanted 1601940 found 1602224 Ignoring transid failure leaf 13576971354112 items 204 free space 4388 generation 1602224 owner EXTENT_TREE leaf 13576971354112 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (13577814360064 METADATA_ITEM 0) itemoff 16250 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 1 key (13577814376448 METADATA_ITEM 0) itemoff 16217 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 2 key (13577814392832 METADATA_ITEM 0) itemoff 16184 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 3 key (13577814409216 METADATA_ITEM 0) itemoff 16151 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 4 key (13577814425600 METADATA_ITEM 0) itemoff 16118 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 5 key (13577814441984 METADATA_ITEM 0) itemoff 16085 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 6 key (13577814458368 METADATA_ITEM 0) itemoff 16052 itemsize 33 refs 1 gen 1365916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 7 key (13577814474752 METADATA_ITEM 0) itemoff 16019 itemsize 33 refs 1 gen 1397297 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 8 key (13577814491136 METADATA_ITEM 0) itemoff 15986 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 9 key (13577814507520 METADATA_ITEM 0) itemoff 15953 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 10 key (13577814523904 METADATA_ITEM 0) itemoff 15920 itemsize 33 refs 1 gen 1524027 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 11 key (13577814556672 METADATA_ITEM 0) itemoff 15887 itemsize 33 refs 1 gen 1510369 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 12 key (13577814589440 METADATA_ITEM 0) itemoff 15854 itemsize 33 refs 1 gen 1371035 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 13 key (13577814605824 METADATA_ITEM 0) itemoff 15821 itemsize 33 refs 1 gen 1508897 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 14 key (13577814638592 METADATA_ITEM 0) itemoff 15788 itemsize 33 refs 1 gen 1524031 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 15 key (13577814654976 METADATA_ITEM 0) itemoff 15755 itemsize 33 refs 1 gen 1397297 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 16 key (13577814671360 METADATA_ITEM 0) itemoff 15722 itemsize 33 refs 1 gen 1377633 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 17 key (13577814687744 METADATA_ITEM 0) itemoff 15689 itemsize 33 refs 1 gen 1397297 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 18 key (13577814704128 METADATA_ITEM 0) itemoff 15656 itemsize 33 refs 1 gen 1488520 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 19 key (13577814736896 METADATA_ITEM 0) itemoff 15623 itemsize 33 refs 1 gen 1397297 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 20 key (13577814753280 METADATA_ITEM 0) itemoff 15590 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 21 key (13577814769664 METADATA_ITEM 0) itemoff 15557 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 22 key (13577814786048 METADATA_ITEM 0) itemoff 15524 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 23 key (13577814802432 METADATA_ITEM 0) itemoff 15491 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 24 key (13577814818816 METADATA_ITEM 0) itemoff 15458 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 25 key (13577814835200 METADATA_ITEM 0) itemoff 15425 itemsize 33 refs 1 gen 1524026 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 26 key (13577814851584 METADATA_ITEM 0) itemoff 15392 itemsize 33 refs 1 gen 1174448 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 27 key (13577814867968 METADATA_ITEM 0) itemoff 15359 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 28 key (13577814884352 METADATA_ITEM 0) itemoff 15326 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 29 key (13577814900736 METADATA_ITEM 0) itemoff 15293 itemsize 33 refs 1 gen 1397297 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 30 key (13577814933504 METADATA_ITEM 0) itemoff 15260 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 31 key (13577814949888 METADATA_ITEM 0) itemoff 15227 itemsize 33 refs 1 gen 1509021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 32 key (13577814966272 METADATA_ITEM 0) itemoff 15194 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 33 key (13577814982656 METADATA_ITEM 0) itemoff 15161 itemsize 33 refs 1 gen 1538429 flags TREE_BLOCK tree block skinny level 0 shared block backref parent 15645094871040 item 34 key (13577814999040 METADATA_ITEM 0) itemoff 15128 itemsize 33 refs 1 gen 1524007 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 35 key (13577815015424 METADATA_ITEM 0) itemoff 15095 itemsize 33 refs 1 gen 1482916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 36 key (13577815031808 METADATA_ITEM 1) itemoff 15062 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 1 tree block backref root CSUM_TREE item 37 key (13577815048192 METADATA_ITEM 0) itemoff 15029 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 38 key (13577815064576 METADATA_ITEM 0) itemoff 14996 itemsize 33 refs 1 gen 1441746 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 39 key (13577815080960 METADATA_ITEM 0) itemoff 14963 itemsize 33 refs 1 gen 1353162 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 40 key (13577815097344 METADATA_ITEM 0) itemoff 14930 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 41 key (13577815113728 METADATA_ITEM 0) itemoff 14897 itemsize 33 refs 1 gen 1524007 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 42 key (13577815130112 METADATA_ITEM 0) itemoff 14864 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 43 key (13577815146496 METADATA_ITEM 0) itemoff 14831 itemsize 33 refs 1 gen 1170837 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 44 key (13577815162880 METADATA_ITEM 0) itemoff 14798 itemsize 33 refs 1 gen 1250069 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 45 key (13577815179264 METADATA_ITEM 0) itemoff 14765 itemsize 33 refs 1 gen 1482916 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 46 key (13577815195648 METADATA_ITEM 0) itemoff 14732 itemsize 33 refs 1 gen 1250069 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 47 key (13577815212032 METADATA_ITEM 0) itemoff 14699 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 48 key (13577815228416 METADATA_ITEM 0) itemoff 14666 itemsize 33 refs 1 gen 1509021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 49 key (13577815244800 METADATA_ITEM 0) itemoff 14633 itemsize 33 refs 1 gen 1173729 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 50 key (13577815261184 METADATA_ITEM 0) itemoff 14600 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 51 key (13577815277568 METADATA_ITEM 0) itemoff 14567 itemsize 33 refs 1 gen 1525997 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 52 key (13577815293952 METADATA_ITEM 0) itemoff 14534 itemsize 33 refs 1 gen 1441746 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 53 key (13577815310336 METADATA_ITEM 0) itemoff 14501 itemsize 33 refs 1 gen 1524033 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 54 key (13577815326720 METADATA_ITEM 0) itemoff 14468 itemsize 33 refs 1 gen 1399581 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 55 key (13577815343104 METADATA_ITEM 0) itemoff 14435 itemsize 33 refs 1 gen 1462080 flags TREE_BLOCK|FULL_BACKREF tree block skinny level 0 shared block backref parent 10679216750592 item 56 key (13577815359488 METADATA_ITEM 0) itemoff 14402 itemsize 33 refs 1 gen 1509021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 57 key (13577815375872 METADATA_ITEM 0) itemoff 14369 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 58 key (13577815392256 METADATA_ITEM 0) itemoff 14336 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 59 key (13577815408640 METADATA_ITEM 0) itemoff 14303 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 60 key (13577815425024 METADATA_ITEM 0) itemoff 14270 itemsize 33 refs 1 gen 1509130 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 61 key (13577815441408 METADATA_ITEM 0) itemoff 14237 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 62 key (13577815457792 METADATA_ITEM 0) itemoff 14204 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 63 key (13577815474176 METADATA_ITEM 0) itemoff 14171 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 64 key (13577815490560 METADATA_ITEM 0) itemoff 14138 itemsize 33 refs 1 gen 1371037 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 65 key (13577815506944 METADATA_ITEM 0) itemoff 14105 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 66 key (13577815523328 METADATA_ITEM 0) itemoff 14072 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 67 key (13577815539712 METADATA_ITEM 0) itemoff 14039 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 68 key (13577815556096 METADATA_ITEM 0) itemoff 14006 itemsize 33 refs 1 gen 1390120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 69 key (13577815572480 METADATA_ITEM 0) itemoff 13973 itemsize 33 refs 1 gen 1536322 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 70 key (13577815588864 METADATA_ITEM 0) itemoff 13940 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 71 key (13577815605248 METADATA_ITEM 0) itemoff 13907 itemsize 33 refs 1 gen 1510274 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 72 key (13577815621632 METADATA_ITEM 0) itemoff 13874 itemsize 33 refs 1 gen 1371036 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 73 key (13577815638016 METADATA_ITEM 0) itemoff 13841 itemsize 33 refs 1 gen 1536322 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 74 key (13577815654400 METADATA_ITEM 0) itemoff 13808 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 75 key (13577815687168 METADATA_ITEM 0) itemoff 13775 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 76 key (13577815719936 METADATA_ITEM 0) itemoff 13742 itemsize 33 refs 1 gen 1522187 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 77 key (13577815736320 METADATA_ITEM 0) itemoff 13709 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 78 key (13577815752704 METADATA_ITEM 0) itemoff 13676 itemsize 33 refs 1 gen 1442268 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11221 item 79 key (13577815769088 METADATA_ITEM 0) itemoff 13643 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 80 key (13577815785472 METADATA_ITEM 0) itemoff 13610 itemsize 33 refs 1 gen 1453429 flags TREE_BLOCK tree block skinny level 0 tree block backref root FS_TREE item 81 key (13577815801856 METADATA_ITEM 0) itemoff 13577 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 82 key (13577815818240 METADATA_ITEM 0) itemoff 13544 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 83 key (13577815834624 METADATA_ITEM 0) itemoff 13511 itemsize 33 refs 1 gen 1377634 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 84 key (13577815851008 METADATA_ITEM 0) itemoff 13478 itemsize 33 refs 1 gen 1524033 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 85 key (13577815867392 METADATA_ITEM 0) itemoff 13445 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 86 key (13577815883776 METADATA_ITEM 0) itemoff 13412 itemsize 33 refs 1 gen 1397299 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 87 key (13577815900160 METADATA_ITEM 0) itemoff 13379 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 88 key (13577815932928 METADATA_ITEM 0) itemoff 13346 itemsize 33 refs 1 gen 1509021 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11288 item 89 key (13577815949312 METADATA_ITEM 0) itemoff 13313 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 90 key (13577815965696 METADATA_ITEM 0) itemoff 13280 itemsize 33 refs 1 gen 1443708 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 91 key (13577815998464 METADATA_ITEM 0) itemoff 13247 itemsize 33 refs 1 gen 1534021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 92 key (13577816014848 METADATA_ITEM 0) itemoff 13214 itemsize 33 refs 1 gen 1534021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 93 key (13577816031232 METADATA_ITEM 0) itemoff 13118 itemsize 96 refs 8 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11288 shared block backref parent 15645306437632 shared block backref parent 11822225473536 shared block backref parent 4866918383616 shared block backref parent 4866918072320 shared block backref parent 4866917679104 shared block backref parent 4866917367808 shared block backref parent 4866908192768 item 94 key (13577816047616 METADATA_ITEM 0) itemoff 13085 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 95 key (13577816064000 METADATA_ITEM 0) itemoff 13052 itemsize 33 refs 1 gen 1440669 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 96 key (13577816080384 METADATA_ITEM 0) itemoff 13019 itemsize 33 refs 1 gen 1440669 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 97 key (13577816096768 METADATA_ITEM 0) itemoff 12986 itemsize 33 refs 1 gen 1440669 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 98 key (13577816113152 METADATA_ITEM 0) itemoff 12953 itemsize 33 refs 1 gen 1440669 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 99 key (13577816129536 METADATA_ITEM 0) itemoff 12920 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 100 key (13577816145920 METADATA_ITEM 0) itemoff 12887 itemsize 33 refs 1 gen 1399582 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 101 key (13577816162304 METADATA_ITEM 0) itemoff 12854 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 102 key (13577816178688 METADATA_ITEM 0) itemoff 12821 itemsize 33 refs 1 gen 1399582 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 103 key (13577816195072 METADATA_ITEM 0) itemoff 12788 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 104 key (13577816211456 METADATA_ITEM 0) itemoff 12755 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 105 key (13577816227840 METADATA_ITEM 0) itemoff 12722 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 106 key (13577816244224 METADATA_ITEM 0) itemoff 12689 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 107 key (13577816260608 METADATA_ITEM 0) itemoff 12656 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 108 key (13577816276992 METADATA_ITEM 0) itemoff 12623 itemsize 33 refs 1 gen 1482917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 109 key (13577816293376 METADATA_ITEM 0) itemoff 12590 itemsize 33 refs 1 gen 1522210 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 110 key (13577816309760 METADATA_ITEM 0) itemoff 12557 itemsize 33 refs 1 gen 1482917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 111 key (13577816326144 METADATA_ITEM 0) itemoff 12524 itemsize 33 refs 1 gen 1365917 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 112 key (13577816342528 METADATA_ITEM 0) itemoff 12491 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 113 key (13577816358912 METADATA_ITEM 0) itemoff 12458 itemsize 33 refs 1 gen 1509637 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 114 key (13577816375296 METADATA_ITEM 0) itemoff 12425 itemsize 33 refs 1 gen 1534021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 115 key (13577816391680 METADATA_ITEM 0) itemoff 12392 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 116 key (13577816408064 METADATA_ITEM 0) itemoff 12359 itemsize 33 refs 1 gen 1371038 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 117 key (13577816424448 METADATA_ITEM 0) itemoff 12326 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 118 key (13577816440832 METADATA_ITEM 0) itemoff 12293 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 119 key (13577816457216 METADATA_ITEM 0) itemoff 12260 itemsize 33 refs 1 gen 1371038 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 120 key (13577816473600 METADATA_ITEM 0) itemoff 12227 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 121 key (13577816489984 METADATA_ITEM 0) itemoff 12194 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 122 key (13577816506368 METADATA_ITEM 0) itemoff 12161 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 123 key (13577816522752 METADATA_ITEM 0) itemoff 12128 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 124 key (13577816539136 METADATA_ITEM 0) itemoff 12095 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 125 key (13577816555520 METADATA_ITEM 0) itemoff 12062 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 126 key (13577816571904 METADATA_ITEM 0) itemoff 12029 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 127 key (13577816588288 METADATA_ITEM 0) itemoff 11996 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 128 key (13577816604672 METADATA_ITEM 0) itemoff 11963 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 129 key (13577816621056 METADATA_ITEM 0) itemoff 11930 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 130 key (13577816637440 METADATA_ITEM 0) itemoff 11897 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 131 key (13577816653824 METADATA_ITEM 0) itemoff 11864 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 132 key (13577816670208 METADATA_ITEM 0) itemoff 11831 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 133 key (13577816686592 METADATA_ITEM 0) itemoff 11798 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 134 key (13577816702976 METADATA_ITEM 0) itemoff 11765 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 135 key (13577816719360 METADATA_ITEM 0) itemoff 11732 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 136 key (13577816735744 METADATA_ITEM 0) itemoff 11699 itemsize 33 refs 1 gen 1500489 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 137 key (13577816752128 METADATA_ITEM 0) itemoff 11666 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 138 key (13577816768512 METADATA_ITEM 0) itemoff 11633 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 139 key (13577816784896 METADATA_ITEM 0) itemoff 11600 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 140 key (13577816801280 METADATA_ITEM 0) itemoff 11567 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 141 key (13577816817664 METADATA_ITEM 0) itemoff 11534 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 142 key (13577816834048 METADATA_ITEM 0) itemoff 11501 itemsize 33 refs 1 gen 1369802 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 143 key (13577816850432 METADATA_ITEM 0) itemoff 11468 itemsize 33 refs 1 gen 1397293 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 144 key (13577816866816 METADATA_ITEM 0) itemoff 11435 itemsize 33 refs 1 gen 1369804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 145 key (13577816883200 METADATA_ITEM 0) itemoff 11402 itemsize 33 refs 1 gen 1369804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 146 key (13577816899584 METADATA_ITEM 0) itemoff 11369 itemsize 33 refs 1 gen 1369804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 147 key (13577816915968 METADATA_ITEM 0) itemoff 11336 itemsize 33 refs 1 gen 1397293 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 148 key (13577816932352 METADATA_ITEM 0) itemoff 11303 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 149 key (13577816948736 METADATA_ITEM 0) itemoff 11270 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 150 key (13577816965120 METADATA_ITEM 0) itemoff 11237 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 151 key (13577816981504 METADATA_ITEM 0) itemoff 11204 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 152 key (13577816997888 METADATA_ITEM 0) itemoff 11171 itemsize 33 refs 1 gen 1377635 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 153 key (13577817014272 METADATA_ITEM 0) itemoff 11138 itemsize 33 refs 1 gen 1369804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 154 key (13577817030656 METADATA_ITEM 0) itemoff 11105 itemsize 33 refs 1 gen 1369804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 155 key (13577817047040 METADATA_ITEM 0) itemoff 11072 itemsize 33 refs 1 gen 1397293 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 156 key (13577817063424 METADATA_ITEM 0) itemoff 11039 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 157 key (13577817079808 METADATA_ITEM 0) itemoff 11006 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 158 key (13577817096192 METADATA_ITEM 0) itemoff 10973 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 159 key (13577817112576 METADATA_ITEM 0) itemoff 10940 itemsize 33 refs 1 gen 1090952 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 160 key (13577817128960 METADATA_ITEM 0) itemoff 10907 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 161 key (13577817145344 METADATA_ITEM 0) itemoff 10874 itemsize 33 refs 1 gen 1396849 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 162 key (13577817178112 METADATA_ITEM 0) itemoff 10841 itemsize 33 refs 1 gen 1443709 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 163 key (13577817194496 METADATA_ITEM 0) itemoff 10808 itemsize 33 refs 1 gen 1397293 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 164 key (13577817227264 METADATA_ITEM 0) itemoff 10775 itemsize 33 refs 1 gen 1371039 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 165 key (13577817243648 METADATA_ITEM 0) itemoff 10742 itemsize 33 refs 1 gen 1397293 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 166 key (13577817260032 METADATA_ITEM 0) itemoff 10709 itemsize 33 refs 1 gen 1537120 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 167 key (13577817276416 METADATA_ITEM 0) itemoff 10676 itemsize 33 refs 1 gen 1443708 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 168 key (13577817292800 METADATA_ITEM 0) itemoff 10643 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 169 key (13577817309184 METADATA_ITEM 0) itemoff 10610 itemsize 33 refs 1 gen 1443709 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 170 key (13577817325568 METADATA_ITEM 0) itemoff 10577 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 171 key (13577817358336 METADATA_ITEM 0) itemoff 10544 itemsize 33 refs 1 gen 1500489 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 172 key (13577817374720 METADATA_ITEM 0) itemoff 10511 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 173 key (13577817391104 METADATA_ITEM 0) itemoff 10478 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 174 key (13577817407488 METADATA_ITEM 0) itemoff 10445 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 175 key (13577817423872 METADATA_ITEM 0) itemoff 10412 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 176 key (13577817440256 METADATA_ITEM 0) itemoff 10379 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 177 key (13577817456640 METADATA_ITEM 0) itemoff 10346 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 178 key (13577817473024 METADATA_ITEM 0) itemoff 10313 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 179 key (13577817489408 METADATA_ITEM 0) itemoff 10280 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 180 key (13577817505792 METADATA_ITEM 0) itemoff 10247 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 181 key (13577817522176 METADATA_ITEM 0) itemoff 10214 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 182 key (13577817538560 METADATA_ITEM 0) itemoff 10181 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 183 key (13577817554944 METADATA_ITEM 0) itemoff 10148 itemsize 33 refs 1 gen 1534021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 184 key (13577817571328 METADATA_ITEM 0) itemoff 10115 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 185 key (13577817587712 METADATA_ITEM 0) itemoff 10082 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 186 key (13577817604096 METADATA_ITEM 0) itemoff 10049 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 187 key (13577817620480 METADATA_ITEM 0) itemoff 10016 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 188 key (13577817636864 METADATA_ITEM 0) itemoff 9983 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 189 key (13577817653248 METADATA_ITEM 0) itemoff 9950 itemsize 33 refs 1 gen 1406255 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 190 key (13577817669632 METADATA_ITEM 0) itemoff 9917 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 191 key (13577817686016 METADATA_ITEM 0) itemoff 9884 itemsize 33 refs 1 gen 1509032 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 192 key (13577817702400 METADATA_ITEM 0) itemoff 9851 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 193 key (13577817718784 METADATA_ITEM 0) itemoff 9818 itemsize 33 refs 1 gen 1369803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 194 key (13577817735168 METADATA_ITEM 0) itemoff 9785 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 195 key (13577817751552 METADATA_ITEM 0) itemoff 9752 itemsize 33 refs 1 gen 1406255 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 196 key (13577817767936 METADATA_ITEM 0) itemoff 9719 itemsize 33 refs 1 gen 1534021 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 197 key (13577817784320 METADATA_ITEM 0) itemoff 9686 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 198 key (13577817800704 METADATA_ITEM 0) itemoff 9653 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 199 key (13577817817088 METADATA_ITEM 0) itemoff 9620 itemsize 33 refs 1 gen 1587580 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 200 key (13577817833472 METADATA_ITEM 0) itemoff 9587 itemsize 33 refs 1 gen 1390121 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 201 key (13577817849856 METADATA_ITEM 0) itemoff 9554 itemsize 33 refs 1 gen 1369805 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 202 key (13577817866240 METADATA_ITEM 0) itemoff 9521 itemsize 33 refs 1 gen 1369805 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 203 key (13577817882624 METADATA_ITEM 0) itemoff 9488 itemsize 33 refs 1 gen 1369805 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE leaf 364866551808 items 123 free space 55 generation 1453942 owner ROOT_TREE leaf 364866551808 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (541 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 224 transid 224 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746358.640866684 (2018-02-15 17:59:18) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 1 key (541 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 224 type 1 (regular) extent data disk byte 307043405824 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (542 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 224 transid 224 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746358.640866684 (2018-02-15 17:59:18) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 3 key (542 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 224 type 1 (regular) extent data disk byte 307043667968 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (543 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 224 transid 224 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746358.640866684 (2018-02-15 17:59:18) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 5 key (543 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 224 type 1 (regular) extent data disk byte 307043930112 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (544 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 1448952 transid 1448952 size 262144 nbytes 118226944 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 451 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699261.678203037 (2020-07-13 21:01:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 7 key (544 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 1448952 type 1 (regular) extent data disk byte 385910050816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (545 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 228 transid 228 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746485.615047014 (2018-02-15 18:01:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 9 key (545 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 228 type 1 (regular) extent data disk byte 311951056896 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (546 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 231 transid 231 size 262144 nbytes 1835008 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746581.861669265 (2018-02-15 18:03:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 11 key (546 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 231 type 1 (regular) extent data disk byte 315042164736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (547 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 228 transid 228 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746485.615047014 (2018-02-15 18:01:25) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 13 key (547 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 228 type 1 (regular) extent data disk byte 313239584768 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (548 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 230 transid 230 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746551.226107662 (2018-02-15 18:02:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 15 key (548 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 230 type 1 (regular) extent data disk byte 315171475456 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (549 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 229 transid 229 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746520.422548598 (2018-02-15 18:02:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 17 key (549 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 229 type 1 (regular) extent data disk byte 315041640448 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (550 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 230 transid 230 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746551.226107662 (2018-02-15 18:02:31) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 19 key (550 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 230 type 1 (regular) extent data disk byte 315171737600 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (551 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 231 transid 231 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746581.861669265 (2018-02-15 18:03:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 21 key (551 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 231 type 1 (regular) extent data disk byte 315042426880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (552 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 1448972 transid 1448972 size 262144 nbytes 11272192 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594699941.400568279 (2020-07-13 21:12:21) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 23 key (552 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 1448972 type 1 (regular) extent data disk byte 385932996608 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (553 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 232 transid 232 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746614.633200448 (2018-02-15 18:03:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 25 key (553 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 232 type 1 (regular) extent data disk byte 320134049792 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (554 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 233 transid 233 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746649.448702556 (2018-02-15 18:04:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 27 key (554 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 233 type 1 (regular) extent data disk byte 318224015360 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (555 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 255 transid 255 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747394.918080025 (2018-02-15 18:16:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 29 key (555 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 255 type 1 (regular) extent data disk byte 349531496448 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (556 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 234 transid 234 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746684.268204777 (2018-02-15 18:04:44) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 31 key (556 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 234 type 1 (regular) extent data disk byte 320134991872 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (557 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 242 transid 242 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746956.648316509 (2018-02-15 18:09:16) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 33 key (557 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 242 type 1 (regular) extent data disk byte 339572670464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (558 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 235 transid 235 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746719.87707165 (2018-02-15 18:05:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 35 key (558 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 235 type 1 (regular) extent data disk byte 325533712384 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (559 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 243 transid 243 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.467820159 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 37 key (559 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 243 type 1 (regular) extent data disk byte 340941766656 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (560 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 235 transid 235 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746719.87707165 (2018-02-15 18:05:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 39 key (560 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 235 type 1 (regular) extent data disk byte 325534236672 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (561 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 235 transid 235 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746719.91707108 (2018-02-15 18:05:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 41 key (561 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 235 type 1 (regular) extent data disk byte 325534498816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (562 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 256 transid 256 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747429.853583893 (2018-02-15 18:17:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 43 key (562 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 256 type 1 (regular) extent data disk byte 350605619200 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (563 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 243 transid 243 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.467820159 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 45 key (563 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 243 type 1 (regular) extent data disk byte 340942028800 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (564 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 243 transid 243 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.467820159 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 47 key (564 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341099249664 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (565 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 245 transid 245 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747054.954915557 (2018-02-15 18:10:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 49 key (565 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 245 type 1 (regular) extent data disk byte 344332042240 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (566 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 243 transid 243 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.467820159 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 51 key (566 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341099773952 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (567 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 243 transid 243 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 53 key (567 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341100036096 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (568 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 243 transid 243 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 55 key (568 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341100298240 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (569 INODE_ITEM 0) itemoff 10159 itemsize 160 generation 243 transid 243 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 57 key (569 EXTENT_DATA 0) itemoff 10106 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341100560384 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (570 INODE_ITEM 0) itemoff 9946 itemsize 160 generation 246 transid 246 size 262144 nbytes 2097152 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 8 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747087.718448921 (2018-02-15 18:11:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 59 key (570 EXTENT_DATA 0) itemoff 9893 itemsize 53 generation 246 type 1 (regular) extent data disk byte 346001625088 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 60 key (571 INODE_ITEM 0) itemoff 9733 itemsize 160 generation 243 transid 243 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 61 key (571 EXTENT_DATA 0) itemoff 9680 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341101084672 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 62 key (572 INODE_ITEM 0) itemoff 9520 itemsize 160 generation 243 transid 243 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 63 key (572 EXTENT_DATA 0) itemoff 9467 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341101346816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 64 key (573 INODE_ITEM 0) itemoff 9307 itemsize 160 generation 243 transid 243 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 65 key (573 EXTENT_DATA 0) itemoff 9254 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341101608960 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 66 key (574 INODE_ITEM 0) itemoff 9094 itemsize 160 generation 243 transid 243 size 262144 nbytes 524288 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518746991.471820102 (2018-02-15 18:09:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 67 key (574 EXTENT_DATA 0) itemoff 9041 itemsize 53 generation 243 type 1 (regular) extent data disk byte 341101871104 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 68 key (575 INODE_ITEM 0) itemoff 8881 itemsize 160 generation 257 transid 257 size 262144 nbytes 3670016 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747464.617090343 (2018-02-15 18:17:44) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 69 key (575 EXTENT_DATA 0) itemoff 8828 itemsize 53 generation 257 type 1 (regular) extent data disk byte 350604308480 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 70 key (576 INODE_ITEM 0) itemoff 8668 itemsize 160 generation 245 transid 245 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747054.954915557 (2018-02-15 18:10:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 71 key (576 EXTENT_DATA 0) itemoff 8615 itemsize 53 generation 245 type 1 (regular) extent data disk byte 344332566528 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 72 key (577 INODE_ITEM 0) itemoff 8455 itemsize 160 generation 246 transid 246 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747087.718448921 (2018-02-15 18:11:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 73 key (577 EXTENT_DATA 0) itemoff 8402 itemsize 53 generation 246 type 1 (regular) extent data disk byte 346002149376 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 74 key (578 INODE_ITEM 0) itemoff 8242 itemsize 160 generation 246 transid 246 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747087.718448921 (2018-02-15 18:11:27) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 75 key (578 EXTENT_DATA 0) itemoff 8189 itemsize 53 generation 246 type 1 (regular) extent data disk byte 346002411520 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 76 key (579 INODE_ITEM 0) itemoff 8029 itemsize 160 generation 247 transid 247 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747120.485982364 (2018-02-15 18:12:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 77 key (579 EXTENT_DATA 0) itemoff 7976 itemsize 53 generation 247 type 1 (regular) extent data disk byte 347432411136 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 78 key (580 INODE_ITEM 0) itemoff 7816 itemsize 160 generation 247 transid 247 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747120.485982364 (2018-02-15 18:12:00) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 79 key (580 EXTENT_DATA 0) itemoff 7763 itemsize 53 generation 247 type 1 (regular) extent data disk byte 347432673280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 80 key (581 INODE_ITEM 0) itemoff 7603 itemsize 160 generation 1448974 transid 1448974 size 262144 nbytes 11272192 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 43 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594700018.221151781 (2020-07-13 21:13:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 81 key (581 EXTENT_DATA 0) itemoff 7550 itemsize 53 generation 1448974 type 1 (regular) extent data disk byte 385934254080 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 82 key (582 INODE_ITEM 0) itemoff 7390 itemsize 160 generation 256 transid 256 size 262144 nbytes 1048576 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747429.853583893 (2018-02-15 18:17:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 83 key (582 EXTENT_DATA 0) itemoff 7337 itemsize 53 generation 256 type 1 (regular) extent data disk byte 352508796928 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 84 key (583 INODE_ITEM 0) itemoff 7177 itemsize 160 generation 384 transid 384 size 262144 nbytes 3670016 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 14 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518751904.682990094 (2018-02-15 19:31:44) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 85 key (583 EXTENT_DATA 0) itemoff 7124 itemsize 53 generation 384 type 1 (regular) extent data disk byte 430598922240 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 86 key (584 INODE_ITEM 0) itemoff 6964 itemsize 160 generation 263 transid 263 size 262144 nbytes 4194304 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518747669.350186333 (2018-02-15 18:21:09) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 87 key (584 EXTENT_DATA 0) itemoff 6911 itemsize 53 generation 263 type 1 (regular) extent data disk byte 353824210944 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 88 key (585 INODE_ITEM 0) itemoff 6751 itemsize 160 generation 1448979 transid 1448979 size 262144 nbytes 13369344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594700214.808521242 (2020-07-13 21:16:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 89 key (585 EXTENT_DATA 0) itemoff 6698 itemsize 53 generation 1448979 type 1 (regular) extent data disk byte 385938366464 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 90 key (586 INODE_ITEM 0) itemoff 6538 itemsize 160 generation 385 transid 385 size 262144 nbytes 5505024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 21 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518751939.550509777 (2018-02-15 19:32:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 91 key (586 EXTENT_DATA 0) itemoff 6485 itemsize 53 generation 385 type 1 (regular) extent data disk byte 612341153792 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 92 key (587 INODE_ITEM 0) itemoff 6325 itemsize 160 generation 1448979 transid 1448979 size 262144 nbytes 44302336 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 169 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594700214.808521242 (2020-07-13 21:16:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 93 key (587 EXTENT_DATA 0) itemoff 6272 itemsize 53 generation 1448979 type 1 (regular) extent data disk byte 385938628608 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 94 key (588 INODE_ITEM 0) itemoff 6112 itemsize 160 generation 355 transid 355 size 262144 nbytes 15728640 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 60 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750929.764423640 (2018-02-15 19:15:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 95 key (588 EXTENT_DATA 0) itemoff 6059 itemsize 53 generation 355 type 1 (regular) extent data disk byte 371543683072 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 96 key (589 INODE_ITEM 0) itemoff 5899 itemsize 160 generation 285 transid 285 size 262144 nbytes 6029312 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 23 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518748435.303357430 (2018-02-15 18:33:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 97 key (589 EXTENT_DATA 0) itemoff 5846 itemsize 53 generation 285 type 1 (regular) extent data disk byte 386576044032 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 98 key (590 INODE_ITEM 0) itemoff 5686 itemsize 160 generation 385 transid 385 size 262144 nbytes 10747904 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 41 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518751939.554509722 (2018-02-15 19:32:19) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 99 key (590 EXTENT_DATA 0) itemoff 5633 itemsize 53 generation 385 type 1 (regular) extent data disk byte 613844824064 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 100 key (591 INODE_ITEM 0) itemoff 5473 itemsize 160 generation 353 transid 353 size 262144 nbytes 8388608 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 32 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750860.313380907 (2018-02-15 19:14:20) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 101 key (591 EXTENT_DATA 0) itemoff 5420 itemsize 53 generation 353 type 1 (regular) extent data disk byte 369396273152 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 102 key (592 INODE_ITEM 0) itemoff 5260 itemsize 160 generation 285 transid 285 size 262144 nbytes 2359296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518748435.307357375 (2018-02-15 18:33:55) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 103 key (592 EXTENT_DATA 0) itemoff 5207 itemsize 53 generation 285 type 1 (regular) extent data disk byte 390869721088 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 104 key (593 INODE_ITEM 0) itemoff 5047 itemsize 160 generation 293 transid 293 size 262144 nbytes 4456448 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 17 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518748722.27313744 (2018-02-15 18:38:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 105 key (593 EXTENT_DATA 0) itemoff 4994 itemsize 53 generation 293 type 1 (regular) extent data disk byte 390861172736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 106 key (594 INODE_ITEM 0) itemoff 4834 itemsize 160 generation 355 transid 355 size 262144 nbytes 14942208 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 57 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750929.768423587 (2018-02-15 19:15:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 107 key (594 EXTENT_DATA 0) itemoff 4781 itemsize 53 generation 355 type 1 (regular) extent data disk byte 374764580864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 108 key (595 INODE_ITEM 0) itemoff 4621 itemsize 160 generation 311 transid 311 size 262144 nbytes 1310720 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518749369.346217917 (2018-02-15 18:49:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 109 key (595 EXTENT_DATA 0) itemoff 4568 itemsize 53 generation 311 type 1 (regular) extent data disk byte 424157044736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 110 key (596 INODE_ITEM 0) itemoff 4408 itemsize 160 generation 1453942 transid 1453942 size 262144 nbytes 13369344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 51 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594936992.288440916 (2020-07-16 15:03:12) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 111 key (596 EXTENT_DATA 0) itemoff 4355 itemsize 53 generation 1453942 type 1 (regular) extent data disk byte 749577441280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 112 key (597 INODE_ITEM 0) itemoff 4195 itemsize 160 generation 354 transid 354 size 262144 nbytes 15204352 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 58 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750894.952903454 (2018-02-15 19:14:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 113 key (597 EXTENT_DATA 0) itemoff 4142 itemsize 53 generation 354 type 1 (regular) extent data disk byte 390868410368 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 114 key (598 INODE_ITEM 0) itemoff 3982 itemsize 160 generation 357 transid 357 size 262144 nbytes 5242880 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 20 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518751001.451435601 (2018-02-15 19:16:41) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 115 key (598 EXTENT_DATA 0) itemoff 3929 itemsize 53 generation 357 type 1 (regular) extent data disk byte 390863429632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 116 key (599 INODE_ITEM 0) itemoff 3769 itemsize 160 generation 354 transid 354 size 262144 nbytes 4194304 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 16 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750894.952903454 (2018-02-15 19:14:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 117 key (599 EXTENT_DATA 0) itemoff 3716 itemsize 53 generation 354 type 1 (regular) extent data disk byte 400534085632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 118 key (600 INODE_ITEM 0) itemoff 3556 itemsize 160 generation 311 transid 311 size 262144 nbytes 786432 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518749369.346217917 (2018-02-15 18:49:29) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 119 key (600 EXTENT_DATA 0) itemoff 3503 itemsize 53 generation 311 type 1 (regular) extent data disk byte 426303086592 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 120 key (601 INODE_ITEM 0) itemoff 3343 itemsize 160 generation 1448976 transid 1448976 size 262144 nbytes 15466496 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 59 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1594700101.145491792 (2020-07-13 21:15:01) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 121 key (601 EXTENT_DATA 0) itemoff 3290 itemsize 53 generation 1448976 type 1 (regular) extent data disk byte 385936875520 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 122 key (602 INODE_ITEM 0) itemoff 3130 itemsize 160 generation 354 transid 354 size 262144 nbytes 13631488 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 52 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1518750894.956903399 (2018-02-15 19:14:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) parent transid verify failed on 11970666151936 wanted 1601669 found 1602194 checksum verify failed on 11970666151936 wanted 0x2bb9614c found 0xea514a34 checksum verify failed on 11970666151936 wanted 0x2bb9614c found 0xea514a34 bad tree block 11970666151936, bad level, 234 > 8 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 0:58 ` Marc MERLIN @ 2022-04-05 1:06 ` Josef Bacik 2022-04-05 1:16 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 1:06 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > > > > Binary identical after rebuild. > > > > > > > > Sigh time for printf sanity checks, thanks, > > > > > > > I'm dumb, try again please, thanks, > > progress :) Ok, lets try btrfs inspect-internal dump-tree -b 13577779511296 and see what that gives us, the root we think is ok is missing the part with the actual root items. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 1:06 ` Josef Bacik @ 2022-04-05 1:16 ` Marc MERLIN 2022-04-05 1:22 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 1:16 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 09:06:01PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote: > > > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > > > > > Binary identical after rebuild. > > > > > > > > > > Sigh time for printf sanity checks, thanks, > > > > > > > > > > I'm dumb, try again please, thanks, > > > > progress :) > > Ok, lets try > > btrfs inspect-internal dump-tree -b 13577779511296 > > and see what that gives us, the root we think is ok is missing the > part with the actual root items. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577779511296 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55de98e14470 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55de98e14470 btrfs-progs v5.16.2 leaf 13577779511296 items 201 free space 1149 generation 1602242 owner EXTENT_TREE leaf 13577779511296 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (747913936896 EXTENT_ITEM 16384) itemoff 16230 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953489408 count 1 item 1 key (747913953280 EXTENT_ITEM 4096) itemoff 16177 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953620480 count 1 item 2 key (747913957376 EXTENT_ITEM 16384) itemoff 16124 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953645056 count 1 item 3 key (747913973760 EXTENT_ITEM 20480) itemoff 16071 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953776128 count 1 item 4 key (747913994240 EXTENT_ITEM 12288) itemoff 16018 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953907200 count 1 item 5 key (747914006528 EXTENT_ITEM 16384) itemoff 15965 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12953985024 count 1 item 6 key (747914022912 EXTENT_ITEM 20480) itemoff 15912 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954083328 count 1 item 7 key (747914043392 EXTENT_ITEM 20480) itemoff 15859 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954214400 count 1 item 8 key (747914063872 EXTENT_ITEM 4096) itemoff 15806 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954345472 count 1 item 9 key (747914067968 EXTENT_ITEM 9043968) itemoff 15753 itemsize 53 refs 1 gen 1467378 flags DATA extent data backref root 11288 objectid 38180 offset 413532160 count 1 item 10 key (747923111936 EXTENT_ITEM 36864) itemoff 15716 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 11 key (747923148800 EXTENT_ITEM 36864) itemoff 15679 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 12 key (747923210240 EXTENT_ITEM 28672) itemoff 15626 itemsize 53 refs 1 gen 1539316 flags DATA extent data backref root 11288 objectid 72889 offset 169021440 count 1 item 13 key (747923238912 EXTENT_ITEM 131072) itemoff 15573 itemsize 53 refs 1 gen 1486211 flags DATA extent data backref root 11288 objectid 39190 offset 0 count 1 item 14 key (747923369984 EXTENT_ITEM 4096) itemoff 15520 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733089280 count 1 item 15 key (747923374080 EXTENT_ITEM 262144) itemoff 15467 itemsize 53 refs 1 gen 1472819 flags DATA extent data backref root ROOT_TREE objectid 1377 offset 0 count 1 item 16 key (747923636224 EXTENT_ITEM 53248) itemoff 15414 itemsize 53 refs 1 gen 1509130 flags DATA extent data backref root 11288 objectid 49146 offset 0 count 1 item 17 key (747923689472 EXTENT_ITEM 57344) itemoff 15361 itemsize 53 refs 1 gen 1509130 flags DATA extent data backref root 11288 objectid 49147 offset 0 count 1 item 18 key (747923746816 EXTENT_ITEM 8192) itemoff 15308 itemsize 53 refs 1 gen 1509156 flags DATA extent data backref root 11288 objectid 50588 offset 0 count 1 item 19 key (747923755008 EXTENT_ITEM 8192) itemoff 15255 itemsize 53 refs 1 gen 1509156 flags DATA extent data backref root 11288 objectid 50613 offset 0 count 1 item 20 key (747923763200 EXTENT_ITEM 4096) itemoff 15202 itemsize 53 refs 1 gen 1509865 flags DATA extent data backref root 11288 objectid 70849 offset 0 count 1 item 21 key (747923767296 EXTENT_ITEM 77824) itemoff 15149 itemsize 53 refs 1 gen 1509129 flags DATA extent data backref root 11288 objectid 49044 offset 0 count 1 item 22 key (747923845120 EXTENT_ITEM 53248) itemoff 15096 itemsize 53 refs 1 gen 1509130 flags DATA extent data backref root 11288 objectid 49148 offset 0 count 1 item 23 key (747923898368 EXTENT_ITEM 114688) itemoff 15043 itemsize 53 refs 1 gen 1484194 flags DATA extent data backref root 11288 objectid 39120 offset 2883584 count 1 item 24 key (747924013056 EXTENT_ITEM 36864) itemoff 15006 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 25 key (747924049920 EXTENT_ITEM 36864) itemoff 14969 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 26 key (747924086784 EXTENT_ITEM 36864) itemoff 14932 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 27 key (747924144128 EXTENT_ITEM 8192) itemoff 14879 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1823997 offset 0 count 1 item 28 key (747924152320 EXTENT_ITEM 8192) itemoff 14826 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1823998 offset 0 count 1 item 29 key (747924160512 EXTENT_ITEM 86016) itemoff 14773 itemsize 53 refs 1 gen 1508916 flags DATA extent data backref root 11288 objectid 40988 offset 393216 count 1 item 30 key (747924246528 EXTENT_ITEM 114688) itemoff 14720 itemsize 53 refs 1 gen 1508916 flags DATA extent data backref root 11288 objectid 40988 offset 524288 count 1 item 31 key (747924361216 EXTENT_ITEM 32768) itemoff 14667 itemsize 53 refs 1 gen 1508973 flags DATA extent data backref root 11288 objectid 41258 offset 0 count 1 item 32 key (747924393984 EXTENT_ITEM 28672) itemoff 14614 itemsize 53 refs 1 gen 1508973 flags DATA extent data backref root 11288 objectid 41261 offset 0 count 1 item 33 key (747924422656 EXTENT_ITEM 16384) itemoff 14561 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12942655488 count 1 item 34 key (747924439040 EXTENT_ITEM 16384) itemoff 14508 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12942786560 count 1 item 35 key (747924455424 EXTENT_ITEM 196608) itemoff 14455 itemsize 53 refs 1 gen 1482902 flags DATA extent data backref root 11288 objectid 39004 offset 2231795712 count 1 item 36 key (747924652032 EXTENT_ITEM 53248) itemoff 14402 itemsize 53 refs 1 gen 1482903 flags DATA extent data backref root 11288 objectid 39005 offset 131072 count 1 item 37 key (747924705280 EXTENT_ITEM 36864) itemoff 14365 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 38 key (747924742144 EXTENT_ITEM 36864) itemoff 14328 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 39 key (747924779008 EXTENT_ITEM 36864) itemoff 14291 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 40 key (747924815872 EXTENT_ITEM 36864) itemoff 14254 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 41 key (747924852736 EXTENT_ITEM 36864) itemoff 14217 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 42 key (747924889600 EXTENT_ITEM 36864) itemoff 14180 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 43 key (747924934656 EXTENT_ITEM 28672) itemoff 14127 itemsize 53 refs 1 gen 1565039 flags DATA extent data backref root 11288 objectid 75169 offset 6160384 count 1 item 44 key (747924963328 EXTENT_ITEM 4096) itemoff 14074 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733220352 count 1 item 45 key (747924967424 EXTENT_ITEM 4096) itemoff 14021 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733351424 count 1 item 46 key (747924971520 EXTENT_ITEM 4096) itemoff 13968 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733482496 count 1 item 47 key (747924975616 EXTENT_ITEM 4096) itemoff 13915 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733613568 count 1 item 48 key (747924979712 EXTENT_ITEM 36864) itemoff 13862 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 0 count 1 item 49 key (747925016576 EXTENT_ITEM 36864) itemoff 13809 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 131072 count 1 item 50 key (747925053440 EXTENT_ITEM 36864) itemoff 13756 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 262144 count 1 item 51 key (747925090304 EXTENT_ITEM 36864) itemoff 13703 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 393216 count 1 item 52 key (747925127168 EXTENT_ITEM 36864) itemoff 13650 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 524288 count 1 item 53 key (747925164032 EXTENT_ITEM 36864) itemoff 13597 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 655360 count 1 item 54 key (747925200896 EXTENT_ITEM 36864) itemoff 13544 itemsize 53 refs 1 gen 1597228 flags DATA extent data backref root 11221 objectid 414410 offset 786432 count 1 item 55 key (747925241856 EXTENT_ITEM 155648) itemoff 13491 itemsize 53 refs 1 gen 1509071 flags DATA extent data backref root 11288 objectid 46649 offset 0 count 1 item 56 key (747925397504 EXTENT_ITEM 86016) itemoff 13454 itemsize 37 refs 1 gen 1509093 flags DATA shared data backref parent 12512079577088 count 1 item 57 key (747925483520 EXTENT_ITEM 20480) itemoff 13401 itemsize 53 refs 1 gen 1509156 flags DATA extent data backref root 11288 objectid 50590 offset 0 count 1 item 58 key (747925504000 EXTENT_ITEM 36864) itemoff 13364 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 59 key (747925540864 EXTENT_ITEM 36864) itemoff 13327 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 60 key (747925577728 EXTENT_ITEM 36864) itemoff 13290 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 61 key (747925614592 EXTENT_ITEM 36864) itemoff 13253 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 62 key (747925651456 EXTENT_ITEM 36864) itemoff 13216 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 63 key (747925700608 EXTENT_ITEM 4096) itemoff 13163 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824000 offset 0 count 1 item 64 key (747925704704 EXTENT_ITEM 4096) itemoff 13110 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824005 offset 0 count 1 item 65 key (747925708800 EXTENT_ITEM 4096) itemoff 13057 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824007 offset 0 count 1 item 66 key (747925712896 EXTENT_ITEM 12288) itemoff 13004 itemsize 53 refs 1 gen 1553656 flags DATA extent data backref root 11288 objectid 74580 offset 0 count 1 item 67 key (747925725184 EXTENT_ITEM 4096) itemoff 12951 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733744640 count 1 item 68 key (747925729280 EXTENT_ITEM 49152) itemoff 12898 itemsize 53 refs 1 gen 1484713 flags DATA extent data backref root 11288 objectid 39130 offset 0 count 1 item 69 key (747925778432 EXTENT_ITEM 57344) itemoff 12845 itemsize 53 refs 1 gen 1484739 flags DATA extent data backref root 11288 objectid 39131 offset 727973888 count 1 item 70 key (747925835776 EXTENT_ITEM 32768) itemoff 12808 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510516736 count 1 item 71 key (747925868544 EXTENT_ITEM 4096) itemoff 12755 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824014 offset 0 count 1 item 72 key (747925872640 EXTENT_ITEM 4096) itemoff 12702 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5733875712 count 1 item 73 key (747925876736 EXTENT_ITEM 4096) itemoff 12649 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734006784 count 1 item 74 key (747925880832 EXTENT_ITEM 4096) itemoff 12596 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734137856 count 1 item 75 key (747925884928 EXTENT_ITEM 4096) itemoff 12543 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734268928 count 1 item 76 key (747925889024 EXTENT_ITEM 4096) itemoff 12490 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734400000 count 1 item 77 key (747925893120 EXTENT_ITEM 4096) itemoff 12437 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734531072 count 1 item 78 key (747925897216 EXTENT_ITEM 118784) itemoff 12384 itemsize 53 refs 1 gen 1484194 flags DATA extent data backref root 11288 objectid 39120 offset 3014656 count 1 item 79 key (747926016000 EXTENT_ITEM 114688) itemoff 12331 itemsize 53 refs 1 gen 1484194 flags DATA extent data backref root 11288 objectid 39120 offset 3145728 count 1 item 80 key (747926130688 EXTENT_ITEM 36864) itemoff 12294 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 81 key (747926167552 EXTENT_ITEM 36864) itemoff 12257 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 82 key (747926204416 EXTENT_ITEM 36864) itemoff 12220 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 83 key (747926241280 EXTENT_ITEM 36864) itemoff 12183 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 84 key (747926278144 EXTENT_ITEM 36864) itemoff 12146 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 85 key (747926315008 EXTENT_ITEM 36864) itemoff 12109 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 86 key (747926360064 EXTENT_ITEM 20480) itemoff 12056 itemsize 53 refs 1 gen 1590234 flags DATA extent data backref root 11223 objectid 1823985 offset 0 count 1 item 87 key (747926380544 EXTENT_ITEM 4096) itemoff 12003 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824015 offset 0 count 1 item 88 key (747926384640 EXTENT_ITEM 4096) itemoff 11950 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824020 offset 0 count 1 item 89 key (747926388736 EXTENT_ITEM 4096) itemoff 11897 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734662144 count 1 item 90 key (747926392832 EXTENT_ITEM 36864) itemoff 11860 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 91 key (747926458368 EXTENT_ITEM 4096) itemoff 11807 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824030 offset 0 count 1 item 92 key (747926462464 EXTENT_ITEM 4096) itemoff 11754 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824032 offset 0 count 1 item 93 key (747926466560 EXTENT_ITEM 4096) itemoff 11701 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734793216 count 1 item 94 key (747926470656 EXTENT_ITEM 4096) itemoff 11648 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5734924288 count 1 item 95 key (747926474752 EXTENT_ITEM 53248) itemoff 11595 itemsize 53 refs 1 gen 1483072 flags DATA extent data backref root 11288 objectid 39029 offset 0 count 1 item 96 key (747926528000 EXTENT_ITEM 4096) itemoff 11542 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735055360 count 1 item 97 key (747926532096 EXTENT_ITEM 4096) itemoff 11489 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735186432 count 1 item 98 key (747926536192 EXTENT_ITEM 4096) itemoff 11436 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735317504 count 1 item 99 key (747926540288 EXTENT_ITEM 4096) itemoff 11383 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735448576 count 1 item 100 key (747926544384 EXTENT_ITEM 4096) itemoff 11330 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735579648 count 1 item 101 key (747926548480 EXTENT_ITEM 4096) itemoff 11277 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735710720 count 1 item 102 key (747926552576 EXTENT_ITEM 53248) itemoff 11224 itemsize 53 refs 1 gen 1482894 flags DATA extent data backref root 11288 objectid 39000 offset 1179648 count 1 item 103 key (747926605824 EXTENT_ITEM 57344) itemoff 11171 itemsize 53 refs 1 gen 1482894 flags DATA extent data backref root 11288 objectid 39000 offset 1310720 count 1 item 104 key (747926663168 EXTENT_ITEM 53248) itemoff 11118 itemsize 53 refs 1 gen 1482894 flags DATA extent data backref root 11288 objectid 39000 offset 1441792 count 1 item 105 key (747926716416 EXTENT_ITEM 53248) itemoff 11065 itemsize 53 refs 1 gen 1482894 flags DATA extent data backref root 11288 objectid 39000 offset 1572864 count 1 item 106 key (747926769664 EXTENT_ITEM 32768) itemoff 11012 itemsize 53 refs 1 gen 1482894 flags DATA extent data backref root 11288 objectid 39001 offset 524288 count 1 item 107 key (747926802432 EXTENT_ITEM 4096) itemoff 10959 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735841792 count 1 item 108 key (747926806528 EXTENT_ITEM 4096) itemoff 10906 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5735972864 count 1 item 109 key (747926810624 EXTENT_ITEM 4096) itemoff 10853 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5736103936 count 1 item 110 key (747926814720 EXTENT_ITEM 262144) itemoff 10800 itemsize 53 refs 1 gen 1474024 flags DATA extent data backref root ROOT_TREE objectid 2566 offset 0 count 1 item 111 key (747927076864 EXTENT_ITEM 16384) itemoff 10747 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943011840 count 1 item 112 key (747927093248 EXTENT_ITEM 16384) itemoff 10694 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943142912 count 1 item 113 key (747927109632 EXTENT_ITEM 8192) itemoff 10641 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943273984 count 1 item 114 key (747927117824 EXTENT_ITEM 32768) itemoff 10588 itemsize 53 refs 1 gen 1602242 flags DATA extent data backref root 11221 objectid 414456 offset 0 count 1 item 115 key (747927150592 EXTENT_ITEM 36864) itemoff 10535 itemsize 53 refs 1 gen 1602242 flags DATA extent data backref root 11221 objectid 414456 offset 131072 count 1 item 116 key (747927187456 EXTENT_ITEM 32768) itemoff 10482 itemsize 53 refs 1 gen 1602242 flags DATA extent data backref root 11221 objectid 414456 offset 262144 count 1 item 117 key (747927379968 EXTENT_ITEM 229376) itemoff 10429 itemsize 53 refs 1 gen 1482585 flags DATA extent data backref root 11288 objectid 38979 offset 5445386240 count 1 item 118 key (747927609344 EXTENT_ITEM 32768) itemoff 10376 itemsize 53 refs 1 gen 1482590 flags DATA extent data backref root 11288 objectid 38979 offset 9466732544 count 1 item 119 key (747927642112 EXTENT_ITEM 229376) itemoff 10323 itemsize 53 refs 1 gen 1482593 flags DATA extent data backref root 11288 objectid 38979 offset 11495415808 count 1 item 120 key (747927871488 EXTENT_ITEM 32768) itemoff 10286 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510516736 count 1 item 121 key (747927904256 EXTENT_ITEM 524288) itemoff 10233 itemsize 53 refs 1 gen 1483575 flags DATA extent data backref root 11288 objectid 39051 offset 524288 count 1 item 122 key (747928428544 EXTENT_ITEM 36864) itemoff 10196 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 123 key (747928494080 EXTENT_ITEM 4096) itemoff 10143 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824033 offset 0 count 1 item 124 key (747928498176 EXTENT_ITEM 4096) itemoff 10090 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824036 offset 0 count 1 item 125 key (747928502272 EXTENT_ITEM 4096) itemoff 10037 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5736235008 count 1 item 126 key (747928506368 EXTENT_ITEM 4096) itemoff 9984 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5736366080 count 1 item 127 key (747928510464 EXTENT_ITEM 8192) itemoff 9931 itemsize 53 refs 1 gen 1553656 flags DATA extent data backref root 11288 objectid 74580 offset 393216 count 1 item 128 key (747928518656 EXTENT_ITEM 8192) itemoff 9878 itemsize 53 refs 1 gen 1553656 flags DATA extent data backref root 11288 objectid 74581 offset 131072 count 1 item 129 key (747928526848 EXTENT_ITEM 110592) itemoff 9825 itemsize 53 refs 1 gen 1477712 flags DATA extent data backref root 11288 objectid 38340 offset 3955474432 count 1 item 130 key (747928637440 EXTENT_ITEM 32768) itemoff 9788 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510516736 count 1 item 131 key (747928670208 EXTENT_ITEM 4096) itemoff 9735 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824038 offset 0 count 1 item 132 key (747928674304 EXTENT_ITEM 8192) itemoff 9682 itemsize 53 refs 1 gen 1480427 flags DATA extent data backref root 11288 objectid 38887 offset 0 count 1 item 133 key (747928682496 EXTENT_ITEM 8192) itemoff 9629 itemsize 53 refs 1 gen 1477753 flags DATA extent data backref root 11288 objectid 38401 offset 2228224 count 1 item 134 key (747928690688 EXTENT_ITEM 229376) itemoff 9576 itemsize 53 refs 1 gen 1482583 flags DATA extent data backref root 11288 objectid 38979 offset 5248192512 count 1 item 135 key (747928920064 EXTENT_ITEM 32768) itemoff 9523 itemsize 53 refs 1 gen 1482590 flags DATA extent data backref root 11288 objectid 38979 offset 9702588416 count 1 item 136 key (747928952832 EXTENT_ITEM 262144) itemoff 9470 itemsize 53 refs 1 gen 1482580 flags DATA extent data backref root ROOT_TREE objectid 5552 offset 0 count 1 item 137 key (747929214976 EXTENT_ITEM 16384) itemoff 9417 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943327232 count 1 item 138 key (747929231360 EXTENT_ITEM 16384) itemoff 9364 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943458304 count 1 item 139 key (747929247744 EXTENT_ITEM 4096) itemoff 9311 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943589376 count 1 item 140 key (747929251840 EXTENT_ITEM 524288) itemoff 9258 itemsize 53 refs 1 gen 1480794 flags DATA extent data backref root 11288 objectid 38927 offset 1572864 count 1 item 141 key (747929776128 EXTENT_ITEM 196608) itemoff 9205 itemsize 53 refs 1 gen 1482583 flags DATA extent data backref root 11288 objectid 38979 offset 5799989248 count 1 item 142 key (747929972736 EXTENT_ITEM 65536) itemoff 9152 itemsize 53 refs 1 gen 1482585 flags DATA extent data backref root 11288 objectid 38979 offset 6808727552 count 1 item 143 key (747930038272 EXTENT_ITEM 65536) itemoff 9099 itemsize 53 refs 1 gen 1482586 flags DATA extent data backref root 11288 objectid 38979 offset 6901035008 count 1 item 144 key (747930103808 EXTENT_ITEM 65536) itemoff 9046 itemsize 53 refs 1 gen 1482586 flags DATA extent data backref root 11288 objectid 38979 offset 7791390720 count 1 item 145 key (747930169344 EXTENT_ITEM 61440) itemoff 8993 itemsize 53 refs 1 gen 1530100 flags DATA extent data backref root 11288 objectid 72155 offset 367656960 count 1 item 146 key (747930230784 EXTENT_ITEM 61440) itemoff 8940 itemsize 53 refs 1 gen 1530100 flags DATA extent data backref root 11288 objectid 72155 offset 367788032 count 1 item 147 key (747930292224 EXTENT_ITEM 8192) itemoff 8887 itemsize 53 refs 1 gen 1553656 flags DATA extent data backref root 11288 objectid 74581 offset 262144 count 1 item 148 key (747930300416 EXTENT_ITEM 69632) itemoff 8834 itemsize 53 refs 1 gen 1530011 flags DATA extent data backref root 11288 objectid 72146 offset 0 count 1 item 149 key (747930370048 EXTENT_ITEM 61440) itemoff 8781 itemsize 53 refs 1 gen 1530025 flags DATA extent data backref root 11288 objectid 72148 offset 367656960 count 1 item 150 key (747930431488 EXTENT_ITEM 131072) itemoff 8728 itemsize 53 refs 1 gen 1482583 flags DATA extent data backref root 11288 objectid 38979 offset 6275604480 count 1 item 151 key (747930562560 EXTENT_ITEM 229376) itemoff 8675 itemsize 53 refs 1 gen 1482588 flags DATA extent data backref root 11288 objectid 38979 offset 8650186752 count 1 item 152 key (747930791936 EXTENT_ITEM 32768) itemoff 8622 itemsize 53 refs 1 gen 1482590 flags DATA extent data backref root 11288 objectid 38979 offset 9860608000 count 1 item 153 key (747930824704 EXTENT_ITEM 196608) itemoff 8569 itemsize 53 refs 1 gen 1482582 flags DATA extent data backref root 11288 objectid 38979 offset 6011453440 count 1 item 154 key (747931021312 EXTENT_ITEM 65536) itemoff 8516 itemsize 53 refs 1 gen 1482585 flags DATA extent data backref root 11288 objectid 38979 offset 7213383680 count 1 item 155 key (747931086848 EXTENT_ITEM 94208) itemoff 8463 itemsize 53 refs 1 gen 1477873 flags DATA extent data backref root 11288 objectid 38444 offset 262144 count 1 item 156 key (747931181056 EXTENT_ITEM 20480) itemoff 8410 itemsize 53 refs 1 gen 1477873 flags DATA extent data backref root 11288 objectid 38444 offset 2621440 count 1 item 157 key (747931201536 EXTENT_ITEM 114688) itemoff 8357 itemsize 53 refs 1 gen 1477874 flags DATA extent data backref root 11288 objectid 38444 offset 113442816 count 1 item 158 key (747931316224 EXTENT_ITEM 16384) itemoff 8304 itemsize 53 refs 1 gen 1477879 flags DATA extent data backref root 11288 objectid 38444 offset 848715776 count 1 item 159 key (747931332608 EXTENT_ITEM 4096) itemoff 8251 itemsize 53 refs 1 gen 1477880 flags DATA extent data backref root 11288 objectid 38444 offset 848728064 count 1 item 160 key (747931336704 EXTENT_ITEM 12288) itemoff 8198 itemsize 53 refs 1 gen 1477883 flags DATA extent data backref root 11288 objectid 38448 offset 2752512 count 1 item 161 key (747931348992 EXTENT_ITEM 8192) itemoff 8145 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943601664 count 1 item 162 key (747931357184 EXTENT_ITEM 229376) itemoff 8092 itemsize 53 refs 1 gen 1482585 flags DATA extent data backref root 11288 objectid 38979 offset 6740770816 count 1 item 163 key (747931586560 EXTENT_ITEM 32768) itemoff 8039 itemsize 53 refs 1 gen 1482590 flags DATA extent data backref root 11288 objectid 38979 offset 9932337152 count 1 item 164 key (747931619328 EXTENT_ITEM 36864) itemoff 8002 itemsize 37 refs 1 gen 1592383 flags DATA shared data backref parent 15645802905600 count 1 item 165 key (747931656192 EXTENT_ITEM 36864) itemoff 7965 itemsize 37 refs 1 gen 1592383 flags DATA shared data backref parent 15645802905600 count 1 item 166 key (747931693056 EXTENT_ITEM 36864) itemoff 7928 itemsize 37 refs 1 gen 1592383 flags DATA shared data backref parent 15645802905600 count 1 item 167 key (747931750400 EXTENT_ITEM 36864) itemoff 7891 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 168 key (747931787264 EXTENT_ITEM 36864) itemoff 7854 itemsize 37 refs 1 gen 1592254 flags DATA shared data backref parent 15645510156288 count 1 item 169 key (747931848704 EXTENT_ITEM 4096) itemoff 7801 itemsize 53 refs 1 gen 1590235 flags DATA extent data backref root 11223 objectid 1824046 offset 0 count 1 item 170 key (747931852800 EXTENT_ITEM 24576) itemoff 7748 itemsize 53 refs 1 gen 1539342 flags DATA extent data backref root 11288 objectid 72973 offset 327155712 count 1 item 171 key (747931877376 EXTENT_ITEM 4096) itemoff 7695 itemsize 53 refs 1 gen 1502140 flags DATA extent data backref root 11288 objectid 40707 offset 5736497152 count 1 item 172 key (747931881472 EXTENT_ITEM 8192) itemoff 7642 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943663104 count 1 item 173 key (747931889664 EXTENT_ITEM 65536) itemoff 7589 itemsize 53 refs 1 gen 1482895 flags DATA extent data backref root 11288 objectid 39001 offset 370343936 count 1 item 174 key (747931955200 EXTENT_ITEM 65536) itemoff 7536 itemsize 53 refs 1 gen 1482895 flags DATA extent data backref root 11288 objectid 39001 offset 853999616 count 1 item 175 key (747932020736 EXTENT_ITEM 98304) itemoff 7483 itemsize 53 refs 1 gen 1482897 flags DATA extent data backref root 11288 objectid 39001 offset 1256620032 count 1 item 176 key (747932119040 EXTENT_ITEM 32768) itemoff 7430 itemsize 53 refs 1 gen 1482897 flags DATA extent data backref root 11288 objectid 39001 offset 1269202944 count 1 item 177 key (747932151808 EXTENT_ITEM 139264) itemoff 7377 itemsize 53 refs 1 gen 1509071 flags DATA extent data backref root 11288 objectid 46650 offset 0 count 1 item 178 key (747932291072 EXTENT_ITEM 122880) itemoff 7340 itemsize 37 refs 1 gen 1509093 flags DATA shared data backref parent 12512079577088 count 1 item 179 key (747932413952 EXTENT_ITEM 12288) itemoff 7287 itemsize 53 refs 1 gen 1453686 flags DATA extent data backref root FS_TREE objectid 1003 offset 12943728640 count 1 item 180 key (747932426240 EXTENT_ITEM 262144) itemoff 7234 itemsize 53 refs 1 gen 1472827 flags DATA extent data backref root ROOT_TREE objectid 1376 offset 0 count 1 item 181 key (747932688384 EXTENT_ITEM 262144) itemoff 7181 itemsize 53 refs 1 gen 1482897 flags DATA extent data backref root 11288 objectid 39001 offset 1145307136 count 1 item 182 key (747932950528 EXTENT_ITEM 40960) itemoff 7128 itemsize 53 refs 1 gen 1509030 flags DATA extent data backref root 11288 objectid 43699 offset 0 count 1 item 183 key (747932991488 EXTENT_ITEM 53248) itemoff 7075 itemsize 53 refs 1 gen 1509030 flags DATA extent data backref root 11288 objectid 43700 offset 0 count 1 item 184 key (747933044736 EXTENT_ITEM 69632) itemoff 7022 itemsize 53 refs 1 gen 1509030 flags DATA extent data backref root 11288 objectid 43701 offset 0 count 1 item 185 key (747933114368 EXTENT_ITEM 77824) itemoff 6969 itemsize 53 refs 1 gen 1509030 flags DATA extent data backref root 11288 objectid 43702 offset 0 count 1 item 186 key (747933192192 EXTENT_ITEM 262144) itemoff 6916 itemsize 53 refs 1 gen 1550304 flags DATA extent data backref root ROOT_TREE objectid 57451 offset 0 count 1 item 187 key (747933454336 EXTENT_ITEM 20480) itemoff 6863 itemsize 53 refs 1 gen 1509156 flags DATA extent data backref root 11288 objectid 50591 offset 0 count 1 item 188 key (747933474816 EXTENT_ITEM 262144) itemoff 6810 itemsize 53 refs 1 gen 1550313 flags DATA extent data backref root ROOT_TREE objectid 57448 offset 0 count 1 item 189 key (747933736960 EXTENT_ITEM 262144) itemoff 6757 itemsize 53 refs 1 gen 1472802 flags DATA extent data backref root ROOT_TREE objectid 1368 offset 0 count 1 item 190 key (747933999104 EXTENT_ITEM 16384) itemoff 6704 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12949532672 count 1 item 191 key (747934015488 EXTENT_ITEM 16384) itemoff 6651 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12949663744 count 1 item 192 key (747934031872 EXTENT_ITEM 8192) itemoff 6598 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12949794816 count 1 item 193 key (747934040064 EXTENT_ITEM 8192) itemoff 6545 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12949843968 count 1 item 194 key (747934048256 EXTENT_ITEM 16384) itemoff 6492 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12949905408 count 1 item 195 key (747934064640 EXTENT_ITEM 8192) itemoff 6439 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12950036480 count 1 item 196 key (747934072832 EXTENT_ITEM 12288) itemoff 6386 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12950093824 count 1 item 197 key (747934085120 EXTENT_ITEM 4096) itemoff 6333 itemsize 53 refs 1 gen 1453688 flags DATA extent data backref root FS_TREE objectid 1003 offset 12950089728 count 1 item 198 key (747934089216 EXTENT_ITEM 20480) itemoff 6280 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954370048 count 1 item 199 key (747934109696 EXTENT_ITEM 20480) itemoff 6227 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954501120 count 1 item 200 key (747934130176 EXTENT_ITEM 4096) itemoff 6174 itemsize 53 refs 1 gen 1453690 flags DATA extent data backref root FS_TREE objectid 1003 offset 12954632192 count 1 Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 1:16 ` Marc MERLIN @ 2022-04-05 1:22 ` Josef Bacik 2022-04-05 1:42 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 1:22 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 9:16 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 09:06:01PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 8:58 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Mon, Apr 04, 2022 at 08:39:14PM -0400, Josef Bacik wrote: > > > > On Mon, Apr 4, 2022 at 8:28 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > > > On Mon, Apr 04, 2022 at 08:24:55PM -0400, Josef Bacik wrote: > > > > > > > Binary identical after rebuild. > > > > > > > > > > > > Sigh time for printf sanity checks, thanks, > > > > > > > > > > > > > I'm dumb, try again please, thanks, > > > > > > progress :) > > > > Ok, lets try > > > > btrfs inspect-internal dump-tree -b 13577779511296 > > > > and see what that gives us, the root we think is ok is missing the > > part with the actual root items. Thanks, > Ok lets try again with -b 13577821667328, and if the owner doesn't say ROOT_TREE try 13577775284224 and then 13577814573056. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 1:22 ` Josef Bacik @ 2022-04-05 1:42 ` Marc MERLIN 2022-04-05 1:55 ` Josef Bacik ` (2 more replies) 0 siblings, 3 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 1:42 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote: > Ok lets try again with -b 13577821667328, and if the owner doesn't say > ROOT_TREE try 13577775284224 and then 13577814573056. Thanks, EXTENT_TREE, FS_TREE, and FS_TREE And shit, I got distracted and sent the text output to /dev/mapper/dshelf1a, so I clobbered about 30K of the device. I'm assuming there was probably something there? Script started on Mon 04 Apr 2022 18:36:51 PDT 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m ./btrfs-inspect-internal dump-tree -b 13577821667328 &>/dev/mapper/dshelf1a btrfs inspect-internal dump-tree: not enough arguments: 0 but at least 1 expected 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K./btrfs inspect-internal dump-tree -b 13577821667328 /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55fe04c74870 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55fe04c74870 leaf 13577821667328 items 197 free space 4857 generation 1602242 owner EXTENT_TREE leaf 13577821667328 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (8733701406720 METADATA_ITEM 0) itemoff 16250 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 1 key (8733701423104 METADATA_ITEM 0) itemoff 16217 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 2 key (8733701439488 METADATA_ITEM 0) itemoff 16184 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 3 key (8733701455872 METADATA_ITEM 0) itemoff 16151 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 4 key (8733701472256 METADATA_ITEM 0) itemoff 16118 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 5 key (8733701488640 METADATA_ITEM 0) itemoff 16085 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 6 key (8733701505024 METADATA_ITEM 0) itemoff 16052 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 7 key (8733701521408 METADATA_ITEM 0) itemoff 16019 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 8 key (8733701537792 METADATA_ITEM 0) itemoff 15986 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 9 key (8733701554176 METADATA_ITEM 0) itemoff 15953 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 10 key (8733701570560 METADATA_ITEM 0) itemoff 15920 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 11 key (8733701586944 METADATA_ITEM 0) itemoff 15887 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 12 key (8733701603328 METADATA_ITEM 0) itemoff 15854 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 13 key (8733701619712 METADATA_ITEM 0) itemoff 15821 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 14 key (8733701636096 METADATA_ITEM 0) itemoff 15788 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 15 key (8733701652480 METADATA_ITEM 0) itemoff 15755 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 16 key (8733701668864 METADATA_ITEM 0) itemoff 15722 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 17 key (8733701685248 METADATA_ITEM 0) itemoff 15689 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 18 key (8733701701632 METADATA_ITEM 0) itemoff 15656 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 19 key (8733701718016 METADATA_ITEM 0) itemoff 15623 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 20 key (8733701734400 METADATA_ITEM 0) itemoff 15590 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 21 key (8733701750784 METADATA_ITEM 0) itemoff 15557 itemsize 33 refs 1 gen 1584113 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 22 key (8733701783552 METADATA_ITEM 0) itemoff 15524 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 23 key (8733701799936 METADATA_ITEM 0) itemoff 15491 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 24 key (8733701816320 METADATA_ITEM 0) itemoff 15458 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 25 key (8733701832704 METADATA_ITEM 0) itemoff 15425 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 26 key (8733701865472 METADATA_ITEM 0) itemoff 15392 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 27 key (8733701881856 METADATA_ITEM 0) itemoff 15359 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 28 key (8733701898240 METADATA_ITEM 1) itemoff 15326 itemsize 33 refs 1 gen 1474318 flags TREE_BLOCK tree block skinny level 1 tree block backref root CSUM_TREE item 29 key (8733701914624 METADATA_ITEM 0) itemoff 15293 itemsize 33 refs 1 gen 1474318 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 30 key (8733701931008 METADATA_ITEM 0) itemoff 15260 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 31 key (8733701947392 METADATA_ITEM 0) itemoff 15227 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 32 key (8733701963776 METADATA_ITEM 0) itemoff 15194 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 33 key (8733701980160 METADATA_ITEM 0) itemoff 15161 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 34 key (8733701996544 METADATA_ITEM 0) itemoff 15128 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 35 key (8733702012928 METADATA_ITEM 0) itemoff 15095 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 36 key (8733702029312 METADATA_ITEM 0) itemoff 15062 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 37 key (8733702045696 METADATA_ITEM 0) itemoff 15029 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 38 key (8733702062080 METADATA_ITEM 0) itemoff 14996 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 39 key (8733702078464 METADATA_ITEM 0) itemoff 14963 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 40 key (8733702094848 METADATA_ITEM 0) itemoff 14930 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 41 key (8733702111232 METADATA_ITEM 0) itemoff 14897 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 42 key (8733702127616 METADATA_ITEM 0) itemoff 14864 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 43 key (8733702144000 METADATA_ITEM 0) itemoff 14831 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 44 key (8733702176768 METADATA_ITEM 0) itemoff 14798 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 45 key (8733702193152 METADATA_ITEM 0) itemoff 14765 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 46 key (8733702209536 METADATA_ITEM 0) itemoff 14732 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 47 key (8733702225920 METADATA_ITEM 0) itemoff 14699 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 48 key (8733702242304 METADATA_ITEM 0) itemoff 14666 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 49 key (8733702258688 METADATA_ITEM 0) itemoff 14633 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 50 key (8733702275072 METADATA_ITEM 0) itemoff 14600 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 51 key (8733702291456 METADATA_ITEM 0) itemoff 14567 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 52 key (8733702307840 METADATA_ITEM 0) itemoff 14534 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 53 key (8733702324224 METADATA_ITEM 0) itemoff 14501 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 54 key (8733702340608 METADATA_ITEM 0) itemoff 14468 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 55 key (8733702356992 METADATA_ITEM 0) itemoff 14435 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 56 key (8733702389760 METADATA_ITEM 0) itemoff 14402 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 57 key (8733702406144 METADATA_ITEM 0) itemoff 14369 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 58 key (8733702422528 METADATA_ITEM 0) itemoff 14336 itemsize 33 refs 1 gen 838551 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 59 key (8733702438912 METADATA_ITEM 0) itemoff 14303 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 60 key (8733702455296 METADATA_ITEM 0) itemoff 14270 itemsize 33 refs 1 gen 1360804 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 61 key (8733702504448 METADATA_ITEM 0) itemoff 14237 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 62 key (8733702520832 METADATA_ITEM 0) itemoff 14204 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 63 key (8733702537216 METADATA_ITEM 0) itemoff 14171 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 64 key (8733702553600 METADATA_ITEM 0) itemoff 14138 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 65 key (8733702569984 METADATA_ITEM 0) itemoff 14105 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 66 key (8733702586368 METADATA_ITEM 0) itemoff 14072 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 67 key (8733702602752 METADATA_ITEM 0) itemoff 14039 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 68 key (8733702619136 METADATA_ITEM 0) itemoff 14006 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 69 key (8733702635520 METADATA_ITEM 0) itemoff 13973 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 70 key (8733702651904 METADATA_ITEM 0) itemoff 13940 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 71 key (8733702668288 METADATA_ITEM 0) itemoff 13907 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 72 key (8733702684672 METADATA_ITEM 0) itemoff 13874 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 73 key (8733702701056 METADATA_ITEM 0) itemoff 13841 itemsize 33 refs 1 gen 300259 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 74 key (8733702717440 METADATA_ITEM 0) itemoff 13808 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 75 key (8733702733824 METADATA_ITEM 0) itemoff 13775 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 76 key (8733702750208 METADATA_ITEM 0) itemoff 13742 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 77 key (8733702766592 METADATA_ITEM 0) itemoff 13709 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 78 key (8733702782976 METADATA_ITEM 0) itemoff 13676 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 79 key (8733702799360 METADATA_ITEM 0) itemoff 13643 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 80 key (8733702815744 METADATA_ITEM 0) itemoff 13610 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 81 key (8733702832128 METADATA_ITEM 0) itemoff 13577 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 82 key (8733702848512 METADATA_ITEM 0) itemoff 13544 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 83 key (8733702864896 METADATA_ITEM 0) itemoff 13511 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 84 key (8733702881280 METADATA_ITEM 0) itemoff 13478 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 85 key (8733702897664 METADATA_ITEM 0) itemoff 13445 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 86 key (8733702914048 METADATA_ITEM 0) itemoff 13412 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 87 key (8733702930432 METADATA_ITEM 0) itemoff 13379 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 88 key (8733702946816 METADATA_ITEM 0) itemoff 13346 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 89 key (8733702963200 METADATA_ITEM 0) itemoff 13313 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 90 key (8733702979584 METADATA_ITEM 0) itemoff 13280 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 91 key (8733702995968 METADATA_ITEM 0) itemoff 13247 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 92 key (8733703012352 METADATA_ITEM 0) itemoff 13214 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 93 key (8733703028736 METADATA_ITEM 0) itemoff 13181 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 94 key (8733703045120 METADATA_ITEM 0) itemoff 13148 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 95 key (8733703061504 METADATA_ITEM 0) itemoff 13115 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 96 key (8733703077888 METADATA_ITEM 0) itemoff 13082 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 97 key (8733703094272 METADATA_ITEM 0) itemoff 13049 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 98 key (8733703110656 METADATA_ITEM 0) itemoff 13016 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 99 key (8733703127040 METADATA_ITEM 0) itemoff 12983 itemsize 33 refs 1 gen 3264 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 100 key (8733703143424 METADATA_ITEM 0) itemoff 12950 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 101 key (8733703159808 METADATA_ITEM 0) itemoff 12917 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 102 key (8733703176192 METADATA_ITEM 0) itemoff 12884 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 103 key (8733703192576 METADATA_ITEM 0) itemoff 12851 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 104 key (8733703208960 METADATA_ITEM 0) itemoff 12818 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 105 key (8733703225344 METADATA_ITEM 0) itemoff 12785 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 106 key (8733703241728 METADATA_ITEM 0) itemoff 12752 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 107 key (8733703258112 METADATA_ITEM 0) itemoff 12719 itemsize 33 refs 1 gen 1581638 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 108 key (8733703274496 METADATA_ITEM 0) itemoff 12686 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 109 key (8733703290880 METADATA_ITEM 0) itemoff 12653 itemsize 33 refs 1 gen 1584112 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 110 key (8733703323648 METADATA_ITEM 0) itemoff 12620 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 111 key (8733703340032 METADATA_ITEM 0) itemoff 12587 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 112 key (8733703454720 METADATA_ITEM 0) itemoff 12554 itemsize 33 refs 1 gen 1535773 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 113 key (8733703471104 METADATA_ITEM 0) itemoff 12521 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 114 key (8733703487488 METADATA_ITEM 0) itemoff 12488 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 115 key (8733703503872 METADATA_ITEM 0) itemoff 12455 itemsize 33 refs 1 gen 519869 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 116 key (8733703520256 METADATA_ITEM 0) itemoff 12422 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 117 key (8733703536640 METADATA_ITEM 0) itemoff 12389 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 118 key (8733703553024 METADATA_ITEM 0) itemoff 12356 itemsize 33 refs 1 gen 1538313 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 119 key (8733703569408 METADATA_ITEM 0) itemoff 12323 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 120 key (8733703602176 METADATA_ITEM 0) itemoff 12290 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 121 key (8733703651328 METADATA_ITEM 0) itemoff 12257 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 122 key (8733703733248 METADATA_ITEM 0) itemoff 12224 itemsize 33 refs 1 gen 1510262 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 123 key (8733703749632 METADATA_ITEM 0) itemoff 12191 itemsize 33 refs 1 gen 1601423 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 124 key (8733703766016 METADATA_ITEM 0) itemoff 12158 itemsize 33 refs 1 gen 1509558 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 125 key (8733703798784 METADATA_ITEM 0) itemoff 12125 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 126 key (8733703815168 METADATA_ITEM 0) itemoff 12092 itemsize 33 refs 1 gen 1480353 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 127 key (8733703831552 METADATA_ITEM 0) itemoff 12059 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 128 key (8733703847936 METADATA_ITEM 0) itemoff 12026 itemsize 33 refs 1 gen 3256 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 129 key (8733703864320 METADATA_ITEM 0) itemoff 11993 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 130 key (8733703880704 METADATA_ITEM 0) itemoff 11960 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 131 key (8733703897088 METADATA_ITEM 0) itemoff 11927 itemsize 33 refs 1 gen 1518676 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 132 key (8733703913472 METADATA_ITEM 0) itemoff 11894 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 133 key (8733703929856 METADATA_ITEM 0) itemoff 11861 itemsize 33 refs 1 gen 3246 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 134 key (8733703946240 METADATA_ITEM 0) itemoff 11828 itemsize 33 refs 1 gen 1590258 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11223 item 135 key (8733703979008 METADATA_ITEM 0) itemoff 11795 itemsize 33 refs 1 gen 749275 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 136 key (8733704060928 METADATA_ITEM 0) itemoff 11762 itemsize 33 refs 1 gen 1509163 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 137 key (8733704126464 METADATA_ITEM 0) itemoff 11729 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 138 key (8733704142848 METADATA_ITEM 0) itemoff 11696 itemsize 33 refs 1 gen 1496691 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 139 key (8733704159232 METADATA_ITEM 0) itemoff 11663 itemsize 33 refs 1 gen 1496691 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 140 key (8733704241152 METADATA_ITEM 0) itemoff 11630 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 141 key (8733704257536 METADATA_ITEM 0) itemoff 11597 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 142 key (8733704273920 METADATA_ITEM 0) itemoff 11564 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 143 key (8733704290304 METADATA_ITEM 0) itemoff 11531 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 144 key (8733704306688 METADATA_ITEM 0) itemoff 11498 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 145 key (8733704323072 METADATA_ITEM 0) itemoff 11465 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 146 key (8733704355840 METADATA_ITEM 0) itemoff 11432 itemsize 33 refs 1 gen 1526001 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 147 key (8733704404992 METADATA_ITEM 0) itemoff 11399 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 148 key (8733704421376 METADATA_ITEM 0) itemoff 11366 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 149 key (8733704437760 METADATA_ITEM 0) itemoff 11333 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 150 key (8733704454144 METADATA_ITEM 0) itemoff 11300 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 151 key (8733704470528 METADATA_ITEM 0) itemoff 11267 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 152 key (8733704519680 METADATA_ITEM 0) itemoff 11234 itemsize 33 refs 1 gen 1497583 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 153 key (8733704536064 METADATA_ITEM 0) itemoff 11201 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 154 key (8733704617984 METADATA_ITEM 0) itemoff 11168 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 155 key (8733704667136 METADATA_ITEM 0) itemoff 11135 itemsize 33 refs 1 gen 441944 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 156 key (8733704683520 METADATA_ITEM 0) itemoff 11102 itemsize 33 refs 1 gen 300521 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 157 key (8733704716288 METADATA_ITEM 0) itemoff 11069 itemsize 33 refs 1 gen 4166 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 158 key (8733704765440 METADATA_ITEM 0) itemoff 11036 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 159 key (8733704798208 METADATA_ITEM 0) itemoff 11003 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 160 key (8733704847360 METADATA_ITEM 0) itemoff 10970 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 161 key (8733704863744 METADATA_ITEM 0) itemoff 10937 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 162 key (8733704880128 METADATA_ITEM 0) itemoff 10904 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 163 key (8733704912896 METADATA_ITEM 0) itemoff 10871 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 164 key (8733704945664 METADATA_ITEM 0) itemoff 10838 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 165 key (8733704962048 METADATA_ITEM 0) itemoff 10805 itemsize 33 refs 1 gen 115450 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 166 key (8733704978432 METADATA_ITEM 0) itemoff 10772 itemsize 33 refs 1 gen 736618 flags TREE_BLOCK tree block skinny level 0 shared block backref parent 10679198629888 item 167 key (8733705043968 METADATA_ITEM 0) itemoff 10739 itemsize 33 refs 1 gen 1554576 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 168 key (8733705093120 METADATA_ITEM 0) itemoff 10706 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 169 key (8733705109504 METADATA_ITEM 0) itemoff 10673 itemsize 33 refs 1 gen 1554576 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 170 key (8733705125888 METADATA_ITEM 0) itemoff 10640 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 171 key (8733705142272 METADATA_ITEM 0) itemoff 10607 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 172 key (8733705175040 METADATA_ITEM 0) itemoff 10574 itemsize 33 refs 1 gen 3221 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 173 key (8733705191424 METADATA_ITEM 0) itemoff 10541 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 174 key (8733705207808 METADATA_ITEM 0) itemoff 10508 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 175 key (8733705224192 METADATA_ITEM 0) itemoff 10475 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 176 key (8733705240576 METADATA_ITEM 0) itemoff 10442 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 177 key (8733705256960 METADATA_ITEM 0) itemoff 10409 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 178 key (8733705273344 METADATA_ITEM 0) itemoff 10376 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 179 key (8733705289728 METADATA_ITEM 0) itemoff 10343 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 180 key (8733705306112 METADATA_ITEM 0) itemoff 10310 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 181 key (8733705322496 METADATA_ITEM 0) itemoff 10277 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 182 key (8733705338880 METADATA_ITEM 0) itemoff 10244 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 183 key (8733705453568 METADATA_ITEM 0) itemoff 10211 itemsize 33 refs 1 gen 68949 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 184 key (8733705469952 METADATA_ITEM 0) itemoff 10178 itemsize 33 refs 1 gen 904828 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 185 key (8733705502720 METADATA_ITEM 0) itemoff 10145 itemsize 33 refs 1 gen 1530500 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 186 key (8733705535488 METADATA_ITEM 0) itemoff 10112 itemsize 33 refs 1 gen 1602171 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 187 key (8733705551872 METADATA_ITEM 0) itemoff 10079 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 188 key (8733705568256 METADATA_ITEM 0) itemoff 10046 itemsize 33 refs 1 gen 904814 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 189 key (8733705584640 METADATA_ITEM 0) itemoff 10013 itemsize 33 refs 1 gen 1530500 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 190 key (8733705601024 METADATA_ITEM 0) itemoff 9980 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 191 key (8733705633792 METADATA_ITEM 0) itemoff 9947 itemsize 33 refs 1 gen 3846 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 192 key (8733705650176 METADATA_ITEM 0) itemoff 9914 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 193 key (8733705666560 METADATA_ITEM 0) itemoff 9881 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 194 key (8733705682944 METADATA_ITEM 0) itemoff 9848 itemsize 33 refs 1 gen 904828 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 195 key (8733705699328 METADATA_ITEM 0) itemoff 9815 itemsize 33 refs 1 gen 1514029 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 196 key (8733705715712 METADATA_ITEM 0) itemoff 9782 itemsize 33 refs 1 gen 1522822 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K.s./btrfsfinspect-internaludump-treeb-b 313577775284224 /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x56312cc84870 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x56312cc84870 leaf 13577775284224 items 102 free space 8215 generation 1602088 owner FS_TREE leaf 13577775284224 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 3 transid 1602088 size 4638 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 152572 flags 0x0(none) atime 1518920225.99299649 (2018-02-17 18:17:05) ctime 1648458062.674404554 (2022-03-28 02:01:02) mtime 1648458062.674404554 (2022-03-28 02:01:02) otime 1518411216.0 (2018-02-11 20:53:36) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 index 0 namelen 2 name: .. item 2 key (256 DIR_ITEM 35173225) itemoff 16055 itemsize 56 location key (165295 ROOT_ITEM 18446744073709551615) type DIR transid 1601935 data_len 0 name_len 26 name: other_ro.20220327_00:24:58 item 3 key (256 DIR_ITEM 37377682) itemoff 15995 itemsize 60 location key (165387 ROOT_ITEM 18446744073709551615) type DIR transid 1602062 data_len 0 name_len 30 name: other_hourly.20220327_22:01:01 item 4 key (256 DIR_ITEM 65985264) itemoff 15939 itemsize 56 location key (164623 ROOT_ITEM 18446744073709551615) type DIR transid 1601065 data_len 0 name_len 26 name: Sound_ro.20220320_00:39:38 item 5 key (256 DIR_ITEM 121085187) itemoff 15885 itemsize 54 location key (164633 ROOT_ITEM 18446744073709551615) type DIR transid 1601080 data_len 0 name_len 24 name: Win_ro.20220320_01:12:38 item 6 key (256 DIR_ITEM 122346812) itemoff 15829 itemsize 56 location key (163920 ROOT_ITEM 18446744073709551615) type DIR transid 1599635 data_len 0 name_len 26 name: Video_ro.20220313_00:38:01 item 7 key (256 DIR_ITEM 167852107) itemoff 15769 itemsize 60 location key (159786 ROOT_ITEM 18446744073709551615) type DIR transid 1591390 data_len 0 name_len 30 name: Sound_weekly.20220130_02:03:01 item 8 key (256 DIR_ITEM 185979643) itemoff 15714 itemsize 55 location key (164622 ROOT_ITEM 18446744073709551615) type DIR transid 1601062 data_len 0 name_len 25 name: Soft_ro.20220320_00:35:42 item 9 key (256 DIR_ITEM 188444463) itemoff 15655 itemsize 59 location key (165400 ROOT_ITEM 18446744073709551615) type DIR transid 1602080 data_len 0 name_len 29 name: Video_daily.20220328_01:02:01 item 10 key (256 DIR_ITEM 222467894) itemoff 15596 itemsize 59 location key (165100 ROOT_ITEM 18446744073709551615) type DIR transid 1601682 data_len 0 name_len 29 name: media_daily.20220325_01:02:01 item 11 key (256 DIR_ITEM 276733595) itemoff 15537 itemsize 59 location key (165307 ROOT_ITEM 18446744073709551615) type DIR transid 1601959 data_len 0 name_len 29 name: other_daily.20220327_01:02:01 item 12 key (256 DIR_ITEM 341062093) itemoff 15481 itemsize 56 location key (163917 ROOT_ITEM 18446744073709551615) type DIR transid 1599627 data_len 0 name_len 26 name: other_ro.20220313_00:26:11 item 13 key (256 DIR_ITEM 350762628) itemoff 15425 itemsize 56 location key (162632 ROOT_ITEM 18446744073709551615) type DIR transid 1596353 data_len 0 name_len 26 name: Video_ro.20220227_00:46:32 item 14 key (256 DIR_ITEM 367443219) itemoff 15365 itemsize 60 location key (165399 ROOT_ITEM 18446744073709551615) type DIR transid 1602078 data_len 0 name_len 30 name: other_hourly.20220328_01:01:01 item 15 key (256 DIR_ITEM 392444674) itemoff 15326 itemsize 39 location key (103481 ROOT_ITEM 18446744073709551615) type DIR transid 1455725 data_len 0 name_len 9 name: .beeshome item 16 key (256 DIR_ITEM 597156124) itemoff 15266 itemsize 60 location key (165398 ROOT_ITEM 18446744073709551615) type DIR transid 1602077 data_len 0 name_len 30 name: media_hourly.20220328_01:01:01 item 17 key (256 DIR_ITEM 640492716) itemoff 15206 itemsize 60 location key (165396 ROOT_ITEM 18446744073709551615) type DIR transid 1602075 data_len 0 name_len 30 name: Video_hourly.20220328_01:01:01 item 18 key (256 DIR_ITEM 668082479) itemoff 15168 itemsize 38 location key (1496 INODE_ITEM 0) type SYMLINK transid 1601950 data_len 0 name_len 8 name: Win_last item 19 key (256 DIR_ITEM 713697142) itemoff 15108 itemsize 60 location key (161198 ROOT_ITEM 18446744073709551615) type DIR transid 1594244 data_len 0 name_len 30 name: Sound_weekly.20220213_02:03:01 item 20 key (256 DIR_ITEM 725111845) itemoff 15053 itemsize 55 location key (163918 ROOT_ITEM 18446744073709551615) type DIR transid 1599629 data_len 0 name_len 25 name: Soft_ro.20220313_00:33:42 item 21 key (256 DIR_ITEM 769653260) itemoff 14994 itemsize 59 location key (165403 ROOT_ITEM 18446744073709551615) type DIR transid 1602083 data_len 0 name_len 29 name: other_daily.20220328_01:02:01 item 22 key (256 DIR_ITEM 916065208) itemoff 14935 itemsize 59 location key (165304 ROOT_ITEM 18446744073709551615) type DIR transid 1601956 data_len 0 name_len 29 name: Video_daily.20220327_01:02:01 item 23 key (256 DIR_ITEM 971435966) itemoff 14879 itemsize 56 location key (163916 ROOT_ITEM 18446744073709551615) type DIR transid 1599626 data_len 0 name_len 26 name: media_ro.20220313_00:24:44 item 24 key (256 DIR_ITEM 981009741) itemoff 14846 itemsize 33 location key (11224 ROOT_ITEM 18446744073709551615) type DIR transid 4062 data_len 0 name_len 3 name: Win item 25 key (256 DIR_ITEM 1200370785) itemoff 14786 itemsize 60 location key (165405 ROOT_ITEM 18446744073709551615) type DIR transid 1602086 data_len 0 name_len 30 name: Sound_hourly.20220328_02:01:01 item 26 key (256 DIR_ITEM 1268043220) itemoff 14726 itemsize 60 location key (165394 ROOT_ITEM 18446744073709551615) type DIR transid 1602072 data_len 0 name_len 30 name: media_hourly.20220328_00:01:01 item 27 key (256 DIR_ITEM 1288340042) itemoff 14691 itemsize 35 location key (11222 ROOT_ITEM 18446744073709551615) type DIR transid 4058 data_len 0 name_len 5 name: media item 28 key (256 DIR_ITEM 1311639652) itemoff 14631 itemsize 60 location key (165392 ROOT_ITEM 18446744073709551615) type DIR transid 1602070 data_len 0 name_len 30 name: Video_hourly.20220328_00:01:01 item 29 key (256 DIR_ITEM 1388422445) itemoff 14575 itemsize 56 location key (162628 ROOT_ITEM 18446744073709551615) type DIR transid 1596345 data_len 0 name_len 26 name: media_ro.20220227_00:24:39 item 30 key (256 DIR_ITEM 1431596328) itemoff 14516 itemsize 59 location key (165200 ROOT_ITEM 18446744073709551615) type DIR transid 1601811 data_len 0 name_len 29 name: media_daily.20220326_01:02:01 item 31 key (256 DIR_ITEM 1438703253) itemoff 14456 itemsize 60 location key (160495 ROOT_ITEM 18446744073709551615) type DIR transid 1592683 data_len 0 name_len 30 name: Sound_weekly.20220206_02:03:01 item 32 key (256 DIR_ITEM 1440530326) itemoff 14423 itemsize 33 location key (1003 INODE_ITEM 0) type FILE transid 1448943 data_len 0 name_len 3 name: DS1 item 33 key (256 DIR_ITEM 1476660117) itemoff 14364 itemsize 59 location key (165401 ROOT_ITEM 18446744073709551615) type DIR transid 1602081 data_len 0 name_len 29 name: Sound_daily.20220328_01:02:01 item 34 key (256 DIR_ITEM 1509090277) itemoff 14304 itemsize 60 location key (165388 ROOT_ITEM 18446744073709551615) type DIR transid 1602064 data_len 0 name_len 30 name: Video_hourly.20220327_23:01:01 item 35 key (256 DIR_ITEM 1516701308) itemoff 14250 itemsize 54 location key (165299 ROOT_ITEM 18446744073709551615) type DIR transid 1601947 data_len 0 name_len 24 name: Win_ro.20220327_00:37:15 item 36 key (256 DIR_ITEM 1548232277) itemoff 14190 itemsize 60 location key (165390 ROOT_ITEM 18446744073709551615) type DIR transid 1602066 data_len 0 name_len 30 name: media_hourly.20220327_23:01:01 item 37 key (256 DIR_ITEM 1578913405) itemoff 14155 itemsize 35 location key (11322 ROOT_ITEM 18446744073709551615) type DIR transid 4223 data_len 0 name_len 5 name: Sound item 38 key (256 DIR_ITEM 1597000160) itemoff 14115 itemsize 40 location key (1494 INODE_ITEM 0) type SYMLINK transid 1601944 data_len 0 name_len 10 name: Sound_last item 39 key (256 DIR_ITEM 1689324197) itemoff 14055 itemsize 60 location key (163317 ROOT_ITEM 18446744073709551615) type DIR transid 1597264 data_len 0 name_len 30 name: Sound_weekly.20220306_02:03:01 item 40 key (256 DIR_ITEM 1705340674) itemoff 13996 itemsize 59 location key (165305 ROOT_ITEM 18446744073709551615) type DIR transid 1601957 data_len 0 name_len 29 name: Sound_daily.20220327_01:02:01 item 41 key (256 DIR_ITEM 1731806220) itemoff 13940 itemsize 56 location key (165298 ROOT_ITEM 18446744073709551615) type DIR transid 1601944 data_len 0 name_len 26 name: Video_ro.20220327_00:35:50 item 42 key (256 DIR_ITEM 1782151770) itemoff 13880 itemsize 60 location key (165391 ROOT_ITEM 18446744073709551615) type DIR transid 1602067 data_len 0 name_len 30 name: other_hourly.20220327_23:01:01 item 43 key (256 DIR_ITEM 1808375180) itemoff 13824 itemsize 56 location key (163301 ROOT_ITEM 18446744073709551615) type DIR transid 1597239 data_len 0 name_len 26 name: Sound_ro.20220306_00:45:46 item 44 key (256 DIR_ITEM 1925286825) itemoff 13768 itemsize 56 location key (164621 ROOT_ITEM 18446744073709551615) type DIR transid 1601060 data_len 0 name_len 26 name: other_ro.20220320_00:25:35 item 45 key (256 DIR_ITEM 2030955372) itemoff 13712 itemsize 56 location key (163919 ROOT_ITEM 18446744073709551615) type DIR transid 1599632 data_len 0 name_len 26 name: Sound_ro.20220313_00:37:25 item 46 key (256 DIR_ITEM 2112197083) itemoff 13652 itemsize 60 location key (165395 ROOT_ITEM 18446744073709551615) type DIR transid 1602073 data_len 0 name_len 30 name: other_hourly.20220328_00:01:01 item 47 key (256 DIR_ITEM 2127973047) itemoff 13617 itemsize 35 location key (11221 ROOT_ITEM 18446744073709551615) type DIR transid 4042 data_len 0 name_len 5 name: other item 48 key (256 DIR_ITEM 2146234148) itemoff 13583 itemsize 34 location key (11223 ROOT_ITEM 18446744073709551615) type DIR transid 4061 data_len 0 name_len 4 name: Soft item 49 key (256 DIR_ITEM 2153154160) itemoff 13523 itemsize 60 location key (165389 ROOT_ITEM 18446744073709551615) type DIR transid 1602065 data_len 0 name_len 30 name: Sound_hourly.20220327_23:01:01 item 50 key (256 DIR_ITEM 2201886496) itemoff 13464 itemsize 59 location key (165101 ROOT_ITEM 18446744073709551615) type DIR transid 1601683 data_len 0 name_len 29 name: other_daily.20220325_01:02:01 item 51 key (256 DIR_ITEM 2279840979) itemoff 13408 itemsize 56 location key (162629 ROOT_ITEM 18446744073709551615) type DIR transid 1596346 data_len 0 name_len 26 name: other_ro.20220227_00:25:38 item 52 key (256 DIR_ITEM 2313060016) itemoff 13348 itemsize 60 location key (160496 ROOT_ITEM 18446744073709551615) type DIR transid 1592684 data_len 0 name_len 30 name: media_weekly.20220206_02:03:01 item 53 key (256 DIR_ITEM 2354277240) itemoff 13308 itemsize 40 location key (1492 INODE_ITEM 0) type SYMLINK transid 1601938 data_len 0 name_len 10 name: other_last item 54 key (256 DIR_ITEM 2355411712) itemoff 13248 itemsize 60 location key (160494 ROOT_ITEM 18446744073709551615) type DIR transid 1592682 data_len 0 name_len 30 name: Video_weekly.20220206_02:03:01 item 55 key (256 DIR_ITEM 2377365299) itemoff 13192 itemsize 56 location key (165297 ROOT_ITEM 18446744073709551615) type DIR transid 1601941 data_len 0 name_len 26 name: Sound_ro.20220327_00:35:22 item 56 key (256 DIR_ITEM 2396976783) itemoff 13132 itemsize 60 location key (163319 ROOT_ITEM 18446744073709551615) type DIR transid 1597266 data_len 0 name_len 30 name: other_weekly.20220306_02:03:01 item 57 key (256 DIR_ITEM 2456187574) itemoff 13076 itemsize 56 location key (163298 ROOT_ITEM 18446744073709551615) type DIR transid 1597234 data_len 0 name_len 26 name: media_ro.20220306_00:24:43 item 58 key (256 DIR_ITEM 2467975514) itemoff 13021 itemsize 55 location key (163300 ROOT_ITEM 18446744073709551615) type DIR transid 1597237 data_len 0 name_len 25 name: Soft_ro.20220306_00:33:55 item 59 key (256 DIR_ITEM 2542431729) itemoff 12961 itemsize 60 location key (165393 ROOT_ITEM 18446744073709551615) type DIR transid 1602071 data_len 0 name_len 30 name: Sound_hourly.20220328_00:01:01 item 60 key (256 DIR_ITEM 2610074692) itemoff 12901 itemsize 60 location key (165406 ROOT_ITEM 18446744073709551615) type DIR transid 1602087 data_len 0 name_len 30 name: media_hourly.20220328_02:01:01 item 61 key (256 DIR_ITEM 2650936973) itemoff 12842 itemsize 59 location key (165306 ROOT_ITEM 18446744073709551615) type DIR transid 1601958 data_len 0 name_len 29 name: media_daily.20220327_01:02:01 item 62 key (256 DIR_ITEM 2653442548) itemoff 12782 itemsize 60 location key (165404 ROOT_ITEM 18446744073709551615) type DIR transid 1602085 data_len 0 name_len 30 name: Video_hourly.20220328_02:01:01 item 63 key (256 DIR_ITEM 2717852207) itemoff 12728 itemsize 54 location key (163921 ROOT_ITEM 18446744073709551615) type DIR transid 1599637 data_len 0 name_len 24 name: Win_ro.20220313_00:38:28 item 64 key (256 DIR_ITEM 2745397786) itemoff 12669 itemsize 59 location key (165402 ROOT_ITEM 18446744073709551615) type DIR transid 1602082 data_len 0 name_len 29 name: media_daily.20220328_01:02:01 item 65 key (256 DIR_ITEM 2782562819) itemoff 12610 itemsize 59 location key (165098 ROOT_ITEM 18446744073709551615) type DIR transid 1601680 data_len 0 name_len 29 name: Video_daily.20220325_01:02:01 item 66 key (256 DIR_ITEM 2793356223) itemoff 12554 itemsize 56 location key (163299 ROOT_ITEM 18446744073709551615) type DIR transid 1597235 data_len 0 name_len 26 name: other_ro.20220306_00:25:40 item 67 key (256 DIR_ITEM 2811536125) itemoff 12519 itemsize 35 location key (11288 ROOT_ITEM 18446744073709551615) type DIR transid 4174 data_len 0 name_len 5 name: Video item 68 key (256 DIR_ITEM 2827986674) itemoff 12485 itemsize 34 location key (1010 INODE_ITEM 0) type FILE transid 1455723 data_len 0 name_len 4 name: copy item 69 key (256 DIR_ITEM 2917386315) itemoff 12425 itemsize 60 location key (165407 ROOT_ITEM 18446744073709551615) type DIR transid 1602088 data_len 0 name_len 30 name: other_hourly.20220328_02:01:01 item 70 key (256 DIR_ITEM 2935166119) itemoff 12366 itemsize 59 location key (165199 ROOT_ITEM 18446744073709551615) type DIR transid 1601810 data_len 0 name_len 29 name: Sound_daily.20220326_01:02:01 item 71 key (256 DIR_ITEM 2935255735) itemoff 12311 itemsize 55 location key (165296 ROOT_ITEM 18446744073709551615) type DIR transid 1601938 data_len 0 name_len 25 name: Soft_ro.20220327_00:32:46 item 72 key (256 DIR_ITEM 3098520192) itemoff 12251 itemsize 60 location key (163318 ROOT_ITEM 18446744073709551615) type DIR transid 1597265 data_len 0 name_len 30 name: media_weekly.20220306_02:03:01 item 73 key (256 DIR_ITEM 3170169652) itemoff 12195 itemsize 56 location key (164624 ROOT_ITEM 18446744073709551615) type DIR transid 1601068 data_len 0 name_len 26 name: Video_ro.20220320_00:40:27 item 74 key (256 DIR_ITEM 3172298544) itemoff 12135 itemsize 60 location key (163316 ROOT_ITEM 18446744073709551615) type DIR transid 1597263 data_len 0 name_len 30 name: Video_weekly.20220306_02:03:01 item 75 key (256 DIR_ITEM 3215925951) itemoff 12075 itemsize 60 location key (160497 ROOT_ITEM 18446744073709551615) type DIR transid 1592685 data_len 0 name_len 30 name: other_weekly.20220206_02:03:01 item 76 key (256 DIR_ITEM 3236289372) itemoff 12015 itemsize 60 location key (161200 ROOT_ITEM 18446744073709551615) type DIR transid 1594246 data_len 0 name_len 30 name: other_weekly.20220213_02:03:01 item 77 key (256 DIR_ITEM 3290748772) itemoff 11959 itemsize 56 location key (163302 ROOT_ITEM 18446744073709551615) type DIR transid 1597242 data_len 0 name_len 26 name: Video_ro.20220306_00:46:14 item 78 key (256 DIR_ITEM 3360445642) itemoff 11903 itemsize 56 location key (162631 ROOT_ITEM 18446744073709551615) type DIR transid 1596351 data_len 0 name_len 26 name: Sound_ro.20220227_00:40:41 item 79 key (256 DIR_ITEM 3470690534) itemoff 11847 itemsize 56 location key (164620 ROOT_ITEM 18446744073709551615) type DIR transid 1601059 data_len 0 name_len 26 name: media_ro.20220320_00:24:47 item 80 key (256 DIR_ITEM 3550830046) itemoff 11787 itemsize 60 location key (159785 ROOT_ITEM 18446744073709551615) type DIR transid 1591389 data_len 0 name_len 30 name: Video_weekly.20220130_02:03:01 item 81 key (256 DIR_ITEM 3592394862) itemoff 11727 itemsize 60 location key (159787 ROOT_ITEM 18446744073709551615) type DIR transid 1591391 data_len 0 name_len 30 name: media_weekly.20220130_02:03:01 item 82 key (256 DIR_ITEM 3676890430) itemoff 11668 itemsize 59 location key (165201 ROOT_ITEM 18446744073709551615) type DIR transid 1601812 data_len 0 name_len 29 name: other_daily.20220326_01:02:01 item 83 key (256 DIR_ITEM 3765355617) itemoff 11608 itemsize 60 location key (159788 ROOT_ITEM 18446744073709551615) type DIR transid 1591392 data_len 0 name_len 30 name: other_weekly.20220130_02:03:01 item 84 key (256 DIR_ITEM 3787964139) itemoff 11554 itemsize 54 location key (163303 ROOT_ITEM 18446744073709551615) type DIR transid 1597244 data_len 0 name_len 24 name: Win_ro.20220306_00:47:32 item 85 key (256 DIR_ITEM 3799902853) itemoff 11514 itemsize 40 location key (1491 INODE_ITEM 0) type SYMLINK transid 1601935 data_len 0 name_len 10 name: media_last item 86 key (256 DIR_ITEM 3872736471) itemoff 11460 itemsize 54 location key (162645 ROOT_ITEM 18446744073709551615) type DIR transid 1596372 data_len 0 name_len 24 name: Win_ro.20220227_02:10:25 item 87 key (256 DIR_ITEM 3951915213) itemoff 11420 itemsize 40 location key (1495 INODE_ITEM 0) type SYMLINK transid 1601947 data_len 0 name_len 10 name: Video_last item 88 key (256 DIR_ITEM 4079895267) itemoff 11360 itemsize 60 location key (161197 ROOT_ITEM 18446744073709551615) type DIR transid 1594243 data_len 0 name_len 30 name: Video_weekly.20220213_02:03:01 item 89 key (256 DIR_ITEM 4084751325) itemoff 11321 itemsize 39 location key (1493 INODE_ITEM 0) type SYMLINK transid 1601941 data_len 0 name_len 9 name: Soft_last item 90 key (256 DIR_ITEM 4136929107) itemoff 11261 itemsize 60 location key (161199 ROOT_ITEM 18446744073709551615) type DIR transid 1594245 data_len 0 name_len 30 name: media_weekly.20220213_02:03:01 item 91 key (256 DIR_ITEM 4142263993) itemoff 11202 itemsize 59 location key (165099 ROOT_ITEM 18446744073709551615) type DIR transid 1601681 data_len 0 name_len 29 name: Sound_daily.20220325_01:02:01 item 92 key (256 DIR_ITEM 4156034002) itemoff 11147 itemsize 55 location key (162630 ROOT_ITEM 18446744073709551615) type DIR transid 1596348 data_len 0 name_len 25 name: Soft_ro.20220227_00:34:48 item 93 key (256 DIR_ITEM 4198095398) itemoff 11091 itemsize 56 location key (165294 ROOT_ITEM 18446744073709551615) type DIR transid 1601934 data_len 0 name_len 26 name: media_ro.20220327_00:24:32 item 94 key (256 DIR_ITEM 4258033693) itemoff 11032 itemsize 59 location key (165198 ROOT_ITEM 18446744073709551615) type DIR transid 1601809 data_len 0 name_len 29 name: Video_daily.20220326_01:02:01 item 95 key (256 DIR_ITEM 4287185209) itemoff 10972 itemsize 60 location key (165397 ROOT_ITEM 18446744073709551615) type DIR transid 1602076 data_len 0 name_len 30 name: Sound_hourly.20220328_01:01:01 item 96 key (256 DIR_INDEX 8) itemoff 10937 itemsize 35 location key (11221 ROOT_ITEM 18446744073709551615) type DIR transid 4042 data_len 0 name_len 5 name: other item 97 key (256 DIR_INDEX 9) itemoff 10902 itemsize 35 location key (11222 ROOT_ITEM 18446744073709551615) type DIR transid 4058 data_len 0 name_len 5 name: media item 98 key (256 DIR_INDEX 10) itemoff 10868 itemsize 34 location key (11223 ROOT_ITEM 18446744073709551615) type DIR transid 4061 data_len 0 name_len 4 name: Soft item 99 key (256 DIR_INDEX 11) itemoff 10835 itemsize 33 location key (11224 ROOT_ITEM 18446744073709551615) type DIR transid 4062 data_len 0 name_len 3 name: Win item 100 key (256 DIR_INDEX 17) itemoff 10800 itemsize 35 location key (11288 ROOT_ITEM 18446744073709551615) type DIR transid 4174 data_len 0 name_len 5 name: Video item 101 key (256 DIR_INDEX 51) itemoff 10765 itemsize 35 location key (11322 ROOT_ITEM 18446744073709551615) type DIR transid 4223 data_len 0 name_len 5 name: Sound 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m K.s./btrfsfinspect-internaludump-tree -b 313577775284224 /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55fb3b29e870 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55fb3b29e870 leaf 13577775284224 items 102 free space 8215 generation 1602088 owner FS_TREE leaf 13577775284224 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 3 transid 1602088 size 4638 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 152572 flags 0x0(none) atime 1518920225.99299649 (2018-02-17 18:17:05) ctime 1648458062.674404554 (2022-03-28 02:01:02) mtime 1648458062.674404554 (2022-03-28 02:01:02) otime 1518411216.0 (2018-02-11 20:53:36) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 index 0 namelen 2 name: .. item 2 key (256 DIR_ITEM 35173225) itemoff 16055 itemsize 56 location key (165295 ROOT_ITEM 18446744073709551615) type DIR transid 1601935 data_len 0 name_len 26 name: other_ro.20220327_00:24:58 item 3 key (256 DIR_ITEM 37377682) itemoff 15995 itemsize 60 location key (165387 ROOT_ITEM 18446744073709551615) type DIR transid 1602062 data_len 0 name_len 30 name: other_hourly.20220327_22:01:01 item 4 key (256 DIR_ITEM 65985264) itemoff 15939 itemsize 56 location key (164623 ROOT_ITEM 18446744073709551615) type DIR transid 1601065 data_len 0 name_len 26 name: Sound_ro.20220320_00:39:38 item 5 key (256 DIR_ITEM 121085187) itemoff 15885 itemsize 54 location key (164633 ROOT_ITEM 18446744073709551615) type DIR transid 1601080 data_len 0 name_len 24 name: Win_ro.20220320_01:12:38 item 6 key (256 DIR_ITEM 122346812) itemoff 15829 itemsize 56 location key (163920 ROOT_ITEM 18446744073709551615) type DIR transid 1599635 data_len 0 name_len 26 name: Video_ro.20220313_00:38:01 item 7 key (256 DIR_ITEM 167852107) itemoff 15769 itemsize 60 location key (159786 ROOT_ITEM 18446744073709551615) type DIR transid 1591390 data_len 0 name_len 30 name: Sound_weekly.20220130_02:03:01 item 8 key (256 DIR_ITEM 185979643) itemoff 15714 itemsize 55 location key (164622 ROOT_ITEM 18446744073709551615) type DIR transid 1601062 data_len 0 name_len 25 name: Soft_ro.20220320_00:35:42 item 9 key (256 DIR_ITEM 188444463) itemoff 15655 itemsize 59 location key (165400 ROOT_ITEM 18446744073709551615) type DIR transid 1602080 data_len 0 name_len 29 name: Video_daily.20220328_01:02:01 item 10 key (256 DIR_ITEM 222467894) itemoff 15596 itemsize 59 location key (165100 ROOT_ITEM 18446744073709551615) type DIR transid 1601682 data_len 0 name_len 29 name: media_daily.20220325_01:02:01 item 11 key (256 DIR_ITEM 276733595) itemoff 15537 itemsize 59 location key (165307 ROOT_ITEM 18446744073709551615) type DIR transid 1601959 data_len 0 name_len 29 name: other_daily.20220327_01:02:01 item 12 key (256 DIR_ITEM 341062093) itemoff 15481 itemsize 56 location key (163917 ROOT_ITEM 18446744073709551615) type DIR transid 1599627 data_len 0 name_len 26 name: other_ro.20220313_00:26:11 item 13 key (256 DIR_ITEM 350762628) itemoff 15425 itemsize 56 location key (162632 ROOT_ITEM 18446744073709551615) type DIR transid 1596353 data_len 0 name_len 26 name: Video_ro.20220227_00:46:32 item 14 key (256 DIR_ITEM 367443219) itemoff 15365 itemsize 60 location key (165399 ROOT_ITEM 18446744073709551615) type DIR transid 1602078 data_len 0 name_len 30 name: other_hourly.20220328_01:01:01 item 15 key (256 DIR_ITEM 392444674) itemoff 15326 itemsize 39 location key (103481 ROOT_ITEM 18446744073709551615) type DIR transid 1455725 data_len 0 name_len 9 name: .beeshome item 16 key (256 DIR_ITEM 597156124) itemoff 15266 itemsize 60 location key (165398 ROOT_ITEM 18446744073709551615) type DIR transid 1602077 data_len 0 name_len 30 name: media_hourly.20220328_01:01:01 item 17 key (256 DIR_ITEM 640492716) itemoff 15206 itemsize 60 location key (165396 ROOT_ITEM 18446744073709551615) type DIR transid 1602075 data_len 0 name_len 30 name: Video_hourly.20220328_01:01:01 item 18 key (256 DIR_ITEM 668082479) itemoff 15168 itemsize 38 location key (1496 INODE_ITEM 0) type SYMLINK transid 1601950 data_len 0 name_len 8 name: Win_last item 19 key (256 DIR_ITEM 713697142) itemoff 15108 itemsize 60 location key (161198 ROOT_ITEM 18446744073709551615) type DIR transid 1594244 data_len 0 name_len 30 name: Sound_weekly.20220213_02:03:01 item 20 key (256 DIR_ITEM 725111845) itemoff 15053 itemsize 55 location key (163918 ROOT_ITEM 18446744073709551615) type DIR transid 1599629 data_len 0 name_len 25 name: Soft_ro.20220313_00:33:42 item 21 key (256 DIR_ITEM 769653260) itemoff 14994 itemsize 59 location key (165403 ROOT_ITEM 18446744073709551615) type DIR transid 1602083 data_len 0 name_len 29 name: other_daily.20220328_01:02:01 item 22 key (256 DIR_ITEM 916065208) itemoff 14935 itemsize 59 location key (165304 ROOT_ITEM 18446744073709551615) type DIR transid 1601956 data_len 0 name_len 29 name: Video_daily.20220327_01:02:01 item 23 key (256 DIR_ITEM 971435966) itemoff 14879 itemsize 56 location key (163916 ROOT_ITEM 18446744073709551615) type DIR transid 1599626 data_len 0 name_len 26 name: media_ro.20220313_00:24:44 item 24 key (256 DIR_ITEM 981009741) itemoff 14846 itemsize 33 location key (11224 ROOT_ITEM 18446744073709551615) type DIR transid 4062 data_len 0 name_len 3 name: Win item 25 key (256 DIR_ITEM 1200370785) itemoff 14786 itemsize 60 location key (165405 ROOT_ITEM 18446744073709551615) type DIR transid 1602086 data_len 0 name_len 30 name: Sound_hourly.20220328_02:01:01 item 26 key (256 DIR_ITEM 1268043220) itemoff 14726 itemsize 60 location key (165394 ROOT_ITEM 18446744073709551615) type DIR transid 1602072 data_len 0 name_len 30 name: media_hourly.20220328_00:01:01 item 27 key (256 DIR_ITEM 1288340042) itemoff 14691 itemsize 35 location key (11222 ROOT_ITEM 18446744073709551615) type DIR transid 4058 data_len 0 name_len 5 name: media item 28 key (256 DIR_ITEM 1311639652) itemoff 14631 itemsize 60 location key (165392 ROOT_ITEM 18446744073709551615) type DIR transid 1602070 data_len 0 name_len 30 name: Video_hourly.20220328_00:01:01 item 29 key (256 DIR_ITEM 1388422445) itemoff 14575 itemsize 56 location key (162628 ROOT_ITEM 18446744073709551615) type DIR transid 1596345 data_len 0 name_len 26 name: media_ro.20220227_00:24:39 item 30 key (256 DIR_ITEM 1431596328) itemoff 14516 itemsize 59 location key (165200 ROOT_ITEM 18446744073709551615) type DIR transid 1601811 data_len 0 name_len 29 name: media_daily.20220326_01:02:01 item 31 key (256 DIR_ITEM 1438703253) itemoff 14456 itemsize 60 location key (160495 ROOT_ITEM 18446744073709551615) type DIR transid 1592683 data_len 0 name_len 30 name: Sound_weekly.20220206_02:03:01 item 32 key (256 DIR_ITEM 1440530326) itemoff 14423 itemsize 33 location key (1003 INODE_ITEM 0) type FILE transid 1448943 data_len 0 name_len 3 name: DS1 item 33 key (256 DIR_ITEM 1476660117) itemoff 14364 itemsize 59 location key (165401 ROOT_ITEM 18446744073709551615) type DIR transid 1602081 data_len 0 name_len 29 name: Sound_daily.20220328_01:02:01 item 34 key (256 DIR_ITEM 1509090277) itemoff 14304 itemsize 60 location key (165388 ROOT_ITEM 18446744073709551615) type DIR transid 1602064 data_len 0 name_len 30 name: Video_hourly.20220327_23:01:01 item 35 key (256 DIR_ITEM 1516701308) itemoff 14250 itemsize 54 location key (165299 ROOT_ITEM 18446744073709551615) type DIR transid 1601947 data_len 0 name_len 24 name: Win_ro.20220327_00:37:15 item 36 key (256 DIR_ITEM 1548232277) itemoff 14190 itemsize 60 location key (165390 ROOT_ITEM 18446744073709551615) type DIR transid 1602066 data_len 0 name_len 30 name: media_hourly.20220327_23:01:01 item 37 key (256 DIR_ITEM 1578913405) itemoff 14155 itemsize 35 location key (11322 ROOT_ITEM 18446744073709551615) type DIR transid 4223 data_len 0 name_len 5 name: Sound item 38 key (256 DIR_ITEM 1597000160) itemoff 14115 itemsize 40 location key (1494 INODE_ITEM 0) type SYMLINK transid 1601944 data_len 0 name_len 10 name: Sound_last item 39 key (256 DIR_ITEM 1689324197) itemoff 14055 itemsize 60 location key (163317 ROOT_ITEM 18446744073709551615) type DIR transid 1597264 data_len 0 name_len 30 name: Sound_weekly.20220306_02:03:01 item 40 key (256 DIR_ITEM 1705340674) itemoff 13996 itemsize 59 location key (165305 ROOT_ITEM 18446744073709551615) type DIR transid 1601957 data_len 0 name_len 29 name: Sound_daily.20220327_01:02:01 item 41 key (256 DIR_ITEM 1731806220) itemoff 13940 itemsize 56 location key (165298 ROOT_ITEM 18446744073709551615) type DIR transid 1601944 data_len 0 name_len 26 name: Video_ro.20220327_00:35:50 item 42 key (256 DIR_ITEM 1782151770) itemoff 13880 itemsize 60 location key (165391 ROOT_ITEM 18446744073709551615) type DIR transid 1602067 data_len 0 name_len 30 name: other_hourly.20220327_23:01:01 item 43 key (256 DIR_ITEM 1808375180) itemoff 13824 itemsize 56 location key (163301 ROOT_ITEM 18446744073709551615) type DIR transid 1597239 data_len 0 name_len 26 name: Sound_ro.20220306_00:45:46 item 44 key (256 DIR_ITEM 1925286825) itemoff 13768 itemsize 56 location key (164621 ROOT_ITEM 18446744073709551615) type DIR transid 1601060 data_len 0 name_len 26 name: other_ro.20220320_00:25:35 item 45 key (256 DIR_ITEM 2030955372) itemoff 13712 itemsize 56 location key (163919 ROOT_ITEM 18446744073709551615) type DIR transid 1599632 data_len 0 name_len 26 name: Sound_ro.20220313_00:37:25 item 46 key (256 DIR_ITEM 2112197083) itemoff 13652 itemsize 60 location key (165395 ROOT_ITEM 18446744073709551615) type DIR transid 1602073 data_len 0 name_len 30 name: other_hourly.20220328_00:01:01 item 47 key (256 DIR_ITEM 2127973047) itemoff 13617 itemsize 35 location key (11221 ROOT_ITEM 18446744073709551615) type DIR transid 4042 data_len 0 name_len 5 name: other item 48 key (256 DIR_ITEM 2146234148) itemoff 13583 itemsize 34 location key (11223 ROOT_ITEM 18446744073709551615) type DIR transid 4061 data_len 0 name_len 4 name: Soft item 49 key (256 DIR_ITEM 2153154160) itemoff 13523 itemsize 60 location key (165389 ROOT_ITEM 18446744073709551615) type DIR transid 1602065 data_len 0 name_len 30 name: Sound_hourly.20220327_23:01:01 item 50 key (256 DIR_ITEM 2201886496) itemoff 13464 itemsize 59 location key (165101 ROOT_ITEM 18446744073709551615) type DIR transid 1601683 data_len 0 name_len 29 name: other_daily.20220325_01:02:01 item 51 key (256 DIR_ITEM 2279840979) itemoff 13408 itemsize 56 location key (162629 ROOT_ITEM 18446744073709551615) type DIR transid 1596346 data_len 0 name_len 26 name: other_ro.20220227_00:25:38 item 52 key (256 DIR_ITEM 2313060016) itemoff 13348 itemsize 60 location key (160496 ROOT_ITEM 18446744073709551615) type DIR transid 1592684 data_len 0 name_len 30 name: media_weekly.20220206_02:03:01 item 53 key (256 DIR_ITEM 2354277240) itemoff 13308 itemsize 40 location key (1492 INODE_ITEM 0) type SYMLINK transid 1601938 data_len 0 name_len 10 name: other_last item 54 key (256 DIR_ITEM 2355411712) itemoff 13248 itemsize 60 location key (160494 ROOT_ITEM 18446744073709551615) type DIR transid 1592682 data_len 0 name_len 30 name: Video_weekly.20220206_02:03:01 item 55 key (256 DIR_ITEM 2377365299) itemoff 13192 itemsize 56 location key (165297 ROOT_ITEM 18446744073709551615) type DIR transid 1601941 data_len 0 name_len 26 name: Sound_ro.20220327_00:35:22 item 56 key (256 DIR_ITEM 2396976783) itemoff 13132 itemsize 60 location key (163319 ROOT_ITEM 18446744073709551615) type DIR transid 1597266 data_len 0 name_len 30 name: other_weekly.20220306_02:03:01 item 57 key (256 DIR_ITEM 2456187574) itemoff 13076 itemsize 56 location key (163298 ROOT_ITEM 18446744073709551615) type DIR transid 1597234 data_len 0 name_len 26 name: media_ro.20220306_00:24:43 item 58 key (256 DIR_ITEM 2467975514) itemoff 13021 itemsize 55 location key (163300 ROOT_ITEM 18446744073709551615) type DIR transid 1597237 data_len 0 name_len 25 name: Soft_ro.20220306_00:33:55 item 59 key (256 DIR_ITEM 2542431729) itemoff 12961 itemsize 60 location key (165393 ROOT_ITEM 18446744073709551615) type DIR transid 1602071 data_len 0 name_len 30 name: Sound_hourly.20220328_00:01:01 item 60 key (256 DIR_ITEM 2610074692) itemoff 12901 itemsize 60 location key (165406 ROOT_ITEM 18446744073709551615) type DIR transid 1602087 data_len 0 name_len 30 name: media_hourly.20220328_02:01:01 item 61 key (256 DIR_ITEM 2650936973) itemoff 12842 itemsize 59 location key (165306 ROOT_ITEM 18446744073709551615) type DIR transid 1601958 data_len 0 name_len 29 name: media_daily.20220327_01:02:01 item 62 key (256 DIR_ITEM 2653442548) itemoff 12782 itemsize 60 location key (165404 ROOT_ITEM 18446744073709551615) type DIR transid 1602085 data_len 0 name_len 30 name: Video_hourly.20220328_02:01:01 item 63 key (256 DIR_ITEM 2717852207) itemoff 12728 itemsize 54 location key (163921 ROOT_ITEM 18446744073709551615) type DIR transid 1599637 data_len 0 name_len 24 name: Win_ro.20220313_00:38:28 item 64 key (256 DIR_ITEM 2745397786) itemoff 12669 itemsize 59 location key (165402 ROOT_ITEM 18446744073709551615) type DIR transid 1602082 data_len 0 name_len 29 name: media_daily.20220328_01:02:01 item 65 key (256 DIR_ITEM 2782562819) itemoff 12610 itemsize 59 location key (165098 ROOT_ITEM 18446744073709551615) type DIR transid 1601680 data_len 0 name_len 29 name: Video_daily.20220325_01:02:01 item 66 key (256 DIR_ITEM 2793356223) itemoff 12554 itemsize 56 location key (163299 ROOT_ITEM 18446744073709551615) type DIR transid 1597235 data_len 0 name_len 26 name: other_ro.20220306_00:25:40 item 67 key (256 DIR_ITEM 2811536125) itemoff 12519 itemsize 35 location key (11288 ROOT_ITEM 18446744073709551615) type DIR transid 4174 data_len 0 name_len 5 name: Video item 68 key (256 DIR_ITEM 2827986674) itemoff 12485 itemsize 34 location key (1010 INODE_ITEM 0) type FILE transid 1455723 data_len 0 name_len 4 name: copy item 69 key (256 DIR_ITEM 2917386315) itemoff 12425 itemsize 60 location key (165407 ROOT_ITEM 18446744073709551615) type DIR transid 1602088 data_len 0 name_len 30 name: other_hourly.20220328_02:01:01 item 70 key (256 DIR_ITEM 2935166119) itemoff 12366 itemsize 59 location key (165199 ROOT_ITEM 18446744073709551615) type DIR transid 1601810 data_len 0 name_len 29 name: Sound_daily.20220326_01:02:01 item 71 key (256 DIR_ITEM 2935255735) itemoff 12311 itemsize 55 location key (165296 ROOT_ITEM 18446744073709551615) type DIR transid 1601938 data_len 0 name_len 25 name: Soft_ro.20220327_00:32:46 item 72 key (256 DIR_ITEM 3098520192) itemoff 12251 itemsize 60 location key (163318 ROOT_ITEM 18446744073709551615) type DIR transid 1597265 data_len 0 name_len 30 name: media_weekly.20220306_02:03:01 item 73 key (256 DIR_ITEM 3170169652) itemoff 12195 itemsize 56 location key (164624 ROOT_ITEM 18446744073709551615) type DIR transid 1601068 data_len 0 name_len 26 name: Video_ro.20220320_00:40:27 item 74 key (256 DIR_ITEM 3172298544) itemoff 12135 itemsize 60 location key (163316 ROOT_ITEM 18446744073709551615) type DIR transid 1597263 data_len 0 name_len 30 name: Video_weekly.20220306_02:03:01 item 75 key (256 DIR_ITEM 3215925951) itemoff 12075 itemsize 60 location key (160497 ROOT_ITEM 18446744073709551615) type DIR transid 1592685 data_len 0 name_len 30 name: other_weekly.20220206_02:03:01 item 76 key (256 DIR_ITEM 3236289372) itemoff 12015 itemsize 60 location key (161200 ROOT_ITEM 18446744073709551615) type DIR transid 1594246 data_len 0 name_len 30 name: other_weekly.20220213_02:03:01 item 77 key (256 DIR_ITEM 3290748772) itemoff 11959 itemsize 56 location key (163302 ROOT_ITEM 18446744073709551615) type DIR transid 1597242 data_len 0 name_len 26 name: Video_ro.20220306_00:46:14 item 78 key (256 DIR_ITEM 3360445642) itemoff 11903 itemsize 56 location key (162631 ROOT_ITEM 18446744073709551615) type DIR transid 1596351 data_len 0 name_len 26 name: Sound_ro.20220227_00:40:41 item 79 key (256 DIR_ITEM 3470690534) itemoff 11847 itemsize 56 location key (164620 ROOT_ITEM 18446744073709551615) type DIR transid 1601059 data_len 0 name_len 26 name: media_ro.20220320_00:24:47 item 80 key (256 DIR_ITEM 3550830046) itemoff 11787 itemsize 60 location key (159785 ROOT_ITEM 18446744073709551615) type DIR transid 1591389 data_len 0 name_len 30 name: Video_weekly.20220130_02:03:01 item 81 key (256 DIR_ITEM 3592394862) itemoff 11727 itemsize 60 location key (159787 ROOT_ITEM 18446744073709551615) type DIR transid 1591391 data_len 0 name_len 30 name: media_weekly.20220130_02:03:01 item 82 key (256 DIR_ITEM 3676890430) itemoff 11668 itemsize 59 location key (165201 ROOT_ITEM 18446744073709551615) type DIR transid 1601812 data_len 0 name_len 29 name: other_daily.20220326_01:02:01 item 83 key (256 DIR_ITEM 3765355617) itemoff 11608 itemsize 60 location key (159788 ROOT_ITEM 18446744073709551615) type DIR transid 1591392 data_len 0 name_len 30 name: other_weekly.20220130_02:03:01 item 84 key (256 DIR_ITEM 3787964139) itemoff 11554 itemsize 54 location key (163303 ROOT_ITEM 18446744073709551615) type DIR transid 1597244 data_len 0 name_len 24 name: Win_ro.20220306_00:47:32 item 85 key (256 DIR_ITEM 3799902853) itemoff 11514 itemsize 40 location key (1491 INODE_ITEM 0) type SYMLINK transid 1601935 data_len 0 name_len 10 name: media_last item 86 key (256 DIR_ITEM 3872736471) itemoff 11460 itemsize 54 location key (162645 ROOT_ITEM 18446744073709551615) type DIR transid 1596372 data_len 0 name_len 24 name: Win_ro.20220227_02:10:25 item 87 key (256 DIR_ITEM 3951915213) itemoff 11420 itemsize 40 location key (1495 INODE_ITEM 0) type SYMLINK transid 1601947 data_len 0 name_len 10 name: Video_last item 88 key (256 DIR_ITEM 4079895267) itemoff 11360 itemsize 60 location key (161197 ROOT_ITEM 18446744073709551615) type DIR transid 1594243 data_len 0 name_len 30 name: Video_weekly.20220213_02:03:01 item 89 key (256 DIR_ITEM 4084751325) itemoff 11321 itemsize 39 location key (1493 INODE_ITEM 0) type SYMLINK transid 1601941 data_len 0 name_len 9 name: Soft_last item 90 key (256 DIR_ITEM 4136929107) itemoff 11261 itemsize 60 location key (161199 ROOT_ITEM 18446744073709551615) type DIR transid 1594245 data_len 0 name_len 30 name: media_weekly.20220213_02:03:01 item 91 key (256 DIR_ITEM 4142263993) itemoff 11202 itemsize 59 location key (165099 ROOT_ITEM 18446744073709551615) type DIR transid 1601681 data_len 0 name_len 29 name: Sound_daily.20220325_01:02:01 item 92 key (256 DIR_ITEM 4156034002) itemoff 11147 itemsize 55 location key (162630 ROOT_ITEM 18446744073709551615) type DIR transid 1596348 data_len 0 name_len 25 name: Soft_ro.20220227_00:34:48 item 93 key (256 DIR_ITEM 4198095398) itemoff 11091 itemsize 56 location key (165294 ROOT_ITEM 18446744073709551615) type DIR transid 1601934 data_len 0 name_len 26 name: media_ro.20220327_00:24:32 item 94 key (256 DIR_ITEM 4258033693) itemoff 11032 itemsize 59 location key (165198 ROOT_ITEM 18446744073709551615) type DIR transid 1601809 data_len 0 name_len 29 name: Video_daily.20220326_01:02:01 item 95 key (256 DIR_ITEM 4287185209) itemoff 10972 itemsize 60 location key (165397 ROOT_ITEM 18446744073709551615) type DIR transid 1602076 data_len 0 name_len 30 name: Sound_hourly.20220328_01:01:01 item 96 key (256 DIR_INDEX 8) itemoff 10937 itemsize 35 location key (11221 ROOT_ITEM 18446744073709551615) type DIR transid 4042 data_len 0 name_len 5 name: other item 97 key (256 DIR_INDEX 9) itemoff 10902 itemsize 35 location key (11222 ROOT_ITEM 18446744073709551615) type DIR transid 4058 data_len 0 name_len 5 name: media item 98 key (256 DIR_INDEX 10) itemoff 10868 itemsize 34 location key (11223 ROOT_ITEM 18446744073709551615) type DIR transid 4061 data_len 0 name_len 4 name: Soft item 99 key (256 DIR_INDEX 11) itemoff 10835 itemsize 33 location key (11224 ROOT_ITEM 18446744073709551615) type DIR transid 4062 data_len 0 name_len 3 name: Win item 100 key (256 DIR_INDEX 17) itemoff 10800 itemsize 35 location key (11288 ROOT_ITEM 18446744073709551615) type DIR transid 4174 data_len 0 name_len 5 name: Video item 101 key (256 DIR_INDEX 51) itemoff 10765 itemsize 35 location key (11322 ROOT_ITEM 18446744073709551615) type DIR transid 4223 data_len 0 name_len 5 name: Sound 1;37mgargamel:/var/local/src/btrfs-progs-josefbacik#m Kexit Script done on Mon 04 Apr 2022 18:38:35 PDT -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 1:42 ` Marc MERLIN @ 2022-04-05 1:55 ` Josef Bacik 2022-04-05 2:07 ` Marc MERLIN 2022-04-05 16:22 ` Roman Mamedov 2022-04-05 18:38 ` Zygo Blaxell 2 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 1:55 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote: > > Ok lets try again with -b 13577821667328, and if the owner doesn't say > > ROOT_TREE try 13577775284224 and then 13577814573056. Thanks, > > EXTENT_TREE, FS_TREE, and FS_TREE > > And shit, I got distracted and sent the text output to > /dev/mapper/dshelf1a, so I clobbered about 30K of the device. > I'm assuming there was probably something there? Yes the chunk tree, but your FS is DUP so we'll find the other copy, it'll be fine. Do you have any subvolumes that aren't snapshots? I think I can re-create everything without knowing what other roots there are, but if you have snapshots/subvolumes I need to be more careful. In any case I'm going to go to bed, it's been a long week. I'll start working on a tool to re-create your root tree in the morning, and then hopefully we can go from there. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 1:55 ` Josef Bacik @ 2022-04-05 2:07 ` Marc MERLIN 2022-04-05 14:11 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 2:07 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Mon, Apr 04, 2022 at 09:55:41PM -0400, Josef Bacik wrote: > On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote: > > > Ok lets try again with -b 13577821667328, and if the owner doesn't say > > > ROOT_TREE try 13577775284224 and then 13577814573056. Thanks, > > > > EXTENT_TREE, FS_TREE, and FS_TREE > > > > And shit, I got distracted and sent the text output to > > /dev/mapper/dshelf1a, so I clobbered about 30K of the device. > > I'm assuming there was probably something there? > > Yes the chunk tree, but your FS is DUP so we'll find the other copy, > it'll be fine. Great news, thanks. > Do you have any subvolumes that aren't snapshots? I think I can > re-create everything without knowing what other roots there are, but > if you have snapshots/subvolumes I need to be more careful. 6 subvolumes plus snwpshots of them: media other Soft Sound Video Win Maybe 50 to 100 snapshots total (mostly snapshots) > In any case I'm going to go to bed, it's been a long week. I'll start > working on a tool to re-create your root tree in the morning, and then > hopefully we can go from there. Thanks, Thanks for your help, have a good night. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 2:07 ` Marc MERLIN @ 2022-04-05 14:11 ` Josef Bacik 2022-04-05 15:53 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 14:11 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Mon, Apr 4, 2022 at 10:07 PM Marc MERLIN <marc@merlins.org> wrote: > > On Mon, Apr 04, 2022 at 09:55:41PM -0400, Josef Bacik wrote: > > On Mon, Apr 4, 2022 at 9:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Mon, Apr 04, 2022 at 09:22:26PM -0400, Josef Bacik wrote: > > > > Ok lets try again with -b 13577821667328, and if the owner doesn't say > > > > ROOT_TREE try 13577775284224 and then 13577814573056. Thanks, > > > > > > EXTENT_TREE, FS_TREE, and FS_TREE > > > > > > And shit, I got distracted and sent the text output to > > > /dev/mapper/dshelf1a, so I clobbered about 30K of the device. > > > I'm assuming there was probably something there? > > > > Yes the chunk tree, but your FS is DUP so we'll find the other copy, > > it'll be fine. > > Great news, thanks. > > > Do you have any subvolumes that aren't snapshots? I think I can > > re-create everything without knowing what other roots there are, but > > if you have snapshots/subvolumes I need to be more careful. > > 6 subvolumes plus snwpshots of them: > media other Soft Sound Video Win > > Maybe 50 to 100 snapshots total (mostly snapshots) > > > In any case I'm going to go to bed, it's been a long week. I'll start > > working on a tool to re-create your root tree in the morning, and then > > hopefully we can go from there. Thanks, > > Thanks for your help, have a good night. > Ok the fs volumes are going to be trickier to put back together, so first lets make sure the fs tree pointer is correct and relatively intact. Can you do ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever I don't need the full dump since it's all the metadata for your fs tree. What I'm looking for is if this looks sane, you'll see the normal errors you see from the fs being generally fucked, but I want this to spew and seem like it worked, and all have owner of FS_ROOT. The problem here is that since the tree root is fucked I can't easily find the subvolumes bytenr, so we're going to have to walk the FS_ROOT and find the ID's for all your subvolumes, and then use btrfs-find-root to find the bytenrs of those roots. From there I can clear your old tree root and create it with just the subvolumes, chunk root, and device root. From there fsck can re-create the extent and csum tree. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 14:11 ` Josef Bacik @ 2022-04-05 15:53 ` Marc MERLIN 2022-04-05 15:55 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 15:53 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote: > Ok the fs volumes are going to be trickier to put back together, so > first lets make sure the fs tree pointer is correct and relatively > intact. Can you do > > ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever > > I don't need the full dump since it's all the metadata for your fs > tree. What I'm looking for is if this looks sane, you'll see the > normal errors you see from the fs being generally fucked, but I want > this to spew and seem like it worked, and all have owner of FS_ROOT. Thanks. It's not too big, attached gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x55741d8f7470 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55741d8f7470 btrfs-progs v5.16.2 leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE leaf 13577799401472 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (1901621264384 METADATA_ITEM 0) itemoff 16250 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 1 key (1901621280768 METADATA_ITEM 0) itemoff 16217 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 2 key (1901621297152 METADATA_ITEM 0) itemoff 16184 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 3 key (1901621313536 METADATA_ITEM 0) itemoff 16151 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 4 key (1901621329920 METADATA_ITEM 0) itemoff 16118 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 5 key (1901621346304 METADATA_ITEM 0) itemoff 16085 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 6 key (1901621362688 METADATA_ITEM 0) itemoff 16052 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 7 key (1901621379072 METADATA_ITEM 0) itemoff 16019 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 8 key (1901621395456 METADATA_ITEM 0) itemoff 15986 itemsize 33 refs 1 gen 2327 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 9 key (1901621411840 METADATA_ITEM 0) itemoff 15953 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 10 key (1901621428224 METADATA_ITEM 0) itemoff 15920 itemsize 33 refs 1 gen 2327 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 11 key (1901621444608 METADATA_ITEM 0) itemoff 15887 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 12 key (1901621460992 METADATA_ITEM 0) itemoff 15854 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 13 key (1901621477376 METADATA_ITEM 0) itemoff 15821 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 14 key (1901621493760 METADATA_ITEM 0) itemoff 15788 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 15 key (1901621510144 METADATA_ITEM 0) itemoff 15755 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 16 key (1901621526528 METADATA_ITEM 0) itemoff 15722 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 17 key (1901621542912 METADATA_ITEM 0) itemoff 15689 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 18 key (1901621559296 METADATA_ITEM 0) itemoff 15656 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 19 key (1901621575680 METADATA_ITEM 0) itemoff 15623 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 20 key (1901621592064 METADATA_ITEM 0) itemoff 15590 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 21 key (1901621608448 METADATA_ITEM 0) itemoff 15557 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 22 key (1901621624832 METADATA_ITEM 0) itemoff 15524 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 23 key (1901621641216 METADATA_ITEM 0) itemoff 15491 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 24 key (1901621657600 METADATA_ITEM 0) itemoff 15458 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 25 key (1901621673984 METADATA_ITEM 0) itemoff 15425 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 26 key (1901621690368 METADATA_ITEM 0) itemoff 15392 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 27 key (1901621706752 METADATA_ITEM 0) itemoff 15359 itemsize 33 refs 1 gen 2955 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 28 key (1901621723136 METADATA_ITEM 0) itemoff 15326 itemsize 33 refs 1 gen 2955 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 29 key (1901621755904 METADATA_ITEM 0) itemoff 15293 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 30 key (1901621772288 METADATA_ITEM 0) itemoff 15260 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 31 key (1901621788672 METADATA_ITEM 0) itemoff 15227 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 32 key (1901621805056 METADATA_ITEM 0) itemoff 15194 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 33 key (1901621821440 METADATA_ITEM 0) itemoff 15161 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 34 key (1901621837824 METADATA_ITEM 0) itemoff 15128 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 35 key (1901621854208 METADATA_ITEM 0) itemoff 15095 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 36 key (1901621870592 METADATA_ITEM 0) itemoff 15062 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 37 key (1901621886976 METADATA_ITEM 0) itemoff 15029 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 38 key (1901621903360 METADATA_ITEM 0) itemoff 14996 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 39 key (1901621919744 METADATA_ITEM 0) itemoff 14963 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 40 key (1901621936128 METADATA_ITEM 0) itemoff 14930 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 41 key (1901621952512 METADATA_ITEM 0) itemoff 14897 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 42 key (1901621968896 METADATA_ITEM 0) itemoff 14864 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 43 key (1901621985280 METADATA_ITEM 0) itemoff 14831 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 44 key (1901622001664 METADATA_ITEM 0) itemoff 14798 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 45 key (1901622018048 METADATA_ITEM 0) itemoff 14765 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 46 key (1901622034432 METADATA_ITEM 0) itemoff 14732 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 47 key (1901622050816 METADATA_ITEM 0) itemoff 14699 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 48 key (1901622067200 METADATA_ITEM 0) itemoff 14666 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 49 key (1901622083584 METADATA_ITEM 0) itemoff 14633 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 50 key (1901622099968 METADATA_ITEM 0) itemoff 14600 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 51 key (1901622116352 METADATA_ITEM 0) itemoff 14567 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 52 key (1901622132736 METADATA_ITEM 0) itemoff 14534 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 53 key (1901622149120 METADATA_ITEM 0) itemoff 14501 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 54 key (1901622165504 METADATA_ITEM 0) itemoff 14468 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 55 key (1901622181888 METADATA_ITEM 0) itemoff 14435 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 56 key (1901622198272 METADATA_ITEM 0) itemoff 14402 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 57 key (1901622214656 METADATA_ITEM 0) itemoff 14369 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 58 key (1901622231040 METADATA_ITEM 0) itemoff 14336 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 59 key (1901622247424 METADATA_ITEM 0) itemoff 14303 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 60 key (1901622263808 METADATA_ITEM 0) itemoff 14270 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 61 key (1901622280192 METADATA_ITEM 0) itemoff 14237 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 62 key (1901622296576 METADATA_ITEM 0) itemoff 14204 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 63 key (1901622312960 METADATA_ITEM 0) itemoff 14171 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 64 key (1901622329344 METADATA_ITEM 0) itemoff 14138 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 65 key (1901622345728 METADATA_ITEM 0) itemoff 14105 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 66 key (1901622362112 METADATA_ITEM 0) itemoff 14072 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 67 key (1901622378496 METADATA_ITEM 0) itemoff 14039 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 68 key (1901622394880 METADATA_ITEM 0) itemoff 14006 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 69 key (1901622411264 METADATA_ITEM 0) itemoff 13973 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 70 key (1901622427648 METADATA_ITEM 0) itemoff 13940 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 71 key (1901622444032 METADATA_ITEM 0) itemoff 13907 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 72 key (1901622460416 METADATA_ITEM 0) itemoff 13874 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 73 key (1901622476800 METADATA_ITEM 0) itemoff 13841 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 74 key (1901622493184 METADATA_ITEM 0) itemoff 13808 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 75 key (1901622509568 METADATA_ITEM 0) itemoff 13775 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 76 key (1901622542336 METADATA_ITEM 0) itemoff 13742 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 77 key (1901622558720 METADATA_ITEM 0) itemoff 13709 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 78 key (1901622575104 METADATA_ITEM 0) itemoff 13676 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 79 key (1901622591488 METADATA_ITEM 0) itemoff 13643 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 80 key (1901622607872 METADATA_ITEM 0) itemoff 13610 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 81 key (1901622624256 METADATA_ITEM 0) itemoff 13577 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 82 key (1901622640640 METADATA_ITEM 0) itemoff 13544 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 83 key (1901622657024 METADATA_ITEM 0) itemoff 13511 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 84 key (1901622673408 METADATA_ITEM 0) itemoff 13478 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 85 key (1901622689792 METADATA_ITEM 0) itemoff 13445 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 86 key (1901622706176 METADATA_ITEM 0) itemoff 13412 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 87 key (1901622722560 METADATA_ITEM 0) itemoff 13379 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 88 key (1901622738944 METADATA_ITEM 0) itemoff 13346 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 89 key (1901622755328 METADATA_ITEM 0) itemoff 13313 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 90 key (1901622771712 METADATA_ITEM 0) itemoff 13280 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 91 key (1901622788096 METADATA_ITEM 0) itemoff 13247 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 92 key (1901622804480 METADATA_ITEM 0) itemoff 13214 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 93 key (1901622820864 METADATA_ITEM 0) itemoff 13181 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 94 key (1901622837248 METADATA_ITEM 0) itemoff 13148 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 95 key (1901622853632 METADATA_ITEM 0) itemoff 13115 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 96 key (1901622870016 METADATA_ITEM 0) itemoff 13082 itemsize 33 refs 1 gen 2327 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 97 key (1901622886400 METADATA_ITEM 0) itemoff 13049 itemsize 33 refs 1 gen 2327 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 98 key (1901622902784 METADATA_ITEM 0) itemoff 13016 itemsize 33 refs 1 gen 2327 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 99 key (1901622919168 METADATA_ITEM 0) itemoff 12983 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 100 key (1901622935552 METADATA_ITEM 0) itemoff 12950 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 101 key (1901622951936 METADATA_ITEM 0) itemoff 12917 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 102 key (1901622968320 METADATA_ITEM 0) itemoff 12884 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 103 key (1901622984704 METADATA_ITEM 0) itemoff 12851 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 104 key (1901623001088 METADATA_ITEM 0) itemoff 12818 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 105 key (1901623017472 METADATA_ITEM 0) itemoff 12785 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 106 key (1901623033856 METADATA_ITEM 0) itemoff 12752 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 107 key (1901623050240 METADATA_ITEM 0) itemoff 12719 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 108 key (1901623066624 METADATA_ITEM 0) itemoff 12686 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 109 key (1901623083008 METADATA_ITEM 0) itemoff 12653 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 110 key (1901623099392 METADATA_ITEM 0) itemoff 12620 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 111 key (1901623115776 METADATA_ITEM 0) itemoff 12587 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 112 key (1901623132160 METADATA_ITEM 0) itemoff 12554 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 113 key (1901623148544 METADATA_ITEM 0) itemoff 12521 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 114 key (1901623164928 METADATA_ITEM 0) itemoff 12488 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 115 key (1901623181312 METADATA_ITEM 0) itemoff 12455 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 116 key (1901623197696 METADATA_ITEM 0) itemoff 12422 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 117 key (1901623214080 METADATA_ITEM 0) itemoff 12389 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 118 key (1901623230464 METADATA_ITEM 0) itemoff 12356 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 119 key (1901623246848 METADATA_ITEM 0) itemoff 12323 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 120 key (1901623263232 METADATA_ITEM 0) itemoff 12290 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 121 key (1901623279616 METADATA_ITEM 0) itemoff 12257 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 122 key (1901623296000 METADATA_ITEM 0) itemoff 12224 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 123 key (1901623312384 METADATA_ITEM 0) itemoff 12191 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 124 key (1901623328768 METADATA_ITEM 0) itemoff 12158 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 125 key (1901623345152 METADATA_ITEM 0) itemoff 12125 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 126 key (1901623361536 METADATA_ITEM 0) itemoff 12092 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 127 key (1901623377920 METADATA_ITEM 0) itemoff 12059 itemsize 33 refs 1 gen 1526014 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 128 key (1901623394304 METADATA_ITEM 0) itemoff 12026 itemsize 33 refs 1 gen 1602167 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 129 key (1901623410688 METADATA_ITEM 0) itemoff 11993 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 130 key (1901623427072 METADATA_ITEM 0) itemoff 11960 itemsize 33 refs 1 gen 1482589 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 131 key (1901623476224 METADATA_ITEM 0) itemoff 11927 itemsize 33 refs 1 gen 1526014 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 132 key (1901623492608 METADATA_ITEM 0) itemoff 11894 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 133 key (1901623508992 METADATA_ITEM 0) itemoff 11861 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 134 key (1901623525376 METADATA_ITEM 0) itemoff 11828 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 135 key (1901623541760 METADATA_ITEM 0) itemoff 11795 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 136 key (1901623558144 METADATA_ITEM 0) itemoff 11762 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 137 key (1901623574528 METADATA_ITEM 0) itemoff 11729 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 138 key (1901623590912 METADATA_ITEM 0) itemoff 11696 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 139 key (1901623607296 METADATA_ITEM 0) itemoff 11663 itemsize 33 refs 1 gen 1482589 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 140 key (1901623623680 METADATA_ITEM 0) itemoff 11630 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 141 key (1901623640064 METADATA_ITEM 0) itemoff 11597 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 142 key (1901623656448 METADATA_ITEM 0) itemoff 11564 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 143 key (1901623672832 METADATA_ITEM 0) itemoff 11531 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 144 key (1901623689216 METADATA_ITEM 0) itemoff 11498 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 145 key (1901623705600 METADATA_ITEM 0) itemoff 11465 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 146 key (1901623721984 METADATA_ITEM 0) itemoff 11432 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 147 key (1901623738368 METADATA_ITEM 0) itemoff 11399 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 148 key (1901623754752 METADATA_ITEM 0) itemoff 11366 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 149 key (1901623771136 METADATA_ITEM 0) itemoff 11333 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 150 key (1901623787520 METADATA_ITEM 0) itemoff 11300 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 151 key (1901623803904 METADATA_ITEM 0) itemoff 11267 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 152 key (1901623820288 METADATA_ITEM 0) itemoff 11234 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 153 key (1901623836672 METADATA_ITEM 0) itemoff 11201 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 154 key (1901623853056 METADATA_ITEM 0) itemoff 11168 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 155 key (1901623869440 METADATA_ITEM 0) itemoff 11135 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 156 key (1901623885824 METADATA_ITEM 0) itemoff 11102 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 157 key (1901623902208 METADATA_ITEM 0) itemoff 11069 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 158 key (1901623918592 METADATA_ITEM 0) itemoff 11036 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 159 key (1901623934976 METADATA_ITEM 0) itemoff 11003 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 160 key (1901623951360 METADATA_ITEM 0) itemoff 10970 itemsize 33 refs 1 gen 1125 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 161 key (1901623967744 METADATA_ITEM 0) itemoff 10937 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 162 key (1901623984128 METADATA_ITEM 0) itemoff 10904 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 163 key (1901624000512 METADATA_ITEM 0) itemoff 10871 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 164 key (1901624016896 METADATA_ITEM 0) itemoff 10838 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 165 key (1901624033280 METADATA_ITEM 0) itemoff 10805 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 166 key (1901624049664 METADATA_ITEM 0) itemoff 10772 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 167 key (1901624066048 METADATA_ITEM 0) itemoff 10739 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 168 key (1901624082432 METADATA_ITEM 0) itemoff 10706 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 169 key (1901624098816 METADATA_ITEM 0) itemoff 10673 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 170 key (1901624115200 METADATA_ITEM 0) itemoff 10640 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 171 key (1901624131584 METADATA_ITEM 0) itemoff 10607 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 172 key (1901624147968 METADATA_ITEM 0) itemoff 10574 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 173 key (1901624164352 METADATA_ITEM 0) itemoff 10541 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 174 key (1901624180736 METADATA_ITEM 0) itemoff 10508 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 175 key (1901624197120 METADATA_ITEM 0) itemoff 10475 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 176 key (1901624213504 METADATA_ITEM 0) itemoff 10442 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 177 key (1901624229888 METADATA_ITEM 0) itemoff 10409 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 178 key (1901624246272 METADATA_ITEM 0) itemoff 10376 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 179 key (1901624262656 METADATA_ITEM 0) itemoff 10343 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 180 key (1901624279040 METADATA_ITEM 0) itemoff 10310 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 181 key (1901624295424 METADATA_ITEM 0) itemoff 10277 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 182 key (1901624311808 METADATA_ITEM 0) itemoff 10244 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 183 key (1901624328192 METADATA_ITEM 0) itemoff 10211 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 184 key (1901624344576 METADATA_ITEM 0) itemoff 10178 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 185 key (1901624360960 METADATA_ITEM 0) itemoff 10145 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 186 key (1901624377344 METADATA_ITEM 0) itemoff 10112 itemsize 33 refs 1 gen 2384 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 187 key (1901624393728 METADATA_ITEM 0) itemoff 10079 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 188 key (1901624410112 METADATA_ITEM 0) itemoff 10046 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 189 key (1901624426496 METADATA_ITEM 0) itemoff 10013 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 190 key (1901624442880 METADATA_ITEM 0) itemoff 9980 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 191 key (1901624459264 METADATA_ITEM 0) itemoff 9947 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 192 key (1901624475648 METADATA_ITEM 0) itemoff 9914 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 193 key (1901624492032 METADATA_ITEM 0) itemoff 9881 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 194 key (1901624508416 METADATA_ITEM 0) itemoff 9848 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 195 key (1901624524800 METADATA_ITEM 0) itemoff 9815 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 196 key (1901624541184 METADATA_ITEM 0) itemoff 9782 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 197 key (1901624573952 METADATA_ITEM 0) itemoff 9749 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 198 key (1901624590336 METADATA_ITEM 0) itemoff 9716 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 199 key (1901624606720 METADATA_ITEM 0) itemoff 9683 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 200 key (1901624623104 METADATA_ITEM 0) itemoff 9650 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 201 key (1901624639488 METADATA_ITEM 0) itemoff 9617 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 202 key (1901624655872 METADATA_ITEM 0) itemoff 9584 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 203 key (1901624672256 METADATA_ITEM 0) itemoff 9551 itemsize 33 refs 1 gen 3116 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 204 key (1901624688640 METADATA_ITEM 0) itemoff 9518 itemsize 33 refs 1 gen 900861 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 205 key (1901624737792 METADATA_ITEM 0) itemoff 9485 itemsize 33 refs 1 gen 1602167 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 206 key (1901624770560 METADATA_ITEM 0) itemoff 9452 itemsize 33 refs 1 gen 1482589 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 207 key (1901624786944 METADATA_ITEM 0) itemoff 9419 itemsize 33 refs 1 gen 1482589 flags TREE_BLOCK tree block skinny level 0 shared block backref parent 15645446602752 item 208 key (1901624819712 METADATA_ITEM 0) itemoff 9386 itemsize 33 refs 1 gen 482803 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 209 key (1901624868864 METADATA_ITEM 0) itemoff 9353 itemsize 33 refs 1 gen 1526014 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 210 key (1901624885248 METADATA_ITEM 0) itemoff 9320 itemsize 33 refs 1 gen 1601407 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 211 key (1901624934400 METADATA_ITEM 0) itemoff 9287 itemsize 33 refs 1 gen 1601407 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 212 key (1901624950784 METADATA_ITEM 0) itemoff 9254 itemsize 33 refs 1 gen 1601407 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 213 key (1901624967168 METADATA_ITEM 0) itemoff 9221 itemsize 33 refs 1 gen 12204 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 214 key (1901624983552 METADATA_ITEM 0) itemoff 9188 itemsize 33 refs 1 gen 1482589 flags TREE_BLOCK tree block skinny level 0 shared block backref parent 15645446602752 item 215 key (1901625032704 METADATA_ITEM 0) itemoff 9155 itemsize 33 refs 1 gen 300256 flags TREE_BLOCK tree block skinny level 0 tree block backref root EXTENT_TREE item 216 key (1901625065472 METADATA_ITEM 0) itemoff 9122 itemsize 33 refs 1 gen 1602167 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 217 key (1901625081856 METADATA_ITEM 0) itemoff 9089 itemsize 33 refs 1 gen 1509218 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 218 key (1901625098240 METADATA_ITEM 0) itemoff 9056 itemsize 33 refs 1 gen 4427 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 219 key (1901625114624 METADATA_ITEM 0) itemoff 9023 itemsize 33 refs 1 gen 4427 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 220 key (1901625131008 METADATA_ITEM 0) itemoff 8990 itemsize 33 refs 1 gen 4427 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 221 key (1901625147392 METADATA_ITEM 0) itemoff 8957 itemsize 33 refs 1 gen 4427 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 222 key (1901625163776 METADATA_ITEM 0) itemoff 8924 itemsize 33 refs 1 gen 1602167 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 223 key (1901625180160 METADATA_ITEM 0) itemoff 8891 itemsize 33 refs 1 gen 1602167 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 224 key (1901625196544 METADATA_ITEM 0) itemoff 8858 itemsize 33 refs 1 gen 3682 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 225 key (1901625212928 METADATA_ITEM 0) itemoff 8825 itemsize 33 refs 1 gen 3682 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 226 key (1901625229312 METADATA_ITEM 0) itemoff 8792 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 227 key (1901625245696 METADATA_ITEM 0) itemoff 8759 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 228 key (1901625262080 METADATA_ITEM 0) itemoff 8726 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 229 key (1901625278464 METADATA_ITEM 0) itemoff 8693 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 230 key (1901625294848 METADATA_ITEM 0) itemoff 8660 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 231 key (1901625311232 METADATA_ITEM 0) itemoff 8627 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 232 key (1901625327616 METADATA_ITEM 0) itemoff 8594 itemsize 33 refs 1 gen 2529 flags TREE_BLOCK tree block skinny level 0 tree block backref root CSUM_TREE item 233 key (1901625344000 METADATA_ITEM 0) itemoff 8462 itemsize 132 refs 12 gen 1526014 flags TREE_BLOCK tree block skinny level 0 tree block backref root 11221 shared block backref parent 15646057824256 shared block backref parent 15645373939712 shared block backref parent 15645039509504 shared block backref parent 13577558376448 shared block backref parent 12512208764928 shared block backref parent 11971124183040 shared block backref parent 11822367277056 shared block backref parent 11822283620352 shared block backref parent 11651814146048 shared block backref parent 4866924920832 shared block backref parent 782827061248 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 15:53 ` Marc MERLIN @ 2022-04-05 15:55 ` Josef Bacik 2022-04-05 17:41 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 15:55 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 11:53 AM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote: > > Ok the fs volumes are going to be trickier to put back together, so > > first lets make sure the fs tree pointer is correct and relatively > > intact. Can you do > > > > ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever > > > > I don't need the full dump since it's all the metadata for your fs > > tree. What I'm looking for is if this looks sane, you'll see the > > normal errors you see from the fs being generally fucked, but I want > > this to spew and seem like it worked, and all have owner of FS_ROOT. > > Thanks. It's not too big, attached > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Ignoring transid failure > FS_INFO IS 0x55741d8f7470 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Ignoring transid failure > Couldn't find the last root for 4 > Couldn't setup device tree > FS_INFO AFTER IS 0x55741d8f7470 > btrfs-progs v5.16.2 > leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE Ok this is what I was afraid of, but I anticipated this wrinkle and I'm writing up something special to try and put everything back together. I'll have something wonky to try in a couple of hours. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 15:55 ` Josef Bacik @ 2022-04-05 17:41 ` Josef Bacik 2022-04-05 18:11 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 17:41 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 11:55 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On Tue, Apr 5, 2022 at 11:53 AM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 10:11:25AM -0400, Josef Bacik wrote: > > > Ok the fs volumes are going to be trickier to put back together, so > > > first lets make sure the fs tree pointer is correct and relatively > > > intact. Can you do > > > > > > ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/whatever > > > > > > I don't need the full dump since it's all the metadata for your fs > > > tree. What I'm looking for is if this looks sane, you'll see the > > > normal errors you see from the fs being generally fucked, but I want > > > this to spew and seem like it worked, and all have owner of FS_ROOT. > > > > Thanks. It's not too big, attached > > > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 13577799401472 /dev/mapper/dshelf1a &>/tmp/d > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > Ignoring transid failure > > FS_INFO IS 0x55741d8f7470 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > Ignoring transid failure > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > Ignoring transid failure > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > Ignoring transid failure > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > Ignoring transid failure > > Couldn't find the last root for 4 > > Couldn't setup device tree > > FS_INFO AFTER IS 0x55741d8f7470 > > btrfs-progs v5.16.2 > > leaf 13577799401472 items 234 free space 2612 generation 1602242 owner EXTENT_TREE > > Ok this is what I was afraid of, but I anticipated this wrinkle and > I'm writing up something special to try and put everything back > together. I'll have something wonky to try in a couple of hours. > Thanks, > Ok I think I'm going to do a humpty-dumpty thing where I scan for the best possible copies of blocks and stitch them into the tree root. I've wired something up to find out how many bad slots we're working with so I've got an idea of how careful I need to be, can you pull your branch again, rebuild, and run ./btrfs-find-root -o 1 /dev/whatever and send me the output? I'm working on the tool right now, hoping it takes me a couple of hours at most (mostly because I have meetings sporadically throughout today). Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 17:41 ` Josef Bacik @ 2022-04-05 18:11 ` Marc MERLIN 2022-04-05 18:36 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 18:11 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 01:41:10PM -0400, Josef Bacik wrote: > Ok I think I'm going to do a humpty-dumpty thing where I scan for the > best possible copies of blocks and stitch them into the tree root. > I've wired something up to find out how many bad slots we're working > with so I've got an idea of how careful I need to be, can you pull > your branch again, rebuild, and run > > ./btrfs-find-root -o 1 /dev/whatever > > and send me the output? I'm working on the tool right now, hoping it > takes me a couple of hours at most (mostly because I have meetings > sporadically throughout today). Thanks, Cool. there you go gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a Couldn't read chunk tree WTF??? ERROR: open ctree failed Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 18:11 ` Marc MERLIN @ 2022-04-05 18:36 ` Josef Bacik 2022-04-05 19:51 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 18:36 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 2:11 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 01:41:10PM -0400, Josef Bacik wrote: > > Ok I think I'm going to do a humpty-dumpty thing where I scan for the > > best possible copies of blocks and stitch them into the tree root. > > I've wired something up to find out how many bad slots we're working > > with so I've got an idea of how careful I need to be, can you pull > > your branch again, rebuild, and run > > > > ./btrfs-find-root -o 1 /dev/whatever > > > > and send me the output? I'm working on the tool right now, hoping it > > takes me a couple of hours at most (mostly because I have meetings > > sporadically throughout today). Thanks, > > Cool. there you go > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 > /dev/mapper/dshelf1a > Couldn't read chunk tree > WTF??? > ERROR: open ctree failed > That's new, the chunk tree wasn't failing before right? Anyway I pushed a change, it should work now, thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 18:36 ` Josef Bacik @ 2022-04-05 19:51 ` Marc MERLIN 2022-04-05 19:56 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 19:51 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote: > > Couldn't read chunk tree > > WTF??? > > ERROR: open ctree failed > > That's new, the chunk tree wasn't failing before right? Anyway I > pushed a change, it should work now, thanks, It failed for one commands we did before: ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a btrfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Couldn't read chunk tree trfs-progs v5.16.2 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree ERROR: unable to open /dev/mapper/dshelf1a -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 19:51 ` Marc MERLIN @ 2022-04-05 19:56 ` Josef Bacik 2022-04-05 19:59 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 19:56 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote: > > > Couldn't read chunk tree > > > WTF??? > > > ERROR: open ctree failed > > > > That's new, the chunk tree wasn't failing before right? Anyway I > > pushed a change, it should work now, thanks, > > It failed for one commands we did before: > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > btrfs-progs v5.16.2 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > Couldn't read chunk tree > Ok I think this is from you redirecting into your device, can you do btrfs inspect-internal dump-super -s 0 btrfs inspect-internal dump-super -s 1 and see if they're different? We may have to put your old super back. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 19:56 ` Josef Bacik @ 2022-04-05 19:59 ` Marc MERLIN 2022-04-05 20:05 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 19:59 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 03:56:31PM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote: > > > > Couldn't read chunk tree > > > > WTF??? > > > > ERROR: open ctree failed > > > > > > That's new, the chunk tree wasn't failing before right? Anyway I > > > pushed a change, it should work now, thanks, > > > > It failed for one commands we did before: > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > > btrfs-progs v5.16.2 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > Couldn't read chunk tree > > > > Ok I think this is from you redirecting into your device, can you do Sorry for being an idiot. > btrfs inspect-internal dump-super -s 0 > btrfs inspect-internal dump-super -s 1 > > and see if they're different? We may have to put your old super back. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-super -s 0 /dev/mapper/dshelf1a > /tmp/0 gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-super -s 1 /dev/mapper/dshelf1a > /tmp/1 gargamel:/var/local/src/btrfs-progs-josefbacik# diff -u /tmp/[01] --- /tmp/0 2022-04-05 12:58:11.757530345 -0700 +++ /tmp/1 2022-04-05 12:58:14.769452304 -0700 @@ -1,27 +1,27 @@ -superblock: bytenr=65536, device=/dev/mapper/dshelf1a +superblock: bytenr=67108864, device=/dev/mapper/dshelf1a --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 -csum 0xd3d00183 [match] -bytenr 65536 +csum 0x70d015f7 [match] +bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 96539b8c-ccc9-47bf-9e6c-29305890941e metadata_uuid 96539b8c-ccc9-47bf-9e6c-29305890941e label dshelf1 -generation 1602089 -root 13577814573056 +generation 1602298 +root 15645253091328 sys_array_size 129 -chunk_root_generation 1600938 +chunk_root_generation 1602203 root_level 1 -chunk_root 21069824 +chunk_root 21479424 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 24004156973056 -bytes_used 15113376952320 +bytes_used 15144801386496 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 @@ -36,13 +36,13 @@ BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA ) -cache_generation 1602089 -uuid_tree_generation 1602089 +cache_generation 1602298 +uuid_tree_generation 1602298 dev_item.uuid 8d4b0f25-0de9-47a6-a993-bdd301287f30 dev_item.fsid 96539b8c-ccc9-47bf-9e6c-29305890941e [match] dev_item.type 0 dev_item.total_bytes 24004156973056 -dev_item.bytes_used 15178439589888 +dev_item.bytes_used 15184882040832 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 19:59 ` Marc MERLIN @ 2022-04-05 20:05 ` Josef Bacik 2022-04-05 20:08 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 20:05 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 3:59 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 03:56:31PM -0400, Josef Bacik wrote: > > On Tue, Apr 5, 2022 at 3:51 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Tue, Apr 05, 2022 at 02:36:29PM -0400, Josef Bacik wrote: > > > > > Couldn't read chunk tree > > > > > WTF??? > > > > > ERROR: open ctree failed > > > > > > > > That's new, the chunk tree wasn't failing before right? Anyway I > > > > pushed a change, it should work now, thanks, > > > > > > It failed for one commands we did before: > > > ./btrfs inspect-internal dump-tree -t ROOT /dev/mapper/dshelf1a > > > btrfs-progs v5.16.2 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > Couldn't read chunk tree > > > > > > > Ok I think this is from you redirecting into your device, can you do > > Sorry for being an idiot. > > > btrfs inspect-internal dump-super -s 0 > > btrfs inspect-internal dump-super -s 1 > > > > and see if they're different? We may have to put your old super back. Thanks, > Well it's still the same, and this thing is 20mib into your fs so IDK how it would be screwing up now. Can you do ./btrfs inspect-internal dump-tree -b 21069824 and see what that spits out? IDK why it would suddenly start complaining about your chunk root. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 20:05 ` Josef Bacik @ 2022-04-05 20:08 ` Marc MERLIN 2022-04-05 20:24 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 20:08 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote: > Well it's still the same, and this thing is 20mib into your fs so IDK > how it would be screwing up now. Can you do > > ./btrfs inspect-internal dump-tree -b 21069824 > > and see what that spits out? IDK why it would suddenly start > complaining about your chunk root. Thanks, Thanks for your patience and sticking with me gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 Ignoring transid failure FS_INFO IS 0x557c43fde470 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Ignoring transid failure Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x557c43fde470 node 21069824 level 1 items 95 free space 398 generation 1600938 owner CHUNK_TREE node 21069824 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a key (DEV_ITEMS DEV_ITEM 1) block 21135360 gen 1600938 key (FIRST_CHUNK_TREE CHUNK_ITEM 159480020992) block 21086208 gen 278 key (FIRST_CHUNK_TREE CHUNK_ITEM 325910003712) block 21102592 gen 357 key (FIRST_CHUNK_TREE CHUNK_ITEM 490729373696) block 21118976 gen 450 key (FIRST_CHUNK_TREE CHUNK_ITEM 657159356416) block 21184512 gen 593 key (FIRST_CHUNK_TREE CHUNK_ITEM 821978726400) block 21938176 gen 1453956 key (FIRST_CHUNK_TREE CHUNK_ITEM 986798096384) block 21151744 gen 696 key (FIRST_CHUNK_TREE CHUNK_ITEM 1153228079104) block 21020672 gen 773 key (FIRST_CHUNK_TREE CHUNK_ITEM 1319658061824) block 21250048 gen 846 key (FIRST_CHUNK_TREE CHUNK_ITEM 1484477431808) block 21266432 gen 921 key (FIRST_CHUNK_TREE CHUNK_ITEM 1650907414528) block 21282816 gen 1027 key (FIRST_CHUNK_TREE CHUNK_ITEM 1817337397248) block 21168128 gen 1106 key (FIRST_CHUNK_TREE CHUNK_ITEM 1982156767232) block 21037056 gen 1170 key (FIRST_CHUNK_TREE CHUNK_ITEM 2148586749952) block 21233664 gen 1231 key (FIRST_CHUNK_TREE CHUNK_ITEM 2315016732672) block 20971520 gen 1293 key (FIRST_CHUNK_TREE CHUNK_ITEM 2479836102656) block 21315584 gen 1346 key (FIRST_CHUNK_TREE CHUNK_ITEM 2646266085376) block 21381120 gen 1394 key (FIRST_CHUNK_TREE CHUNK_ITEM 2812696068096) block 20987904 gen 1443 key (FIRST_CHUNK_TREE CHUNK_ITEM 2977515438080) block 21348352 gen 1502 key (FIRST_CHUNK_TREE CHUNK_ITEM 3143945420800) block 21364736 gen 1557 key (FIRST_CHUNK_TREE CHUNK_ITEM 3310375403520) block 21397504 gen 1608 key (FIRST_CHUNK_TREE CHUNK_ITEM 3475194773504) block 21463040 gen 1649 key (FIRST_CHUNK_TREE CHUNK_ITEM 3641624756224) block 21004288 gen 1692 key (FIRST_CHUNK_TREE CHUNK_ITEM 3808054738944) block 21200896 gen 1732 key (FIRST_CHUNK_TREE CHUNK_ITEM 3972874108928) block 21446656 gen 1774 key (FIRST_CHUNK_TREE CHUNK_ITEM 4139304091648) block 21053440 gen 1822 key (FIRST_CHUNK_TREE CHUNK_ITEM 4305734074368) block 21544960 gen 1868 key (FIRST_CHUNK_TREE CHUNK_ITEM 4470553444352) block 21299200 gen 1597466 key (FIRST_CHUNK_TREE CHUNK_ITEM 4636983427072) block 21577728 gen 1965 key (FIRST_CHUNK_TREE CHUNK_ITEM 4803413409792) block 21413888 gen 2014 key (FIRST_CHUNK_TREE CHUNK_ITEM 4968232779776) block 21528576 gen 2065 key (FIRST_CHUNK_TREE CHUNK_ITEM 5134662762496) block 21626880 gen 2118 key (FIRST_CHUNK_TREE CHUNK_ITEM 5301092745216) block 21561344 gen 2167 key (FIRST_CHUNK_TREE CHUNK_ITEM 5465912115200) block 21594112 gen 2218 key (FIRST_CHUNK_TREE CHUNK_ITEM 5632342097920) block 21430272 gen 2270 key (FIRST_CHUNK_TREE CHUNK_ITEM 5798772080640) block 21692416 gen 2315 key (FIRST_CHUNK_TREE CHUNK_ITEM 5963591450624) block 21643264 gen 2358 key (FIRST_CHUNK_TREE CHUNK_ITEM 6130021433344) block 21659648 gen 2411 key (FIRST_CHUNK_TREE CHUNK_ITEM 6294840803328) block 21676032 gen 2465 key (FIRST_CHUNK_TREE CHUNK_ITEM 6461270786048) block 21757952 gen 2512 key (FIRST_CHUNK_TREE CHUNK_ITEM 6627700768768) block 21495808 gen 2558 key (FIRST_CHUNK_TREE CHUNK_ITEM 6792520138752) block 21708800 gen 2607 key (FIRST_CHUNK_TREE CHUNK_ITEM 6958950121472) block 21807104 gen 2655 key (FIRST_CHUNK_TREE CHUNK_ITEM 7125380104192) block 21725184 gen 2711 key (FIRST_CHUNK_TREE CHUNK_ITEM 7290199474176) block 21512192 gen 2762 key (FIRST_CHUNK_TREE CHUNK_ITEM 7456629456896) block 21774336 gen 2808 key (FIRST_CHUNK_TREE CHUNK_ITEM 7623059439616) block 21872640 gen 1463345 key (FIRST_CHUNK_TREE CHUNK_ITEM 7787878809600) block 21889024 gen 2907 key (FIRST_CHUNK_TREE CHUNK_ITEM 7954308792320) block 21905408 gen 2956 key (FIRST_CHUNK_TREE CHUNK_ITEM 8120738775040) block 21790720 gen 3004 key (FIRST_CHUNK_TREE CHUNK_ITEM 8285558145024) block 21839872 gen 3057 key (FIRST_CHUNK_TREE CHUNK_ITEM 8451988127744) block 21856256 gen 3107 key (FIRST_CHUNK_TREE CHUNK_ITEM 8618418110464) block 21921792 gen 3156 key (FIRST_CHUNK_TREE CHUNK_ITEM 8783237480448) block 21217280 gen 3205 key (FIRST_CHUNK_TREE CHUNK_ITEM 8949667463168) block 22003712 gen 3253 key (FIRST_CHUNK_TREE CHUNK_ITEM 9116097445888) block 21954560 gen 3304 key (FIRST_CHUNK_TREE CHUNK_ITEM 9280916815872) block 22036480 gen 3357 key (FIRST_CHUNK_TREE CHUNK_ITEM 9447346798592) block 22020096 gen 1463291 key (FIRST_CHUNK_TREE CHUNK_ITEM 9613776781312) block 21987328 gen 3462 key (FIRST_CHUNK_TREE CHUNK_ITEM 9778596151296) block 22413312 gen 1463210 key (FIRST_CHUNK_TREE CHUNK_ITEM 9945026134016) block 22052864 gen 3565 key (FIRST_CHUNK_TREE CHUNK_ITEM 10111456116736) block 22085632 gen 1463201 key (FIRST_CHUNK_TREE CHUNK_ITEM 10276275486720) block 21823488 gen 1463183 key (FIRST_CHUNK_TREE CHUNK_ITEM 10442705469440) block 21331968 gen 1463138 key (FIRST_CHUNK_TREE CHUNK_ITEM 10609135452160) block 22118400 gen 3761 key (FIRST_CHUNK_TREE CHUNK_ITEM 10773954822144) block 22315008 gen 1463057 key (FIRST_CHUNK_TREE CHUNK_ITEM 10940384804864) block 22151168 gen 1463003 key (FIRST_CHUNK_TREE CHUNK_ITEM 11106814787584) block 22495232 gen 1572124 key (FIRST_CHUNK_TREE CHUNK_ITEM 11271634157568) block 22200320 gen 1462940 key (FIRST_CHUNK_TREE CHUNK_ITEM 11438064140288) block 22282240 gen 1462904 key (FIRST_CHUNK_TREE CHUNK_ITEM 11607715348480) block 22249472 gen 1462571 key (FIRST_CHUNK_TREE CHUNK_ITEM 11770387234816) block 22265856 gen 1462454 key (FIRST_CHUNK_TREE CHUNK_ITEM 11948091506688) block 22167552 gen 1462274 key (FIRST_CHUNK_TREE CHUNK_ITEM 12136600305664) block 22183936 gen 1462103 key (FIRST_CHUNK_TREE CHUNK_ITEM 12282629193728) block 22134784 gen 1461923 key (FIRST_CHUNK_TREE CHUNK_ITEM 12451240214528) block 22347776 gen 1357546 key (FIRST_CHUNK_TREE CHUNK_ITEM 12619817680896) block 22069248 gen 504747 key (FIRST_CHUNK_TREE CHUNK_ITEM 12794871152640) block 22364160 gen 1404175 key (FIRST_CHUNK_TREE CHUNK_ITEM 12961301135360) block 22298624 gen 1423748 key (FIRST_CHUNK_TREE CHUNK_ITEM 13176183717888) block 21741568 gen 1234369 key (FIRST_CHUNK_TREE CHUNK_ITEM 13357713195008) block 22380544 gen 1332045 key (FIRST_CHUNK_TREE CHUNK_ITEM 13560717508608) block 22396928 gen 1441271 key (FIRST_CHUNK_TREE CHUNK_ITEM 13735838089216) block 22429696 gen 1423550 key (FIRST_CHUNK_TREE CHUNK_ITEM 13904415555584) block 22102016 gen 1461914 key (FIRST_CHUNK_TREE CHUNK_ITEM 14103091347456) block 21970944 gen 1461905 key (FIRST_CHUNK_TREE CHUNK_ITEM 14271702368256) block 22462464 gen 1479229 key (FIRST_CHUNK_TREE CHUNK_ITEM 14443501060096) block 22528000 gen 1572115 key (FIRST_CHUNK_TREE CHUNK_ITEM 14648585748480) block 22233088 gen 1571944 key (FIRST_CHUNK_TREE CHUNK_ITEM 14823605665792) block 22446080 gen 1571791 key (FIRST_CHUNK_TREE CHUNK_ITEM 14990035648512) block 22544384 gen 1556078 key (FIRST_CHUNK_TREE CHUNK_ITEM 15156465631232) block 22511616 gen 1555799 key (FIRST_CHUNK_TREE CHUNK_ITEM 15322895613952) block 22577152 gen 1586277 key (FIRST_CHUNK_TREE CHUNK_ITEM 15489325596672) block 22478848 gen 1561557 key (FIRST_CHUNK_TREE CHUNK_ITEM 15664345513984) block 22593536 gen 1590219 key (FIRST_CHUNK_TREE CHUNK_ITEM 15847955365888) block 22216704 gen 1600938 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 20:08 ` Marc MERLIN @ 2022-04-05 20:24 ` Josef Bacik 2022-04-05 20:37 ` Marc MERLIN 2022-04-05 23:51 ` Zygo Blaxell 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-05 20:24 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote: > > Well it's still the same, and this thing is 20mib into your fs so IDK > > how it would be screwing up now. Can you do > > > > ./btrfs inspect-internal dump-tree -b 21069824 > > > > and see what that spits out? IDK why it would suddenly start > > complaining about your chunk root. Thanks, > > Thanks for your patience and sticking with me > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null Ok well that worked, which means it found the chunk tree fine, I'm going to chalk this up to it just fucking with me and ignore it for now. I pushed some changes for the find root thing, can you re-run ./btrfs-find-root -o 1 /dev/whatever it should be less noisy and spit out one line at the end, "Found tree root at blah blah". Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 20:24 ` Josef Bacik @ 2022-04-05 20:37 ` Marc MERLIN 2022-04-05 21:07 ` Josef Bacik 2022-04-05 23:51 ` Zygo Blaxell 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 20:37 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote: > > > Well it's still the same, and this thing is 20mib into your fs so IDK > > > how it would be screwing up now. Can you do > > > > > > ./btrfs inspect-internal dump-tree -b 21069824 > > > > > > and see what that spits out? IDK why it would suddenly start > > > complaining about your chunk root. Thanks, > > > > Thanks for your patience and sticking with me > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null > > Ok well that worked, which means it found the chunk tree fine, I'm > going to chalk this up to it just fucking with me and ignore it for > now. I pushed some changes for the find root thing, can you re-run Typo I fixed: kernel-shared/disk-io.c: In function ‘verify_parent_transid’: kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’? 278 | if (eb->fs_info && eb->fs_info->suppress_check_block_errrs) | ^~~~~~~~~~~~~~~~~~~~~~~~~~ | suppress_check_block_errors > ./btrfs-find-root -o 1 /dev/whatever > > it should be less noisy and spit out one line at the end, "Found tree > root at blah blah". Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root -rwxr-xr-x 1 root staff 3375608 Apr 5 13:29 btrfs-find-root* gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56289ce172a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x56289ce172a0 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure (many deleted) Ignoring transid failure Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1 Good news :) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 20:37 ` Marc MERLIN @ 2022-04-05 21:07 ` Josef Bacik 2022-04-05 21:14 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 21:07 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 4:37 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote: > > On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote: > > > > Well it's still the same, and this thing is 20mib into your fs so IDK > > > > how it would be screwing up now. Can you do > > > > > > > > ./btrfs inspect-internal dump-tree -b 21069824 > > > > > > > > and see what that spits out? IDK why it would suddenly start > > > > complaining about your chunk root. Thanks, > > > > > > Thanks for your patience and sticking with me > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null > > > > Ok well that worked, which means it found the chunk tree fine, I'm > > going to chalk this up to it just fucking with me and ignore it for > > now. I pushed some changes for the find root thing, can you re-run > > > Typo I fixed: > kernel-shared/disk-io.c: In function ‘verify_parent_transid’: > kernel-shared/disk-io.c:278:35: error: ‘struct btrfs_fs_info’ has no member named ‘suppress_check_block_errrs’; did you mean ‘suppress_check_block_errors’? > 278 | if (eb->fs_info && eb->fs_info->suppress_check_block_errrs) > | ^~~~~~~~~~~~~~~~~~~~~~~~~~ > | suppress_check_block_errors > > > ./btrfs-find-root -o 1 /dev/whatever > > > > it should be less noisy and spit out one line at the end, "Found tree > > root at blah blah". Thanks, > > gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs-find-root > -rwxr-xr-x 1 root staff 3375608 Apr 5 13:29 btrfs-find-root* > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > FS_INFO IS 0x56289ce172a0 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Couldn't find the last root for 4 > Couldn't setup device tree > FS_INFO AFTER IS 0x56289ce172a0 > Ignoring transid failure > Ignoring transid failure > Ignoring transid failure > Ignoring transid failure > (many deleted) > Ignoring transid failure > Superblock thinks the generation is 1602089 > Superblock thinks the level is 1 > Found tree root at 13577814573056 gen 1602089 level 1 > > Good news :) Heh, except it's because I forgot to tell it to treat transid mismatch's as a real problem. Added some more code, lets give that a whirl, thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:07 ` Josef Bacik @ 2022-04-05 21:14 ` Marc MERLIN 2022-04-05 21:19 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 21:14 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 05:07:27PM -0400, Josef Bacik wrote: > Heh, except it's because I forgot to tell it to treat transid > mismatch's as a real problem. Added some more code, lets give that a > whirl, thanks, Looks better gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x559090e392a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x559090e392a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:14 ` Marc MERLIN @ 2022-04-05 21:19 ` Josef Bacik 2022-04-05 21:25 ` Marc MERLIN 2022-04-05 21:26 ` Marc MERLIN 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-05 21:19 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 5:14 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 05:07:27PM -0400, Josef Bacik wrote: > > Heh, except it's because I forgot to tell it to treat transid > > mismatch's as a real problem. Added some more code, lets give that a > > whirl, thanks, > > Looks better > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > FS_INFO IS 0x559090e392a0 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Couldn't find the last root for 4 > Couldn't setup device tree > FS_INFO AFTER IS 0x559090e392a0 > Superblock thinks the generation is 1602089 > Superblock thinks the level is 1 > Found tree root at 13577814573056 gen 1602089 level 1 Otra vez por favor, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:19 ` Josef Bacik @ 2022-04-05 21:25 ` Marc MERLIN 2022-04-05 21:26 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 21:25 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote: > Otra vez por favor, sauron:/var/local/src/btrfs-progs-josefbacik# make [CC] btrfs-find-root.o btrfs-find-root.c:381:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function] 381 | static void print_find_root_result(struct cache_tree *result, I commented out a couple of unused functions, built and gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556c1c6bb2a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x556c1c6bb2a0 gargamel:/var/local/src/btrfs-progs-josefbacik# -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:19 ` Josef Bacik 2022-04-05 21:25 ` Marc MERLIN @ 2022-04-05 21:26 ` Marc MERLIN 2022-04-05 21:35 ` Josef Bacik 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 21:26 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote: > Otra vez por favor, (gdb) run -o 1 /dev/mapper/dshelf1a Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x5555555cf2a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x5555555cf2a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Program received signal SIGSEGV, Segmentation fault. lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384) at common/extent-cache.c:142 142 range.start = start; (gdb) bt #0 lookup_cache_extent (tree=tree@entry=0x5555555cf308, start=start@entry=505282560, size=size@entry=16384) at common/extent-cache.c:142 #1 0x0000555555579d1e in alloc_extent_buffer (fs_info=0x5555555cf2a0, bytenr=505282560, blocksize=16384) at kernel-shared/extent_io.c:737 #2 0x000055555556e44c in btrfs_find_create_tree_block (fs_info=<optimized out>, bytenr=<optimized out>) at kernel-shared/disk-io.c:230 #3 0x0000555555570208 in read_tree_block (fs_info=0x5555555cf2a0, bytenr=<optimized out>, parent_transid=1601370) at kernel-shared/disk-io.c:374 #4 0x00005555555a06df in try_read_block (slot=0, eb=0x5555721ef1a0) at btrfs-find-root.c:109 #5 count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:162 #6 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #7 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #8 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #9 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #10 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 #11 0x00005555555a0724 in count_bad_items (eb=0x5555721ef1a0) at btrfs-find-root.c:164 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:26 ` Marc MERLIN @ 2022-04-05 21:35 ` Josef Bacik 2022-04-05 21:43 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 21:35 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote: > > Otra vez por favor, > > (gdb) run -o 1 /dev/mapper/dshelf1a > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > FS_INFO IS 0x5555555cf2a0 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Couldn't find the last root for 4 > Couldn't setup device tree > FS_INFO AFTER IS 0x5555555cf2a0 > Superblock thinks the generation is 1602089 > Superblock thinks the level is 1 > > Program received signal SIGSEGV, Segmentation fault. Ugh sorry, try again. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:35 ` Josef Bacik @ 2022-04-05 21:43 ` Marc MERLIN 2022-04-05 22:41 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 21:43 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 05:35:02PM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote: > > > Otra vez por favor, > > > > (gdb) run -o 1 /dev/mapper/dshelf1a > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > FS_INFO IS 0x5555555cf2a0 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > Couldn't find the last root for 4 > > Couldn't setup device tree > > FS_INFO AFTER IS 0x5555555cf2a0 > > Superblock thinks the generation is 1602089 > > Superblock thinks the level is 1 > > > > Program received signal SIGSEGV, Segmentation fault. > > Ugh sorry, try again. Thanks, Thanks. Note that you still have 2 functions to comment out From https://github.com/josefbacik/btrfs-progs ce32ea15..914c9847 for-marc -> origin/for-marc Updating ce32ea15..914c9847 Fast-forward btrfs-find-root.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) sauron:/var/local/src/btrfs-progs-josefbacik# make [CC] btrfs-find-root.o btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function] 383 | static void print_find_root_result(struct cache_tree *result, New version after I fixed the build warnings that were errors for me gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x564897ec32a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x564897ec32a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 Found tree root at 13577814573056 gen 1602089 level 1 gargamel:/var/local/src/btrfs-progs-josefbacik# -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 21:43 ` Marc MERLIN @ 2022-04-05 22:41 ` Josef Bacik 2022-04-05 22:58 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-05 22:41 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 5:43 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 05:35:02PM -0400, Josef Bacik wrote: > > On Tue, Apr 5, 2022 at 5:26 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Tue, Apr 05, 2022 at 05:19:57PM -0400, Josef Bacik wrote: > > > > Otra vez por favor, > > > > > > (gdb) run -o 1 /dev/mapper/dshelf1a > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs-find-root -o 1 /dev/mapper/dshelf1a > > > [Thread debugging using libthread_db enabled] > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > > > FS_INFO IS 0x5555555cf2a0 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > > > Couldn't find the last root for 4 > > > Couldn't setup device tree > > > FS_INFO AFTER IS 0x5555555cf2a0 > > > Superblock thinks the generation is 1602089 > > > Superblock thinks the level is 1 > > > > > > Program received signal SIGSEGV, Segmentation fault. > > > > Ugh sorry, try again. Thanks, > > > Thanks. Note that you still have 2 functions to comment out > From https://github.com/josefbacik/btrfs-progs > ce32ea15..914c9847 for-marc -> origin/for-marc > Updating ce32ea15..914c9847 > Fast-forward > btrfs-find-root.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) > sauron:/var/local/src/btrfs-progs-josefbacik# make > [CC] btrfs-find-root.o > btrfs-find-root.c:383:13: warning: ‘print_find_root_result’ defined but not used [-Wunused-function] > 383 | static void print_find_root_result(struct cache_tree *result, > > > New version after I fixed the build warnings that were errors for me > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/out > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > FS_INFO IS 0x564897ec32a0 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 > Couldn't find the last root for 4 > Couldn't setup device tree > FS_INFO AFTER IS 0x564897ec32a0 > Superblock thinks the generation is 1602089 > Superblock thinks the level is 1 > Found tree root at 13577814573056 gen 1602089 level 1 > gargamel:/var/local/src/btrfs-progs-josefbacik# > I'm wandering down this rabbit hole because if I'm able to read the blocks fine, wtf is the device root not being found. I've pushed more printfs, lets see what that says. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 22:41 ` Josef Bacik @ 2022-04-05 22:58 ` Marc MERLIN 2022-04-06 0:23 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-05 22:58 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 1872 bytes --] On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote: > I'm wandering down this rabbit hole because if I'm able to read the > blocks fine, wtf is the device root not being found. I've pushed more > printfs, lets see what that says. Thanks, Sure thing. Output is kind of big, summary here, full output bzipped and attached But isn't the tree found at the end? Found tree root at 13577814573056 gen 1602089 level 1 output: parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55be0733a2a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x55be0733a2a0 Superblock thinks the generation is 1602089 Superblock thinks the level is 1 checking root item 11221 bad root checking root item 11222 checking root item 11223 bad root checking root item 11224 checking root item 11322 checking root item 11322 bad root checking root item 152935 (...) checking block 13577814573056 bad block checking block 13577814573056 bad block checking block 13577814573056 bad block checking block 13577814573056 checking block 13577814573056 bad block Found tree root at 13577814573056 gen 1602089 level 1 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ [-- Attachment #2: out.bz2 --] [-- Type: application/octet-stream, Size: 16464 bytes --] ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 22:58 ` Marc MERLIN @ 2022-04-06 0:23 ` Josef Bacik 2022-04-06 0:30 ` Marc MERLIN 2022-04-06 0:35 ` Marc MERLIN 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-06 0:23 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 6:58 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote: > > I'm wandering down this rabbit hole because if I'm able to read the > > blocks fine, wtf is the device root not being found. I've pushed more > > printfs, lets see what that says. Thanks, > > Sure thing. Output is kind of big, summary here, full output bzipped and > attached > But isn't the tree found at the end? > Found tree root at 13577814573056 gen 1602089 level 1 > Alright it's time for the crazy. Go ahead and pull and start running. This is going to take a while to run, we're basically going to walk and check all the node pointers in the tree root, if it doesn't look right we're going to search the metadata for the best copy to use, and then update the block to point at the new block. It has to do the full search every time, because we don't have time for me to properly implement a cache, so don't be worried if it takes a while. It may print out stuff, if it looks like it's looping stop it and let me know, but I don't think I fucked it up. You're going to see a lot "fixed root <id>", "fixed slot <whatever>", if you see it repeating the same slot or root then we know we have a problem. Once it's done can you do the print-tree ROOT thing again so I can see if it did the right thing. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 0:23 ` Josef Bacik @ 2022-04-06 0:30 ` Marc MERLIN 2022-04-06 0:35 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 0:30 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 6:58 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 06:41:02PM -0400, Josef Bacik wrote: > > > I'm wandering down this rabbit hole because if I'm able to read the > > > blocks fine, wtf is the device root not being found. I've pushed more > > > printfs, lets see what that says. Thanks, > > > > Sure thing. Output is kind of big, summary here, full output bzipped and > > attached > > But isn't the tree found at the end? > > Found tree root at 13577814573056 gen 1602089 level 1 > > > > Alright it's time for the crazy. Go ahead and pull and start running. > This is going to take a while to run, we're basically going to walk > and check all the node pointers in the tree root, if it doesn't look > right we're going to search the metadata for the best copy to use, and > then update the block to point at the new block. It has to do the > full search every time, because we don't have time for me to properly > implement a cache, so don't be worried if it takes a while. Cool, thanks again for all your help, I hope it helps improving the tools and recovery for all gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x562612a192a0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x562612a192a0 I'll let it run and report back. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 0:23 ` Josef Bacik 2022-04-06 0:30 ` Marc MERLIN @ 2022-04-06 0:35 ` Marc MERLIN 2022-04-06 0:39 ` Josef Bacik 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 0:35 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote: > Alright it's time for the crazy. Go ahead and pull and start running. > This is going to take a while to run, we're basically going to walk > and check all the node pointers in the tree root, if it doesn't look > right we're going to search the metadata for the best copy to use, and > then update the block to point at the new block. It has to do the > full search every time, because we don't have time for me to properly > implement a cache, so don't be worried if it takes a while. > > It may print out stuff, if it looks like it's looping stop it and let > me know, but I don't think I fucked it up. You're going to see a lot > "fixed root <id>", "fixed slot <whatever>", if you see it repeating > the same slot or root then we know we have a problem. I assume this is not good? ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted I am running as root of course. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 0:35 ` Marc MERLIN @ 2022-04-06 0:39 ` Josef Bacik 2022-04-06 1:08 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 0:39 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 8:35 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote: > > Alright it's time for the crazy. Go ahead and pull and start running. > > This is going to take a while to run, we're basically going to walk > > and check all the node pointers in the tree root, if it doesn't look > > right we're going to search the metadata for the best copy to use, and > > then update the block to point at the new block. It has to do the > > full search every time, because we don't have time for me to properly > > implement a cache, so don't be worried if it takes a while. > > > > It may print out stuff, if it looks like it's looping stop it and let > > me know, but I don't think I fucked it up. You're going to see a lot > > "fixed root <id>", "fixed slot <whatever>", if you see it repeating > > the same slot or root then we know we have a problem. > > I assume this is not good? > ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted > > I am running as root of course. > Yeah I'm just a dumbass, try again please. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 0:39 ` Josef Bacik @ 2022-04-06 1:08 ` Josef Bacik 2022-04-06 1:14 ` Marc MERLIN 2022-04-06 3:12 ` Marc MERLIN 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-06 1:08 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 8:39 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On Tue, Apr 5, 2022 at 8:35 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 08:23:42PM -0400, Josef Bacik wrote: > > > Alright it's time for the crazy. Go ahead and pull and start running. > > > This is going to take a while to run, we're basically going to walk > > > and check all the node pointers in the tree root, if it doesn't look > > > right we're going to search the metadata for the best copy to use, and > > > then update the block to point at the new block. It has to do the > > > full search every time, because we don't have time for me to properly > > > implement a cache, so don't be worried if it takes a while. > > > > > > It may print out stuff, if it looks like it's looping stop it and let > > > me know, but I don't think I fucked it up. You're going to see a lot > > > "fixed root <id>", "fixed slot <whatever>", if you see it repeating > > > the same slot or root then we know we have a problem. > > > > I assume this is not good? > > ERROR: failed to write bytenr 13577814573056 length 16384 devid 1 dev_bytenr 13439981355008: Operation not permitted > > > > I am running as root of course. > > > > Yeah I'm just a dumbass, try again please. Thanks, > Also keep in mind this isn't the final fix, this is the pre-repair so hopefully fsck can put the rest of it back together. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 1:08 ` Josef Bacik @ 2022-04-06 1:14 ` Marc MERLIN 2022-04-06 3:12 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 1:14 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > > Yeah I'm just a dumbass, try again please. Thanks, You've kindly given me around 20 fixes/patches in barely a day on a filesystem that is not local to you :) > Also keep in mind this isn't the final fix, this is the pre-repair so > hopefully fsck can put the rest of it back together. Thanks, Ack. It seems to work better or at least I got no output as of right now with more runtime than last time. It'll likely run overnight and I'll report tomorrow, thanks. gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x557e9cc375f0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x557e9cc375f0 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 1:08 ` Josef Bacik 2022-04-06 1:14 ` Marc MERLIN @ 2022-04-06 3:12 ` Marc MERLIN 2022-04-06 3:34 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 3:12 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > Also keep in mind this isn't the final fix, this is the pre-repair so > hopefully fsck can put the rest of it back together. Thanks, Done, so now I can run btrfs check --repair /dev/mapper/dshelf1a ? with or without mode=lowmem? Do I need to reset the ctree too, or repair will do it? [642321.755553] BTRFS warning (device dm-17): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [642321.755566] BTRFS info (device dm-17): trying to use backup root at mount time [642321.755568] BTRFS info (device dm-17): disk space caching is enabled [642321.755569] BTRFS info (device dm-17): has skinny extents [642322.242161] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [642322.282393] BTRFS error (device dm-17): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [642322.313951] BTRFS error (device dm-17): failed to read chunk tree: -5 [642322.339490] BTRFS error (device dm-17): open_ctree failed gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outn parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x557e9cc375f0 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 parent transid verify failed on 13577821667328 wanted 1602089 found 1602242 Couldn't find the last root for 4 Couldn't setup device tree FS_INFO AFTER IS 0x557e9cc375f0 fixed slot 0 fixed slot 2 fixed slot 5 fixed slot 7 (...) Couldn't find a replacement block for slot 261 fixed slot 263 fixed slot 264 fixed slot 265 fixed slot 267 fixed slot 269 fixed slot 271 fixed slot 273 fixed slot 274 fixed slot 276 fixed slot 277 fixed slot 278 fixed slot 279 fixed slot 282 fixed slot 283 fixed slot 285 fixed slot 286 fixed slot 287 fixed slot 288 fixed slot 289 fixed slot 291 fixed slot 293 fixed slot 294 fixed slot 296 fixed slot 297 fixed slot 298 fixed slot 300 fixed slot 301 fixed slot 302 fixed slot 303 fixed slot 304 fixed slot 305 fixed slot 306 fixed slot 307 fixed slot 308 fixed slot 309 fixed slot 311 fixed slot 312 fixed slot 313 fixed slot 314 fixed slot 315 fixed slot 316 fixed slot 317 fixed slot 318 fixed slot 319 fixed slot 320 fixed slot 321 Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 3:12 ` Marc MERLIN @ 2022-04-06 3:34 ` Marc MERLIN 2022-04-06 15:20 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 3:34 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote: > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > > Also keep in mind this isn't the final fix, this is the pre-repair so > > hopefully fsck can put the rest of it back together. Thanks, > > Done, so now I can run > btrfs check --repair /dev/mapper/dshelf1a ? Here is regular check: gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x564d665f1e50 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x564d665f1e50 Checking filesystem on /dev/mapper/dshelf1a UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e [1/7] checking root items checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to repair root items: Input/output error [2/7] checking extents parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645121789952 wanted 1602089 found 1600922 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 checksum verify failed on 13577870934016 wanted 0xfffb4d18 found 0xb0be1c94 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645280632832 wanted 1602089 found 1601000 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645399678976 wanted 1602089 found 1602243 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 bad tree block 15645399678976, bad level, 236 > 8 ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space cache [4/7] checking fs roots parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 ERROR: invalid generation for root 5, have 1602297 expect (0, 1602090] checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 parent transid verify failed on 15645248897024 wanted 1602297 found 1600937 checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 bad tree block 15645248897024, bad level, 66 > 8 The following tree block(s) is corrupted in tree 5: tree block bytenr: 15645248880640, level: 2, node key: (256, 1, 0) root 5 root dir 256 not found parent transid verify failed on 13577702703104 wanted 1602089 found 1602241 parent transid verify failed on 12511683117056 wanted 1602089 found 1602202 parent transid verify failed on 15645121789952 wanted 1602089 found 1602135 parent transid verify failed on 15645274947584 wanted 1602089 found 1602298 parent transid verify failed on 11160884019200 wanted 1602089 found 1602180 parent transid verify failed on 15645280632832 wanted 1602089 found 1602298 parent transid verify failed on 15645397942272 wanted 1602089 found 1602141 parent transid verify failed on 15645399678976 wanted 1602089 found 1602243 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 checksum verify failed on 15645399678976 wanted 0xf1e9450d found 0xe8cf0418 bad tree block 15645399678976, bad level, 236 > 8 ERROR: errors found in fs roots found 0 bytes used, error(s) found total csum bytes: 0 total tree bytes: 0 total fs tree bytes: 0 total extent tree bytes: 0 btree space waste bytes: 0 file data blocks allocated: 0 referenced 0 gargamel:/var/local/src/btrfs-progs-josefbacik# -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 3:34 ` Marc MERLIN @ 2022-04-06 15:20 ` Josef Bacik 2022-04-06 18:54 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 15:20 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote: > > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > > > Also keep in mind this isn't the final fix, this is the pre-repair so > > > hopefully fsck can put the rest of it back together. Thanks, > > > > Done, so now I can run > > btrfs check --repair /dev/mapper/dshelf1a ? > > Here is regular check: > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a Woo ok we're almost there, that's great. Can you re-run the btrfs-find-root and see what it spits out this time? It should run faster now that it fixed everything, I'm wondering what it wasn't able to fix and if I should run the same check on all the roots. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 15:20 ` Josef Bacik @ 2022-04-06 18:54 ` Marc MERLIN 2022-04-06 18:57 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 18:54 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 11:20:13AM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote: > > > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > > > > Also keep in mind this isn't the final fix, this is the pre-repair so > > > > hopefully fsck can put the rest of it back together. Thanks, > > > > > > Done, so now I can run > > > btrfs check --repair /dev/mapper/dshelf1a ? > > > > Here is regular check: > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a > > Woo ok we're almost there, that's great. Can you re-run the > btrfs-find-root and see what it spits out this time? It should run > faster now that it fixed everything, I'm wondering what it wasn't able > to fix and if I should run the same check on all the roots. Thanks, took a few hours to run, but it's done now. Looks like it fixed things a 2nd time? Should I run gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a now? output: gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outm parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55cf0ce435f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55cf0ce435f0 fixed slot 0 fixed slot 2 fixed slot 5 fixed slot 7 fixed slot 12 fixed slot 27 fixed slot 28 fixed slot 32 fixed slot 34 fixed slot 35 fixed slot 36 fixed slot 45 fixed slot 48 fixed slot 49 fixed slot 50 fixed slot 52 fixed slot 53 fixed slot 54 fixed slot 55 fixed slot 56 fixed slot 60 fixed slot 61 fixed slot 63 fixed slot 76 fixed slot 77 fixed slot 85 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 98 fixed slot 99 fixed slot 100 fixed slot 103 fixed slot 104 fixed slot 106 fixed slot 107 fixed slot 108 fixed slot 109 fixed slot 110 fixed slot 111 fixed slot 112 fixed slot 114 fixed slot 120 fixed slot 135 fixed slot 140 fixed slot 141 fixed slot 142 fixed slot 146 fixed slot 155 fixed slot 156 fixed slot 158 fixed slot 159 fixed slot 160 fixed slot 161 fixed slot 165 fixed slot 166 fixed slot 171 fixed slot 173 fixed slot 174 fixed slot 180 fixed slot 181 fixed slot 182 fixed slot 183 fixed slot 184 fixed slot 185 fixed slot 186 fixed slot 212 fixed slot 213 fixed slot 221 fixed slot 222 fixed slot 224 fixed slot 226 fixed slot 227 fixed slot 228 fixed slot 229 fixed slot 231 fixed slot 232 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 244 fixed slot 245 fixed slot 248 fixed slot 252 fixed slot 259 fixed slot 260 Couldn't find a replacement block for slot 261 fixed slot 263 fixed slot 264 fixed slot 265 fixed slot 267 fixed slot 269 fixed slot 271 fixed slot 273 fixed slot 274 fixed slot 276 fixed slot 277 fixed slot 278 fixed slot 279 fixed slot 282 fixed slot 283 fixed slot 285 fixed slot 286 fixed slot 287 fixed slot 288 fixed slot 289 fixed slot 291 fixed slot 293 fixed slot 294 fixed slot 296 fixed slot 297 fixed slot 298 fixed slot 300 fixed slot 301 fixed slot 302 fixed slot 303 fixed slot 304 fixed slot 305 fixed slot 306 fixed slot 307 fixed slot 308 fixed slot 309 fixed slot 311 fixed slot 312 fixed slot 313 fixed slot 314 fixed slot 315 fixed slot 316 fixed slot 317 fixed slot 318 fixed slot 319 fixed slot 320 fixed slot 321 gargamel:/var/local/src/btrfs-progs-josefbacik# -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 18:54 ` Marc MERLIN @ 2022-04-06 18:57 ` Josef Bacik 2022-04-06 19:16 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 18:57 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 6, 2022 at 2:54 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 06, 2022 at 11:20:13AM -0400, Josef Bacik wrote: > > On Tue, Apr 5, 2022 at 11:34 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Tue, Apr 05, 2022 at 08:12:55PM -0700, Marc MERLIN wrote: > > > > On Tue, Apr 05, 2022 at 09:08:32PM -0400, Josef Bacik wrote: > > > > > Also keep in mind this isn't the final fix, this is the pre-repair so > > > > > hopefully fsck can put the rest of it back together. Thanks, > > > > > > > > Done, so now I can run > > > > btrfs check --repair /dev/mapper/dshelf1a ? > > > > > > Here is regular check: > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check /dev/mapper/dshelf1a > > > > Woo ok we're almost there, that's great. Can you re-run the > > btrfs-find-root and see what it spits out this time? It should run > > faster now that it fixed everything, I'm wondering what it wasn't able > > to fix and if I should run the same check on all the roots. Thanks, > > took a few hours to run, but it's done now. Looks like it fixed things a 2nd time? > > Should I run > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a > now? > Yeah lets go for that, I saw some errors on your fs tree's earlier, I may need to adapt this to fix that tree, or if it's a snapshot we can just delete it. We can burn that bridge when we get to it, thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 18:57 ` Josef Bacik @ 2022-04-06 19:16 ` Marc MERLIN 2022-04-06 19:53 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 19:16 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 02:57:03PM -0400, Josef Bacik wrote: > Yeah lets go for that, I saw some errors on your fs tree's earlier, I > may need to adapt this to fix that tree, or if it's a snapshot we can > just delete it. We can burn that bridge when we get to it, thanks, Sounds good. Is mode=lowmem a thing of the past by the way? (I remember that the old regular repair would eat up my 32GB of RAM and take the machine down) The current one failed quickly though gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a enabling repair mode WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55e2e75defc0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x55e2e75defc0 Checking filesystem on /dev/mapper/dshelf1a UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e [1/7] checking root items checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 bad tree block 15645261971456, bad level, 127 > 8 ERROR: failed to repair root items: Input/output error -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 19:16 ` Marc MERLIN @ 2022-04-06 19:53 ` Josef Bacik 2022-04-06 20:34 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 19:53 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 6, 2022 at 3:16 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 06, 2022 at 02:57:03PM -0400, Josef Bacik wrote: > > Yeah lets go for that, I saw some errors on your fs tree's earlier, I > > may need to adapt this to fix that tree, or if it's a snapshot we can > > just delete it. We can burn that bridge when we get to it, thanks, > > Sounds good. Is mode=lowmem a thing of the past by the way? > (I remember that the old regular repair would eat up my 32GB of RAM and > take the machine down) > > The current one failed quickly though > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --repair /dev/mapper/dshelf1a > enabling repair mode > WARNING: > > Do not use --repair unless you are advised to do so by a developer > or an experienced user, and then only after having accepted that no > fsck can successfully repair all types of filesystem corruption. Eg. > some software or hardware bugs can fatally damage a volume. > The operation will start in 10 seconds. > Use Ctrl-C to stop it. > 10 9 8 7 6 5 4 3 2 1 > Starting repair. > Opening filesystem to check... > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > parent transid verify failed on 22216704 wanted 1600938 found 1602177 > FS_INFO IS 0x55e2e75defc0 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 > Couldn't find the last root for 8 > checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 > parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 > checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 > bad tree block 15645261971456, bad level, 127 > 8 > ERROR: failed to read block groups: Input/output error > FS_INFO AFTER IS 0x55e2e75defc0 > Checking filesystem on /dev/mapper/dshelf1a > UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e > [1/7] checking root items > checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 > parent transid verify failed on 15645261971456 wanted 1602297 found 1600989 > checksum verify failed on 15645261971456 wanted 0x10a0c9b9 found 0x08b85944 > bad tree block 15645261971456, bad level, 127 > 8 > ERROR: failed to repair root items: Input/output error > Alright so it's up to you, clearly my put the tree back together stuff takes forever, you can use --init-extent-tree with --lowmem if you'd like, I have no idea what the time on that is going to look like. You still may run into problems with that if your subvols are screwed, but then I just have to do the work to put subvols back together. I *think* the --init-extent-tree thing will be faster, but let me know what you want to do. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 19:53 ` Josef Bacik @ 2022-04-06 20:34 ` Marc MERLIN 2022-04-06 20:38 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 20:34 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 03:53:34PM -0400, Josef Bacik wrote: > Alright so it's up to you, clearly my put the tree back together stuff > takes forever, you can use --init-extent-tree with --lowmem if you'd > like, I have no idea what the time on that is going to look like. You > still may run into problems with that if your subvols are screwed, but > then I just have to do the work to put subvols back together. I > *think* the --init-extent-tree thing will be faster, but let me know > what you want to do. Thanks, Thanks for sticking with me all this time. So, it was pretty quick before it failed: gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a enabling repair mode WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x556ddb2fffc0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x556ddb2fffc0 Checking filesystem on /dev/mapper/dshelf1a UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e Creating a new extent tree checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8 Error reading tree block error pinning down used bytes ERROR: commit_root already set when starting transaction extent buffer leak: start 13577814573056 len 16384 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 20:34 ` Marc MERLIN @ 2022-04-06 20:38 ` Josef Bacik 2022-04-06 20:56 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 20:38 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 6, 2022 at 4:34 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 06, 2022 at 03:53:34PM -0400, Josef Bacik wrote: > > Alright so it's up to you, clearly my put the tree back together stuff > > takes forever, you can use --init-extent-tree with --lowmem if you'd > > like, I have no idea what the time on that is going to look like. You > > still may run into problems with that if your subvols are screwed, but > > then I just have to do the work to put subvols back together. I > > *think* the --init-extent-tree thing will be faster, but let me know > > what you want to do. Thanks, > > Thanks for sticking with me all this time. > So, it was pretty quick before it failed: > Ah yeah I should have expected that, can you pull and re-run the same command? It'll tell me which root we need to target next. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 20:38 ` Josef Bacik @ 2022-04-06 20:56 ` Marc MERLIN 2022-04-06 21:05 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 20:56 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 04:38:06PM -0400, Josef Bacik wrote: > Ah yeah I should have expected that, can you pull and re-run the same > command? It'll tell me which root we need to target next. Thanks, Not sure it helped. gargamel:/var/local/src/btrfs-progs-josefbacik# l btrfs -rwxr-xr-x 1 root staff 6217688 Apr 6 13:38 btrfs* gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5 commit d51753abbc0ce345b9e163fe0fdbff086c9c609d Author: Josef Bacik <josef@toxicpanda.com> Date: Wed Apr 6 16:37:17 2022 -0400 i need the owner gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1a enabling repair mode WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55b85082dfc0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55b85082dfc0 Checking filesystem on /dev/mapper/dshelf1a UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e Creating a new extent tree checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 15645099524096 wanted 0x84d38588 found 0xebe9c53b checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645256777728 wanted 0x6f64534d found 0x8547f2a5 checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645277880320 wanted 0x94805cc2 found 0xb68a1cef checksum verify failed on 15645849485312 wanted 0x7afc435f found 0x6977ed7c checksum verify failed on 15645146660864 wanted 0xfa138a06 found 0x10cddfd6 checksum verify failed on 15645610672128 wanted 0x2ceeada6 found 0x2db846f5 checksum verify failed on 15645208985600 wanted 0xbaaa42be found 0x769e23f0 checksum verify failed on 11822142046208 wanted 0xc4cc3d0a found 0xace45394 checksum verify failed on 11822142177280 wanted 0x1ac605a6 found 0x17c9826e checksum verify failed on 6294487416832 wanted 0xe2be154a found 0xff9fb6b0 checksum verify failed on 11971034955776 wanted 0xcab8f347 found 0xdc99be9f checksum verify failed on 15646027497472 wanted 0xfe85f37e found 0x4e0364ef checksum verify failed on 15645802315776 wanted 0xdb3aba28 found 0x747ed64e checksum verify failed on 15645883285504 wanted 0x62bd1000 found 0x7f1300cc checksum verify failed on 13577053175808 wanted 0x4cec1ad0 found 0xf847830e checksum verify failed on 15646038179840 wanted 0xe040a9e4 found 0x31183a57 checksum verify failed on 15646040834048 wanted 0x39f56fc2 found 0x336aee8b checksum verify failed on 15646040604672 wanted 0xa2ff0db7 found 0xff9b0719 checksum verify failed on 15645608050688 wanted 0xe1019329 found 0x835ef4b7 checksum verify failed on 15646038212608 wanted 0x06fb70d5 found 0x704233fa checksum verify failed on 15645668425728 wanted 0x12b8b29d found 0x7e5aee23 checksum verify failed on 15645461397504 wanted 0xf186173b found 0xd3930df4 checksum verify failed on 15645684269056 wanted 0x6222cf99 found 0x50b7b0dd checksum verify failed on 15645870178304 wanted 0xf8950821 found 0x1743580d checksum verify failed on 15646019764224 wanted 0xd96815db found 0xa62eb220 checksum verify failed on 15645870358528 wanted 0x722138e3 found 0x9dbf8691 checksum verify failed on 15646016913408 wanted 0xfe1e83f4 found 0x664795e4 checksum verify failed on 10194600787968 wanted 0x37da9d7c found 0xf7d76f11 checksum verify failed on 15645949083648 wanted 0x14efeeac found 0x46344821 checksum verify failed on 13577202368512 wanted 0x4f5909be found 0x91649194 checksum verify failed on 15645136207872 wanted 0x4a16db57 found 0x042787a6 checksum verify failed on 13577220849664 wanted 0x31a222f0 found 0x98e6f9d5 checksum verify failed on 13577388523520 wanted 0x254b7048 found 0x90c5f781 checksum verify failed on 15645210001408 wanted 0xc64a49c0 found 0x8959550a checksum verify failed on 13577220997120 wanted 0xc1cf4d40 found 0xb20065f3 checksum verify failed on 15645422108672 wanted 0x63b9fb01 found 0xef400164 checksum verify failed on 365093765120 wanted 0xef058d7b found 0x6d2e4388 checksum verify failed on 15645943742464 wanted 0xa84c7cdb found 0x7435ced6 checksum verify failed on 15645944037376 wanted 0x1ce3f544 found 0x4110ee5a checksum verify failed on 15645943545856 wanted 0xb7391689 found 0x57835854 checksum verify failed on 15645943644160 wanted 0xd112805c found 0x65133906 checksum verify failed on 15645220012032 wanted 0xb4cb7eb3 found 0xe7134503 checksum verify failed on 11971032399872 wanted 0x1b1c1591 found 0x62f69873 checksum verify failed on 15645123133440 wanted 0xe29086d7 found 0x2bb17c15 checksum verify failed on 15645117890560 wanted 0xf94e5973 found 0xdf7eb70e checksum verify failed on 15645445554176 wanted 0x9dae4aed found 0x1087f009 checksum verify failed on 15645133553664 wanted 0x5840e80b found 0xa0492d9e checksum verify failed on 15645125804032 wanted 0xa71f19d7 found 0xd0d29066 checksum verify failed on 15645870292992 wanted 0x764a5b63 found 0x65f2d096 checksum verify failed on 15645209886720 wanted 0xdc6c9ccf found 0xfa3ec3d2 checksum verify failed on 15645167927296 wanted 0xa4d01cb6 found 0x44fb24c2 checksum verify failed on 15645786374144 wanted 0xb7242739 found 0x969cecbb checksum verify failed on 11971140157440 wanted 0x89f0ba19 found 0x761f18b3 checksum verify failed on 15645076209664 wanted 0xb5e74fba found 0x18d2d38b checksum verify failed on 15645214834688 wanted 0x81dcde00 found 0xc22cbcc7 checksum verify failed on 15645786636288 wanted 0x1edd6683 found 0xeeafe616 checksum verify failed on 15645217849344 wanted 0xf7574b63 found 0x00ba9085 checksum verify failed on 15645652893696 wanted 0x0fbc611c found 0x9539e808 checksum verify failed on 15645734240256 wanted 0x6ab0998e found 0xb51c5e0c checksum verify failed on 15645733978112 wanted 0xff835175 found 0x8f517448 checksum verify failed on 15645734010880 wanted 0x5363464a found 0x2aec1941 checksum verify failed on 13577168486400 wanted 0x56b26cca found 0xeb90f5da checksum verify failed on 13577616375808 wanted 0x190c00d0 found 0x18c748c3 checksum verify failed on 13577184264192 wanted 0x44d0fbb6 found 0x270072ce checksum verify failed on 13577226272768 wanted 0x14900bf7 found 0x2d0d3658 checksum verify failed on 13577199779840 wanted 0x39a4976f found 0xec18a40c checksum verify failed on 13577199730688 wanted 0x3e524f64 found 0xe82c5965 checksum verify failed on 13577225994240 wanted 0xc011d76f found 0x67bc3401 checksum verify failed on 15645260890112 wanted 0x0a7005a3 found 0xd182e084 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 checksum verify failed on 15645260890112 wanted 0x201bc700 found 0x462e9507 bad tree block 15645260890112, bad level, 59 > 8 Error reading tree block owner 7 error pinning down used bytes ERROR: commit_root already set when starting transaction extent buffer leak: start 13577814573056 len 16384 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 20:56 ` Marc MERLIN @ 2022-04-06 21:05 ` Josef Bacik 2022-04-07 1:08 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-06 21:05 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 6, 2022 at 4:56 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 06, 2022 at 04:38:06PM -0400, Josef Bacik wrote: > > Ah yeah I should have expected that, can you pull and re-run the same > > command? It'll tell me which root we need to target next. Thanks, > > Not sure it helped. > Yup that's what I needed, the csum tree is screwed, and actually the reinit stuff won't ignore the csum root if we're also init'ing the csum root which is kind of annoying. I updated the btrfs-find-root tool to do the tree repair thing for the csum root, you can run ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you can re-try the btrfs check. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 21:05 ` Josef Bacik @ 2022-04-07 1:08 ` Marc MERLIN 2022-04-07 1:18 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 1:08 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote: > Yup that's what I needed, the csum tree is screwed, and actually the > reinit stuff won't ignore the csum root if we're also init'ing the > csum root which is kind of annoying. I updated the btrfs-find-root > tool to do the tree repair thing for the csum root, you can run > ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you > can re-try the btrfs check. Thanks, Different output going back and forth, expected? gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55ba30af45f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55ba30af45f0 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 (still running) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 1:08 ` Marc MERLIN @ 2022-04-07 1:18 ` Josef Bacik 2022-04-07 4:37 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-07 1:18 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 6, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote: > > Yup that's what I needed, the csum tree is screwed, and actually the > > reinit stuff won't ignore the csum root if we're also init'ing the > > csum root which is kind of annoying. I updated the btrfs-find-root > > tool to do the tree repair thing for the csum root, you can run > > ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you > > can re-try the btrfs check. Thanks, > > Different output going back and forth, expected? > Yup it's iterating over all the blocks, so it's going to find different slots in different blocks that are fucked. You are having a lot of ones we can't find good matches for, depending on what those look like I may need to adjust the code to simply delete slots we don't find a good match for, which will be fine since we're going to clear this tree anyway. But let it finish and we'll see how the repair goes and I can do that if we need to. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 1:18 ` Josef Bacik @ 2022-04-07 4:37 ` Marc MERLIN 2022-04-07 4:40 ` Marc MERLIN 2022-04-07 5:20 ` Marc MERLIN 0 siblings, 2 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 4:37 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 09:18:17PM -0400, Josef Bacik wrote: > On Wed, Apr 6, 2022 at 9:08 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Wed, Apr 06, 2022 at 05:05:25PM -0400, Josef Bacik wrote: > > > Yup that's what I needed, the csum tree is screwed, and actually the > > > reinit stuff won't ignore the csum root if we're also init'ing the > > > csum root which is kind of annoying. I updated the btrfs-find-root > > > tool to do the tree repair thing for the csum root, you can run > > > ./btrfs-find-root /dev/whatever so it'll fixup that tree, and then you > > > can re-try the btrfs check. Thanks, > > > > Different output going back and forth, expected? > > > > Yup it's iterating over all the blocks, so it's going to find > different slots in different blocks that are fucked. > > You are having a lot of ones we can't find good matches for, depending > on what those look like I may need to adjust the code to simply delete > slots we don't find a good match for, which will be fine since we're > going to clear this tree anyway. But let it finish and we'll see how > the repair goes and I can do that if we need to. Thanks, > It took a lot longer, until it mostly hosed my machine by taking all the memory (32GB) gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x55ba30af45f0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x55ba30af45f0 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 208 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 130 fixed slot 138 fixed slot 33 Couldn't find a replacement block for slot 179 fixed slot 154 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 129 fixed slot 111 Couldn't find a replacement block for slot 103 Couldn't find a replacement block for slot 192 fixed slot 12 Couldn't find a replacement block for slot 205 fixed slot 24 fixed slot 28 fixed slot 31 fixed slot 38 fixed slot 47 fixed slot 62 fixed slot 75 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 108 Couldn't find a replacement block for slot 115 fixed slot 146 fixed slot 218 fixed slot 288 fixed slot 33 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 176 fixed slot 240 Couldn't find a replacement block for slot 241 fixed slot 34 fixed slot 35 Couldn't find a replacement block for slot 71 Couldn't find a replacement block for slot 140 fixed slot 213 Couldn't find a replacement block for slot 155 fixed slot 210 fixed slot 111 fixed slot 147 fixed slot 150 Couldn't find a replacement block for slot 234 Couldn't find a replacement block for slot 236 fixed slot 240 fixed slot 250 Couldn't find a replacement block for slot 259 fixed slot 269 fixed slot 283 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 48 fixed slot 135 fixed slot 192 fixed slot 206 fixed slot 43 fixed slot 111 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 45 fixed slot 47 fixed slot 212 fixed slot 49 fixed slot 190 fixed slot 4 fixed slot 40 fixed slot 110 fixed slot 227 fixed slot 290 fixed slot 296 fixed slot 297 fixed slot 302 fixed slot 349 fixed slot 357 Couldn't find a replacement block for slot 53 fixed slot 304 Couldn't find a replacement block for slot 328 fixed slot 376 fixed slot 54 fixed slot 125 fixed slot 195 Couldn't find a replacement block for slot 89 fixed slot 177 fixed slot 197 Couldn't find a replacement block for slot 219 fixed slot 233 fixed slot 36 fixed slot 52 fixed slot 61 fixed slot 71 fixed slot 113 fixed slot 119 fixed slot 138 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 200 fixed slot 215 Couldn't find a replacement block for slot 173 fixed slot 315 fixed slot 320 fixed slot 326 fixed slot 328 fixed slot 335 fixed slot 65 fixed slot 32 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 34 Couldn't find a replacement block for slot 37 fixed slot 40 Couldn't find a replacement block for slot 179 fixed slot 75 fixed slot 198 fixed slot 79 fixed slot 16 fixed slot 89 fixed slot 111 fixed slot 152 fixed slot 219 fixed slot 220 fixed slot 43 fixed slot 70 fixed slot 85 fixed slot 102 fixed slot 130 fixed slot 145 fixed slot 146 fixed slot 161 fixed slot 188 fixed slot 202 Couldn't find a replacement block for slot 9 fixed slot 62 fixed slot 122 fixed slot 128 fixed slot 137 fixed slot 142 fixed slot 216 Couldn't find a replacement block for slot 26 fixed slot 143 fixed slot 152 fixed slot 174 fixed slot 185 fixed slot 76 fixed slot 87 fixed slot 26 fixed slot 28 fixed slot 31 fixed slot 33 fixed slot 35 fixed slot 43 fixed slot 46 fixed slot 47 fixed slot 56 fixed slot 62 fixed slot 69 fixed slot 73 fixed slot 76 fixed slot 78 fixed slot 81 fixed slot 83 fixed slot 90 fixed slot 92 fixed slot 93 fixed slot 94 fixed slot 98 fixed slot 110 fixed slot 113 fixed slot 114 fixed slot 115 fixed slot 117 fixed slot 118 fixed slot 119 fixed slot 120 fixed slot 121 fixed slot 122 fixed slot 125 fixed slot 142 fixed slot 146 fixed slot 147 fixed slot 148 fixed slot 179 fixed slot 182 fixed slot 185 fixed slot 101 Couldn't find a replacement block for slo^[[2~Timeout, server gargamel not responding. [729314.382500] systemd-udevd[25274]: dm-13: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729314.490868] systemd-udevd[25258]: dm-12: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729351.745188] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 is taking a long time [729352.745164] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 is taking a long time [729352.774312] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 is taking a long time [729354.745172] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 is taking a long time [729354.745174] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] is taking longer than 59s to complete [729354.745175] systemd-udevd[25275]: memory229: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26401] is taking longer than 59s to complete [729354.859212] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 is taking a long time [729354.889966] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 is taking a long time [729354.920686] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 is taking a long time [729354.951365] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 is taking a long time [729357.745162] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 is taking a long time [729357.774644] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 is taking a long time [729357.803818] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 is taking a long time [729471.745176] systemd-udevd[2257]: sda: Worker [25268] processing SEQNUM=101206 killed [729472.745177] systemd-udevd[2257]: sdc: Worker [25273] processing SEQNUM=101228 killed [729472.769590] systemd-udevd[2257]: wakeup25: Worker [25247] processing SEQNUM=101312 killed [729474.745171] systemd-udevd[25306]: memory230: Spawned process 'socket:/org/freedesktop/hal/udev_event' [26406] timed out after 2min 59s, killing [729474.745175] systemd-udevd[2257]: memory229: Worker [25275] processing SEQNUM=101499 killed [729474.812102] systemd-udevd[2257]: memory238: Worker [25257] processing SEQNUM=101509 killed [729474.838576] systemd-udevd[2257]: memory230: Worker [25306] processing SEQNUM=101501 killed [729474.864496] systemd-udevd[2257]: memory236: Worker [25246] processing SEQNUM=101507 killed [729474.890429] systemd-udevd[2257]: memory234: Worker [25262] processing SEQNUM=101505 killed [729474.916324] systemd-udevd[2257]: Worker [25275] terminated by signal 9 (KILL) [729477.442170] systemd-udevd[25244]: sdd4: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729477.465202] systemd-udevd[25248]: sdd1: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729477.475194] systemd-udevd[2257]: sde: Worker [25263] processing SEQNUM=101860 killed [729477.531960] systemd-udevd[2257]: md12: Worker [25269] processing SEQNUM=101854 killed [729477.556548] systemd-udevd[2257]: md13: Worker [25340] processing SEQNUM=101855 killed [729494.433414] systemd-udevd[25271]: sdd2: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729512.577884] systemd-udevd[25265]: sdd3: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729572.745171] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 is taking a long time [729692.745172] systemd-udevd[2257]: sdd: Worker [25248] processing SEQNUM=101875 killed [729828.852640] systemd-udevd[25271]: LNXSYSTM:00: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.880567] systemd-udevd[25294]: cstate_core: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.882126] systemd-udevd[25265]: breakpoint: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.885879] systemd-udevd[25244]: cpu: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.885886] systemd-udevd[25339]: kprobe: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.886061] systemd-udevd[25252]: parport0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.886527] systemd-udevd[25258]: cstate_pkg: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.888578] systemd-udevd[25255]: 0000:00:00.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.888587] systemd-udevd[25274]: i915: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729828.905903] systemd-udevd[25264]: msr: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [729829.377862] elogind-daemon[9054]: Watching system buttons on /dev/input/event0 (Power Button) [729829.469868] elogind-daemon[9054]: Watching system buttons on /dev/input/event1 (Power Button) [729829.833865] elogind-daemon[9054]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard) [729830.359362] BTRFS info (device dm-12): devid 1 device path /dev/mapper/vgds2-1Appliances changed to /dev/dm-12 scanned by systemd-udevd (25274) [729889.745171] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 is taking a long time [730009.745168] systemd-udevd[2257]: 4-1.6.3: Worker [25264] processing SEQNUM=102334 killed [730009.770590] systemd-udevd[2257]: Worker [25264] terminated by signal 9 (KILL) [730009.793062] systemd-udevd[2257]: 4-1.6.3: Worker [25264] failed [730009.813461] systemd-udevd[29139]: 4-1.6.3:1.0: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. [732952.473804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 35s! [732952.498625] Showing busy workqueues and worker pools: [732952.514833] workqueue events: flags=0x0 [732952.527430] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732952.527442] pending: cache_reap [732952.527447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [732952.527449] pending: drm_fb_helper_damage_work [drm_kms_helper] [732952.527472] workqueue writeback: flags=0x4a [732952.615583] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [732952.615587] in-flight: 8793:wb_workfn, 8613:wb_workfn [732952.615608] workqueue btrfs-delalloc: flags=0xe [732952.667841] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732952.667843] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [732952.667859] workqueue btrfs-worker: flags=0xe [732952.723431] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732952.723433] in-flight: 25134:btrfs_work_helper [732952.723436] workqueue btrfs-endio-write: flags=0xe [732952.774041] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732952.774043] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [732952.774047] workqueue btrfs-delalloc: flags=0xe [732952.830056] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732952.830058] in-flight: 25030:btrfs_work_helper [732952.830070] workqueue cifsiod: flags=0xc [732952.878011] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732952.878013] in-flight: 9189:cifs_echo_request [cifs] [732952.878056] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=36s workers=2 manager: 17834 [732952.878059] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 30021 [732983.193804] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 66s! [732983.218550] Showing busy workqueues and worker pools: [732983.234716] workqueue events: flags=0x0 [732983.247202] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732983.247205] pending: cache_reap [732983.247209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [732983.247211] pending: drm_fb_helper_damage_work [drm_kms_helper] [732983.247233] workqueue writeback: flags=0x4a [732983.334945] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [732983.334948] in-flight: 8793:wb_workfn, 8613:wb_workfn [732983.334971] workqueue btrfs-delalloc: flags=0xe [732983.386983] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732983.386985] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [732983.387000] workqueue btrfs-worker: flags=0xe [732983.442412] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732983.442415] in-flight: 25134:btrfs_work_helper [732983.442417] workqueue btrfs-endio-write: flags=0xe [732983.492919] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [732983.492921] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [732983.492925] workqueue btrfs-delalloc: flags=0xe [732983.548832] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [732983.548843] in-flight: 25030:btrfs_work_helper [732983.548854] workqueue cifsiod: flags=0xc [732983.596634] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=1/256 refcnt=2 [732983.596637] in-flight: 9189:cifs_echo_request [cifs] [732983.596680] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=66s workers=2 manager: 17834 [732983.596683] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 [733013.909809] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 97s! -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 4:37 ` Marc MERLIN @ 2022-04-07 4:40 ` Marc MERLIN 2022-04-07 7:30 ` Martin Steigerwald 2022-04-07 5:20 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 4:40 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs Damn, it's hosed so badly that even sysrq-o isn't working, never seen that before. [733866.717217] sysrq: Terminate All Tasks [733866.740424] elogind-daemon[9054]: Received signal 15 [TERM] [733868.540292] sysrq: Emergency Remount R/O [733870.249149] sysrq: Emergency Sync [733904.793816] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 988s! [733904.818717] Showing busy workqueues and worker pools: [733904.834771] workqueue events: flags=0x0 [733904.847171] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6 [733904.847174] pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work [733904.847205] workqueue writeback: flags=0x4a [733904.917846] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [733904.917849] in-flight: 8793:wb_workfn, 8613:wb_workfn [733904.917868] workqueue btrfs-delalloc: flags=0xe [733904.969635] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733904.969637] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [733904.969652] workqueue btrfs-worker: flags=0xe [733905.024811] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733905.024813] in-flight: 25134:btrfs_work_helper [733905.024817] workqueue btrfs-endio-write: flags=0xe [733905.075033] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733905.075035] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [733905.075039] workqueue btrfs-delalloc: flags=0xe [733905.130695] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733905.130697] in-flight: 25030:btrfs_work_helper [733905.130709] workqueue cifsiod: flags=0xc [733905.178289] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4 [733905.178291] in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs] [733905.178363] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=988s workers=2 manager: 17834 [733905.178366] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 idle: 2903 [733909.916792] sysrq: Power Off [733966.261816] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x1 nice=0 stuck for 55s! [733966.286477] BUG: workqueue lockup - pool cpus=6 node=0 flags=0x1 nice=0 stuck for 1049s! [733966.311622] Showing busy workqueues and worker pools: [733966.327664] workqueue events: flags=0x0 [733966.340063] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=5/256 refcnt=6 [733966.340067] pending: cache_reap, drm_fb_helper_damage_work [drm_kms_helper], free_work, do_emergency_remount, do_sync_work [733966.340093] pwq 0: cpus=0 node=0 flags=0x1 nice=0 active=3/256 refcnt=4 [733966.340095] in-flight: 17295:do_poweroff [733966.340098] pending: vmstat_shepherd, cache_reap [733966.340106] workqueue writeback: flags=0x4a [733966.461521] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/256 refcnt=4 [733966.461524] in-flight: 8793:wb_workfn, 8613:wb_workfn [733966.461543] workqueue btrfs-delalloc: flags=0xe [733966.513325] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733966.513328] in-flight: 8795:btrfs_work_helper, 29765:btrfs_work_helper [733966.513344] workqueue btrfs-worker: flags=0xe [733966.568487] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733966.568489] in-flight: 25134:btrfs_work_helper [733966.568492] workqueue btrfs-endio-write: flags=0xe [733966.618683] pwq 16: cpus=0-7 flags=0x5 nice=0 active=2/8 refcnt=4 [733966.618685] in-flight: 6405:btrfs_work_helper, 25135:btrfs_work_helper [733966.618698] workqueue btrfs-delalloc: flags=0xe [733966.674361] pwq 16: cpus=0-7 flags=0x5 nice=0 active=1/8 refcnt=3 [733966.674364] in-flight: 25030:btrfs_work_helper [733966.674376] workqueue cifsiod: flags=0xc [733966.721929] pwq 12: cpus=6 node=0 flags=0x1 nice=0 active=2/256 refcnt=4 [733966.721932] in-flight: 9189:cifs_echo_request [cifs], 26186(RESCUER):cifs_resolve_server [cifs] [733966.722002] pool 0: cpus=0 node=0 flags=0x1 nice=0 hung=55s workers=2 manager: 9899 [733966.722006] pool 12: cpus=6 node=0 flags=0x1 nice=0 hung=1049s workers=2 manager: 17834 [733966.722008] pool 16: cpus=0-7 flags=0x5 nice=0 hung=0s workers=11 manager: 6295 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 4:40 ` Marc MERLIN @ 2022-04-07 7:30 ` Martin Steigerwald 0 siblings, 0 replies; 479+ messages in thread From: Martin Steigerwald @ 2022-04-07 7:30 UTC (permalink / raw) To: Josef Bacik, Marc MERLIN; +Cc: Linux BTRFS mailing list Hi! Marc MERLIN - 07.04.22, 06:40:06 CEST: > Damn, it's hosed so badly that even sysrq-o isn't working, never seen > that before. I did not keep statistics, but this might be one of the longest threads on BTRFS mailing list. Good luck with restoring your filesystem or at least recovering all data from it! Best, -- Martin ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 4:37 ` Marc MERLIN 2022-04-07 4:40 ` Marc MERLIN @ 2022-04-07 5:20 ` Marc MERLIN 2022-04-07 16:29 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 5:20 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote: > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo there may have been something else that ate all my memory. I killed shinobi and re-ran your code overnight while dumping memory. I'll have a memory trace running this time while it re-runs overnight. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 5:20 ` Marc MERLIN @ 2022-04-07 16:29 ` Marc MERLIN 2022-04-07 17:07 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 16:29 UTC (permalink / raw) To: Josef Bacik, Martin Steigerwald; +Cc: linux-btrfs On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote: > I did not keep statistics, but this might be one of the longest threads > on BTRFS mailing list. > > Good luck with restoring your filesystem or at least recovering all data > from it! Haha, thanks. All props go to Josef for not giving up :) Even if I don't get my data back without going to the backup, this will have been a useful exercise, however I managed to hose my FS, made for an interesting recovery case. On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote: > On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote: > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo > > there may have been something else that ate all my memory. > I killed shinobi and re-ran your code overnight while dumping > memory. I'll have a memory trace running this time while it re-runs > overnight. Ok, good news is that it was actually shinobi that somehow managed to take enough RAM to OOM hang the system, but in such a way that I had never seen before (sysrq not even able to work anymore, which is worrisome). btrfs-find-root just took another 8G which was enough to tip the glass over, but shinobi (and ultimately the kernel for being unable to kill userspace that takes too much RAM), was at fault. Overnight, it ran stable at 8GB: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-root But the bad news is that it didn't complete, and it's not looking like it's converging. gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1 2>&1 |tee /tmp/outo parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 208 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 130 fixed slot 138 fixed slot 33 Couldn't find a replacement block for slot 179 fixed slot 154 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 129 fixed slot 111 Couldn't find a replacement block for slot 103 Couldn't find a replacement block for slot 192 fixed slot 12 Couldn't find a replacement block for slot 205 fixed slot 24 fixed slot 28 fixed slot 31 fixed slot 38 fixed slot 47 fixed slot 62 fixed slot 75 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 108 Couldn't find a replacement block for slot 115 fixed slot 146 fixed slot 218 fixed slot 288 fixed slot 33 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 176 fixed slot 240 Couldn't find a replacement block for slot 241 fixed slot 34 fixed slot 35 Couldn't find a replacement block for slot 71 Couldn't find a replacement block for slot 140 fixed slot 213 Couldn't find a replacement block for slot 155 fixed slot 210 fixed slot 111 fixed slot 147 fixed slot 150 Couldn't find a replacement block for slot 234 Couldn't find a replacement block for slot 236 fixed slot 240 fixed slot 250 Couldn't find a replacement block for slot 259 fixed slot 269 fixed slot 283 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 48 fixed slot 135 fixed slot 192 fixed slot 206 fixed slot 43 fixed slot 111 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 45 fixed slot 47 fixed slot 212 fixed slot 49 fixed slot 190 fixed slot 4 fixed slot 40 fixed slot 110 fixed slot 227 fixed slot 290 fixed slot 296 fixed slot 297 fixed slot 302 fixed slot 349 fixed slot 357 Couldn't find a replacement block for slot 53 fixed slot 304 Couldn't find a replacement block for slot 328 fixed slot 376 fixed slot 54 fixed slot 125 fixed slot 195 Couldn't find a replacement block for slot 89 fixed slot 177 fixed slot 197 Couldn't find a replacement block for slot 219 fixed slot 233 fixed slot 36 fixed slot 52 fixed slot 61 fixed slot 71 fixed slot 113 fixed slot 119 fixed slot 138 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 200 fixed slot 215 Couldn't find a replacement block for slot 173 fixed slot 315 fixed slot 320 fixed slot 326 fixed slot 328 fixed slot 335 fixed slot 65 fixed slot 32 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 34 Couldn't find a replacement block for slot 37 fixed slot 40 Couldn't find a replacement block for slot 179 fixed slot 75 fixed slot 198 fixed slot 79 fixed slot 16 fixed slot 89 fixed slot 111 fixed slot 152 fixed slot 219 fixed slot 220 fixed slot 43 fixed slot 70 fixed slot 85 fixed slot 102 fixed slot 130 fixed slot 145 fixed slot 146 fixed slot 161 fixed slot 188 fixed slot 202 Couldn't find a replacement block for slot 9 fixed slot 62 fixed slot 122 fixed slot 128 fixed slot 137 fixed slot 142 fixed slot 216 Couldn't find a replacement block for slot 26 fixed slot 143 fixed slot 152 fixed slot 174 fixed slot 185 fixed slot 76 fixed slot 87 fixed slot 26 fixed slot 28 fixed slot 31 fixed slot 33 fixed slot 35 fixed slot 43 fixed slot 46 fixed slot 47 fixed slot 56 fixed slot 62 fixed slot 69 fixed slot 73 fixed slot 76 fixed slot 78 fixed slot 81 fixed slot 83 fixed slot 90 fixed slot 92 fixed slot 93 fixed slot 94 fixed slot 98 fixed slot 110 fixed slot 113 fixed slot 114 fixed slot 115 fixed slot 117 fixed slot 118 fixed slot 119 fixed slot 120 fixed slot 121 fixed slot 122 fixed slot 125 fixed slot 142 fixed slot 146 fixed slot 147 fixed slot 148 fixed slot 179 fixed slot 182 fixed slot 185 fixed slot 101 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 430 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 41 fixed slot 63 fixed slot 33 fixed slot 36 Couldn't find a replacement block for slot 47 fixed slot 52 fixed slot 58 Couldn't find a replacement block for slot 426 Couldn't find a replacement block for slot 64 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 223 fixed slot 76 Couldn't find a replacement block for slot 53 Couldn't find a replacement block for slot 92 fixed slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 173 Couldn't find a replacement block for slot 245 fixed slot 153 Couldn't find a replacement block for slot 115 Couldn't find a replacement block for slot 193 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 241 fixed slot 125 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 185 fixed slot 200 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 204 Couldn't find a replacement block for slot 103 fixed slot 249 Couldn't find a replacement block for slot 250 fixed slot 284 fixed slot 305 fixed slot 128 fixed slot 144 Couldn't find a replacement block for slot 342 fixed slot 164 fixed slot 204 fixed slot 58 fixed slot 132 Couldn't find a replacement block for slot 161 fixed slot 177 fixed slot 209 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 248 Couldn't find a replacement block for slot 2 Couldn't find a replacement block for slot 96 fixed slot 5 fixed slot 114 Couldn't find a replacement block for slot 90 fixed slot 237 fixed slot 19 fixed slot 7 Couldn't find a replacement block for slot 27 fixed slot 48 fixed slot 53 Couldn't find a replacement block for slot 45 Couldn't find a replacement block for slot 225 fixed slot 50 Couldn't find a replacement block for slot 52 fixed slot 57 fixed slot 208 fixed slot 210 fixed slot 78 fixed slot 230 fixed slot 61 Couldn't find a replacement block for slot 81 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 174 fixed slot 96 fixed slot 63 fixed slot 113 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 53 fixed slot 78 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 174 fixed slot 182 Couldn't find a replacement block for slot 183 fixed slot 184 Couldn't find a replacement block for slot 185 fixed slot 199 Couldn't find a replacement block for slot 201 Couldn't find a replacement block for slot 213 fixed slot 215 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 124 fixed slot 125 Couldn't find a replacement block for slot 127 Couldn't find a replacement block for slot 139 fixed slot 140 Couldn't find a replacement block for slot 148 Couldn't find a replacement block for slot 150 Couldn't find a replacement block for slot 156 Couldn't find a replacement block for slot 166 fixed slot 179 fixed slot 186 fixed slot 111 fixed slot 0 Couldn't find a replacement block for slot 9 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 44 fixed slot 49 Couldn't find a replacement block for slot 67 fixed slot 68 fixed slot 69 fixed slot 70 fixed slot 76 fixed slot 80 fixed slot 83 Couldn't find a replacement block for slot 86 fixed slot 113 fixed slot 140 fixed slot 143 fixed slot 144 fixed slot 184 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 194 Couldn't find a replacement block for slot 195 Couldn't find a replacement block for slot 196 fixed slot 197 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 199 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 205 Couldn't find a replacement block for slot 215 Couldn't find a replacement block for slot 239 fixed slot 245 fixed slot 247 fixed slot 248 fixed slot 260 fixed slot 20 fixed slot 28 fixed slot 44 fixed slot 45 Couldn't find a replacement block for slot 48 Couldn't find a replacement block for slot 57 Couldn't find a replacement block for slot 61 Couldn't find a replacement block for slot 69 fixed slot 82 Couldn't find a replacement block for slot 86 fixed slot 90 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 97 fixed slot 110 fixed slot 116 fixed slot 121 fixed slot 138 fixed slot 150 fixed slot 158 fixed slot 160 fixed slot 162 Couldn't find a replacement block for slot 168 Couldn't find a replacement block for slot 169 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 173 fixed slot 175 fixed slot 179 fixed slot 181 Couldn't find a replacement block for slot 182 fixed slot 185 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 188 fixed slot 193 fixed slot 205 fixed slot 215 Couldn't find a replacement block for slot 224 fixed slot 233 fixed slot 237 fixed slot 243 fixed slot 256 Couldn't find a replacement block for slot 11 Couldn't find a replacement block for slot 15 fixed slot 21 fixed slot 26 Couldn't find a replacement block for slot 32 fixed slot 50 fixed slot 55 Couldn't find a replacement block for slot 57 fixed slot 112 fixed slot 113 fixed slot 114 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 127 fixed slot 131 Couldn't find a replacement block for slot 140 fixed slot 146 Couldn't find a replacement block for slot 153 fixed slot 154 Couldn't find a replacement block for slot 159 fixed slot 163 fixed slot 164 fixed slot 177 fixed slot 202 fixed slot 214 Couldn't find a replacement block for slot 220 fixed slot 230 Couldn't find a replacement block for slot 246 fixed slot 251 Couldn't find a replacement block for slot 259 Couldn't find a replacement block for slot 261 Couldn't find a replacement block for slot 262 Couldn't find a replacement block for slot 263 fixed slot 264 fixed slot 266 fixed slot 114 fixed slot 5 fixed slot 24 fixed slot 72 fixed slot 85 fixed slot 93 fixed slot 98 Couldn't find a replacement block for slot 111 fixed slot 128 fixed slot 130 Couldn't find a replacement block for slot 131 fixed slot 133 fixed slot 145 fixed slot 146 Couldn't find a replacement block for slot 75 Couldn't find a replacement block for slot 76 Couldn't find a replacement block for slot 77 Couldn't find a replacement block for slot 83 Couldn't find a replacement block for slot 141 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 37 Couldn't find a replacement block for slot 38 fixed slot 111 fixed slot 118 fixed slot 119 fixed slot 206 Couldn't find a replacement block for slot 226 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 229 Couldn't find a replacement block for slot 231 Couldn't find a replacement block for slot 374 Couldn't find a replacement block for slot 376 fixed slot 377 Couldn't find a replacement block for slot 60 fixed slot 122 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 180 fixed slot 126 Couldn't find a replacement block for slot 119 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 129 Couldn't find a replacement block for slot 130 Couldn't find a replacement block for slot 131 Couldn't find a replacement block for slot 132 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 141 fixed slot 129 fixed slot 134 Couldn't find a replacement block for slot 100 fixed slot 101 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 35 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 367 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 244 Couldn't find a replacement block for slot 308 Couldn't find a replacement block for slot 309 Couldn't find a replacement block for slot 310 Couldn't find a replacement block for slot 311 Couldn't find a replacement block for slot 312 Couldn't find a replacement block for slot 326 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 49 fixed slot 13 Couldn't find a replacement block for slot 80 Couldn't find a replacement block for slot 145 Couldn't find a replacement block for slot 76 fixed slot 143 Couldn't find a replacement block for slot 28 fixed slot 159 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 126 fixed slot 148 Couldn't find a replacement block for slot 150 fixed slot 155 fixed slot 161 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 204 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 211 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 220 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 199 fixed slot 161 Couldn't find a replacement block for slot 108 Couldn't find a replacement block for slot 110 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 159 Couldn't find a replacement block for slot 175 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 205 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 263 fixed slot 169 Couldn't find a replacement block for slot 18 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 49 Couldn't find a replacement block for slot 51 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 138 fixed slot 157 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 207 fixed slot 210 fixed slot 212 fixed slot 213 fixed slot 216 fixed slot 219 Couldn't find a replacement block for slot 235 fixed slot 242 fixed slot 243 fixed slot 244 Couldn't find a replacement block for slot 245 Couldn't find a replacement block for slot 269 fixed slot 170 Couldn't find a replacement block for slot 28 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 250 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 257 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 279 Couldn't find a replacement block for slot 280 Couldn't find a replacement block for slot 281 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 9 fixed slot 37 Couldn't find a replacement block for slot 40 Couldn't find a replacement block for slot 68 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 82 Couldn't find a replacement block for slot 90 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 114 fixed slot 115 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 118 fixed slot 127 Couldn't find a replacement block for slot 135 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 179 Couldn't find a replacement block for slot 181 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 219 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 222 Couldn't find a replacement block for slot 235 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 242 Couldn't find a replacement block for slot 264 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 267 fixed slot 16 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 41 fixed slot 44 Couldn't find a replacement block for slot 52 fixed slot 55 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 97 Couldn't find a replacement block for slot 98 fixed slot 100 Couldn't find a replacement block for slot 118 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 125 fixed slot 150 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 191 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 207 Couldn't find a replacement block for slot 208 fixed slot 211 fixed slot 213 Couldn't find a replacement block for slot 218 fixed slot 221 Couldn't find a replacement block for slot 231 fixed slot 248 Couldn't find a replacement block for slot 255 Couldn't find a replacement block for slot 256 fixed slot 173 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 5 fixed slot 28 Couldn't find a replacement block for slot 29 Couldn't find a replacement block for slot 30 fixed slot 31 fixed slot 39 Couldn't find a replacement block for slot 41 fixed slot 55 Couldn't find a replacement block for slot 78 Couldn't find a replacement block for slot 84 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 99 fixed slot 102 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 126 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 209 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 2(... still running) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 16:29 ` Marc MERLIN @ 2022-04-07 17:07 ` Josef Bacik 2022-04-07 19:11 ` Martin Steigerwald 2022-04-07 22:09 ` Josef Bacik 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-07 17:07 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs On Thu, Apr 7, 2022 at 12:29 PM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote: > > I did not keep statistics, but this might be one of the longest threads > > on BTRFS mailing list. > > > > Good luck with restoring your filesystem or at least recovering all data > > from it! > > Haha, thanks. All props go to Josef for not giving up :) > Even if I don't get my data back without going to the backup, this will > have been a useful exercise, however I managed to hose my FS, made for > an interesting recovery case. > > On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote: > > On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote: > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo > > > > there may have been something else that ate all my memory. > > I killed shinobi and re-ran your code overnight while dumping > > memory. I'll have a memory trace running this time while it re-runs > > overnight. > > Ok, good news is that it was actually shinobi that somehow managed to > take enough RAM to OOM hang the system, but in such a way that I had > never seen before (sysrq not even able to work anymore, which is > worrisome). > btrfs-find-root just took another 8G which was enough to tip the glass > over, but shinobi (and ultimately the kernel for being unable to kill > userspace that takes too much RAM), was at fault. > > Overnight, it ran stable at 8GB: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-root > > But the bad news is that it didn't complete, and it's not looking like it's > converging. > Ok let it keep going, I have an idea of how to make this less awful but also not use all the memory. I'm going to start writing this up now, if I finish before your run finishes you can switch to the new shit and see how that goes. I've got meetings until 4pm my time (it's 1pm my time), so you're not going to hear from me for a while. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 17:07 ` Josef Bacik @ 2022-04-07 19:11 ` Martin Steigerwald 2022-04-07 22:09 ` Josef Bacik 1 sibling, 0 replies; 479+ messages in thread From: Martin Steigerwald @ 2022-04-07 19:11 UTC (permalink / raw) To: Marc MERLIN, Josef Bacik; +Cc: linux-btrfs Josef Bacik - 07.04.22, 19:07:14 CEST: > Ok let it keep going, I have an idea of how to make this less awful > but also not use all the memory. I'm going to start writing this up > now, if I finish before your run finishes you can switch to the new > shit and see how that goes. I've got meetings until 4pm my time (it's > 1pm my time), so you're not going to hear from me for a while. You guys are having fun with that, it seems. -- Martin ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 17:07 ` Josef Bacik 2022-04-07 19:11 ` Martin Steigerwald @ 2022-04-07 22:09 ` Josef Bacik 2022-04-07 22:33 ` Marc MERLIN 2022-04-08 10:22 ` Marc MERLIN 1 sibling, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-07 22:09 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs On Thu, Apr 7, 2022 at 1:07 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On Thu, Apr 7, 2022 at 12:29 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Thu, Apr 07, 2022 at 09:30:41AM +0200, Martin Steigerwald wrote: > > > I did not keep statistics, but this might be one of the longest threads > > > on BTRFS mailing list. > > > > > > Good luck with restoring your filesystem or at least recovering all data > > > from it! > > > > Haha, thanks. All props go to Josef for not giving up :) > > Even if I don't get my data back without going to the backup, this will > > have been a useful exercise, however I managed to hose my FS, made for > > an interesting recovery case. > > > > On Wed, Apr 06, 2022 at 10:20:22PM -0700, Marc MERLIN wrote: > > > On Wed, Apr 06, 2022 at 09:37:17PM -0700, Marc MERLIN wrote: > > > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-find-root -o 1 /dev/mapper/dshelf1a 2>&1 |tee /tmp/outo > > > > > > there may have been something else that ate all my memory. > > > I killed shinobi and re-ran your code overnight while dumping > > > memory. I'll have a memory trace running this time while it re-runs > > > overnight. > > > > Ok, good news is that it was actually shinobi that somehow managed to > > take enough RAM to OOM hang the system, but in such a way that I had > > never seen before (sysrq not even able to work anymore, which is > > worrisome). > > btrfs-find-root just took another 8G which was enough to tip the glass > > over, but shinobi (and ultimately the kernel for being unable to kill > > userspace that takes too much RAM), was at fault. > > > > Overnight, it ran stable at 8GB: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 31873 root 20 0 8024m 7.8g 0 R 99.9 25.2 667:57.24 btrfs-find-root > > > > But the bad news is that it didn't complete, and it's not looking like it's > > converging. > > > > Ok let it keep going, I have an idea of how to make this less awful > but also not use all the memory. I'm going to start writing this up > now, if I finish before your run finishes you can switch to the new > shit and see how that goes. I've got meetings until 4pm my time (it's > 1pm my time), so you're not going to hear from me for a while. > Thanks, > Just following up on this, I've got hungry kids, I'm about halfway through the new shit. Depending on how much help kids need with homework I may have this done later tonight, or it'll be tomorrow morning. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 22:09 ` Josef Bacik @ 2022-04-07 22:33 ` Marc MERLIN 2022-04-08 10:22 ` Marc MERLIN 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-07 22:33 UTC (permalink / raw) To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote: > Just following up on this, I've got hungry kids, I'm about halfway > through the new shit. Depending on how much help kids need with > homework I may have this done later tonight, or it'll be tomorrow > morning. Thanks, it's stlil churning along and looping. I have no idea if it's converging, or not. parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x56055fb76600 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x56055fb76600 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 62 fixed slot 88 fixed slot 235 fixed slot 237 Couldn't find a replacement block for slot 238 Couldn't find a replacement block for slot 243 fixed slot 33 fixed slot 35 fixed slot 32 fixed slot 133 fixed slot 136 fixed slot 207 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 484 Couldn't find a replacement block for slot 76 fixed slot 229 fixed slot 172 fixed slot 174 fixed slot 36 Couldn't find a replacement block for slot 20 fixed slot 24 Couldn't find a replacement block for slot 28 fixed slot 115 fixed slot 148 fixed slot 9 Couldn't find a replacement block for slot 318 fixed slot 245 fixed slot 1 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 134 fixed slot 159 fixed slot 170 fixed slot 204 fixed slot 149 fixed slot 185 fixed slot 257 fixed slot 46 fixed slot 60 fixed slot 91 fixed slot 92 fixed slot 95 fixed slot 247 fixed slot 105 Couldn't find a replacement block for slot 109 fixed slot 124 fixed slot 191 fixed slot 244 fixed slot 9 fixed slot 57 Couldn't find a replacement block for slot 166 fixed slot 163 fixed slot 78 fixed slot 179 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 218 fixed slot 26 fixed slot 75 Couldn't find a replacement block for slot 68 fixed slot 56 fixed slot 60 Couldn't find a replacement block for slot 231 fixed slot 141 fixed slot 151 fixed slot 156 fixed slot 159 fixed slot 163 fixed slot 171 fixed slot 200 fixed slot 5 Couldn't find a replacement block for slot 157 fixed slot 224 fixed slot 44 fixed slot 217 fixed slot 122 fixed slot 59 Couldn't find a replacement block for slot 120 fixed slot 181 fixed slot 187 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 220 fixed slot 60 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 192 fixed slot 9 fixed slot 7 Couldn't find a replacement block for slot 226 fixed slot 24 fixed slot 49 fixed slot 22 Couldn't find a replacement block for slot 186 fixed slot 149 fixed slot 150 Couldn't find a replacement block for slot 42 fixed slot 218 fixed slot 223 fixed slot 228 fixed slot 230 fixed slot 233 fixed slot 234 fixed slot 240 fixed slot 155 fixed slot 24 fixed slot 30 fixed slot 376 fixed slot 94 fixed slot 104 fixed slot 197 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 191 fixed slot 139 fixed slot 128 Couldn't find a replacement block for slot 143 fixed slot 16 fixed slot 17 fixed slot 114 fixed slot 28 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 83 fixed slot 86 fixed slot 92 fixed slot 149 fixed slot 117 Couldn't find a replacement block for slot 31 fixed slot 188 Couldn't find a replacement block for slot 102 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 79 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 21 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 276 Couldn't find a replacement block for slot 282 Couldn't find a replacement block for slot 283 Couldn't find a replacement block for slot 284 Couldn't find a replacement block for slot 285 Couldn't find a replacement block for slot 286 Couldn't find a replacement block for slot 287 Couldn't find a replacement block for slot 289 Couldn't find a replacement block for slot 290 Couldn't find a replacement block for slot 291 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 231 fixed slot 208 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 208 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 130 fixed slot 138 fixed slot 33 Couldn't find a replacement block for slot 179 fixed slot 154 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 129 fixed slot 111 Couldn't find a replacement block for slot 103 Couldn't find a replacement block for slot 192 fixed slot 12 Couldn't find a replacement block for slot 205 fixed slot 24 fixed slot 28 fixed slot 31 fixed slot 38 fixed slot 47 fixed slot 62 fixed slot 75 fixed slot 86 fixed slot 89 fixed slot 93 fixed slot 108 Couldn't find a replacement block for slot 115 fixed slot 146 fixed slot 218 fixed slot 288 fixed slot 33 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 176 fixed slot 240 Couldn't find a replacement block for slot 241 fixed slot 34 fixed slot 35 Couldn't find a replacement block for slot 71 Couldn't find a replacement block for slot 140 fixed slot 213 Couldn't find a replacement block for slot 155 fixed slot 210 fixed slot 111 fixed slot 147 fixed slot 150 Couldn't find a replacement block for slot 234 Couldn't find a replacement block for slot 236 fixed slot 240 fixed slot 250 Couldn't find a replacement block for slot 259 fixed slot 269 fixed slot 283 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 48 fixed slot 135 fixed slot 192 fixed slot 206 fixed slot 43 fixed slot 111 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 45 fixed slot 47 fixed slot 212 fixed slot 49 fixed slot 190 fixed slot 4 fixed slot 40 fixed slot 110 fixed slot 227 fixed slot 290 fixed slot 296 fixed slot 297 fixed slot 302 fixed slot 349 fixed slot 357 Couldn't find a replacement block for slot 53 fixed slot 304 Couldn't find a replacement block for slot 328 fixed slot 376 fixed slot 54 fixed slot 125 fixed slot 195 Couldn't find a replacement block for slot 89 fixed slot 177 fixed slot 197 Couldn't find a replacement block for slot 219 fixed slot 233 fixed slot 36 fixed slot 52 fixed slot 61 fixed slot 71 fixed slot 113 fixed slot 119 fixed slot 138 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 200 fixed slot 215 Couldn't find a replacement block for slot 173 fixed slot 315 fixed slot 320 fixed slot 326 fixed slot 328 fixed slot 335 fixed slot 65 fixed slot 32 Couldn't find a replacement block for slot 33 Couldn't find a replacement block for slot 34 Couldn't find a replacement block for slot 37 fixed slot 40 Couldn't find a replacement block for slot 179 fixed slot 75 fixed slot 198 fixed slot 79 fixed slot 16 fixed slot 89 fixed slot 111 fixed slot 152 fixed slot 219 fixed slot 220 fixed slot 43 fixed slot 70 fixed slot 85 fixed slot 102 fixed slot 130 fixed slot 145 fixed slot 146 fixed slot 161 fixed slot 188 fixed slot 202 Couldn't find a replacement block for slot 9 fixed slot 62 fixed slot 122 fixed slot 128 fixed slot 137 fixed slot 142 fixed slot 216 Couldn't find a replacement block for slot 26 fixed slot 143 fixed slot 152 fixed slot 174 fixed slot 185 fixed slot 76 fixed slot 87 fixed slot 26 fixed slot 28 fixed slot 31 fixed slot 33 fixed slot 35 fixed slot 43 fixed slot 46 fixed slot 47 fixed slot 56 fixed slot 62 fixed slot 69 fixed slot 73 fixed slot 76 fixed slot 78 fixed slot 81 fixed slot 83 fixed slot 90 fixed slot 92 fixed slot 93 fixed slot 94 fixed slot 98 fixed slot 110 fixed slot 113 fixed slot 114 fixed slot 115 fixed slot 117 fixed slot 118 fixed slot 119 fixed slot 120 fixed slot 121 fixed slot 122 fixed slot 125 fixed slot 142 fixed slot 146 fixed slot 147 fixed slot 148 fixed slot 179 fixed slot 182 fixed slot 185 fixed slot 101 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 89 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 430 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 41 fixed slot 63 fixed slot 33 fixed slot 36 Couldn't find a replacement block for slot 47 fixed slot 52 fixed slot 58 Couldn't find a replacement block for slot 426 Couldn't find a replacement block for slot 64 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 223 fixed slot 76 Couldn't find a replacement block for slot 53 Couldn't find a replacement block for slot 92 fixed slot 228 Couldn't find a replacement block for slot 109 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 173 Couldn't find a replacement block for slot 245 fixed slot 153 Couldn't find a replacement block for slot 115 Couldn't find a replacement block for slot 193 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 241 fixed slot 125 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 185 fixed slot 200 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 204 Couldn't find a replacement block for slot 103 fixed slot 249 Couldn't find a replacement block for slot 250 fixed slot 284 fixed slot 305 fixed slot 128 fixed slot 144 Couldn't find a replacement block for slot 342 fixed slot 164 fixed slot 204 fixed slot 58 fixed slot 132 Couldn't find a replacement block for slot 161 fixed slot 177 fixed slot 209 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 248 Couldn't find a replacement block for slot 2 Couldn't find a replacement block for slot 96 fixed slot 5 fixed slot 114 Couldn't find a replacement block for slot 90 fixed slot 237 fixed slot 19 fixed slot 7 Couldn't find a replacement block for slot 27 fixed slot 48 fixed slot 53 Couldn't find a replacement block for slot 45 Couldn't find a replacement block for slot 225 fixed slot 50 Couldn't find a replacement block for slot 52 fixed slot 57 fixed slot 208 fixed slot 210 fixed slot 78 fixed slot 230 fixed slot 61 Couldn't find a replacement block for slot 81 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 174 fixed slot 96 fixed slot 63 fixed slot 113 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 53 fixed slot 78 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 174 fixed slot 182 Couldn't find a replacement block for slot 183 fixed slot 184 Couldn't find a replacement block for slot 185 fixed slot 199 Couldn't find a replacement block for slot 201 Couldn't find a replacement block for slot 213 fixed slot 215 Couldn't find a replacement block for slot 106 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 124 fixed slot 125 Couldn't find a replacement block for slot 127 Couldn't find a replacement block for slot 139 fixed slot 140 Couldn't find a replacement block for slot 148 Couldn't find a replacement block for slot 150 Couldn't find a replacement block for slot 156 Couldn't find a replacement block for slot 166 fixed slot 179 fixed slot 186 fixed slot 111 fixed slot 0 Couldn't find a replacement block for slot 9 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 44 fixed slot 49 Couldn't find a replacement block for slot 67 fixed slot 68 fixed slot 69 fixed slot 70 fixed slot 76 fixed slot 80 fixed slot 83 Couldn't find a replacement block for slot 86 fixed slot 113 fixed slot 140 fixed slot 143 fixed slot 144 fixed slot 184 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 194 Couldn't find a replacement block for slot 195 Couldn't find a replacement block for slot 196 fixed slot 197 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 199 fixed slot 202 Couldn't find a replacement block for slot 203 fixed slot 205 Couldn't find a replacement block for slot 215 Couldn't find a replacement block for slot 239 fixed slot 245 fixed slot 247 fixed slot 248 fixed slot 260 fixed slot 20 fixed slot 28 fixed slot 44 fixed slot 45 Couldn't find a replacement block for slot 48 Couldn't find a replacement block for slot 57 Couldn't find a replacement block for slot 61 Couldn't find a replacement block for slot 69 fixed slot 82 Couldn't find a replacement block for slot 86 fixed slot 90 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 97 fixed slot 110 fixed slot 116 fixed slot 121 fixed slot 138 fixed slot 150 fixed slot 158 fixed slot 160 fixed slot 162 Couldn't find a replacement block for slot 168 Couldn't find a replacement block for slot 169 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 173 fixed slot 175 fixed slot 179 fixed slot 181 Couldn't find a replacement block for slot 182 fixed slot 185 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 188 fixed slot 193 fixed slot 205 fixed slot 215 Couldn't find a replacement block for slot 224 fixed slot 233 fixed slot 237 fixed slot 243 fixed slot 256 Couldn't find a replacement block for slot 11 Couldn't find a replacement block for slot 15 fixed slot 21 fixed slot 26 Couldn't find a replacement block for slot 32 fixed slot 50 fixed slot 55 Couldn't find a replacement block for slot 57 fixed slot 112 fixed slot 113 fixed slot 114 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 127 fixed slot 131 Couldn't find a replacement block for slot 140 fixed slot 146 Couldn't find a replacement block for slot 153 fixed slot 154 Couldn't find a replacement block for slot 159 fixed slot 163 fixed slot 164 fixed slot 177 fixed slot 202 fixed slot 214 Couldn't find a replacement block for slot 220 fixed slot 230 Couldn't find a replacement block for slot 246 fixed slot 251 Couldn't find a replacement block for slot 259 Couldn't find a replacement block for slot 261 Couldn't find a replacement block for slot 262 Couldn't find a replacement block for slot 263 fixed slot 264 fixed slot 266 fixed slot 114 fixed slot 5 fixed slot 24 fixed slot 72 fixed slot 85 fixed slot 93 fixed slot 98 Couldn't find a replacement block for slot 111 fixed slot 128 fixed slot 130 Couldn't find a replacement block for slot 131 fixed slot 133 fixed slot 145 fixed slot 146 Couldn't find a replacement block for slot 75 Couldn't find a replacement block for slot 76 Couldn't find a replacement block for slot 77 Couldn't find a replacement block for slot 83 Couldn't find a replacement block for slot 141 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 37 Couldn't find a replacement block for slot 38 fixed slot 111 fixed slot 118 fixed slot 119 fixed slot 206 Couldn't find a replacement block for slot 226 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 229 Couldn't find a replacement block for slot 231 Couldn't find a replacement block for slot 374 Couldn't find a replacement block for slot 376 fixed slot 377 Couldn't find a replacement block for slot 60 fixed slot 122 Couldn't find a replacement block for slot 92 fixed slot 96 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 171 Couldn't find a replacement block for slot 172 Couldn't find a replacement block for slot 178 Couldn't find a replacement block for slot 180 fixed slot 126 Couldn't find a replacement block for slot 119 Couldn't find a replacement block for slot 120 Couldn't find a replacement block for slot 121 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 129 Couldn't find a replacement block for slot 130 Couldn't find a replacement block for slot 131 Couldn't find a replacement block for slot 132 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 141 fixed slot 129 fixed slot 134 Couldn't find a replacement block for slot 100 fixed slot 101 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 35 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 367 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 244 Couldn't find a replacement block for slot 308 Couldn't find a replacement block for slot 309 Couldn't find a replacement block for slot 310 Couldn't find a replacement block for slot 311 Couldn't find a replacement block for slot 312 Couldn't find a replacement block for slot 326 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 49 fixed slot 13 Couldn't find a replacement block for slot 80 Couldn't find a replacement block for slot 145 Couldn't find a replacement block for slot 76 fixed slot 143 Couldn't find a replacement block for slot 28 fixed slot 159 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 126 fixed slot 148 Couldn't find a replacement block for slot 150 fixed slot 155 fixed slot 161 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 204 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 211 Couldn't find a replacement block for slot 214 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 220 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 243 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 199 fixed slot 161 Couldn't find a replacement block for slot 108 Couldn't find a replacement block for slot 110 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 159 Couldn't find a replacement block for slot 175 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 205 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 263 fixed slot 169 Couldn't find a replacement block for slot 18 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 49 Couldn't find a replacement block for slot 51 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 124 Couldn't find a replacement block for slot 138 fixed slot 157 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 170 Couldn't find a replacement block for slot 207 fixed slot 210 fixed slot 212 fixed slot 213 fixed slot 216 fixed slot 219 Couldn't find a replacement block for slot 235 fixed slot 242 fixed slot 243 fixed slot 244 Couldn't find a replacement block for slot 245 Couldn't find a replacement block for slot 269 fixed slot 170 Couldn't find a replacement block for slot 28 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 250 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 257 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 279 Couldn't find a replacement block for slot 280 Couldn't find a replacement block for slot 281 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 9 fixed slot 37 Couldn't find a replacement block for slot 40 Couldn't find a replacement block for slot 68 Couldn't find a replacement block for slot 74 Couldn't find a replacement block for slot 82 Couldn't find a replacement block for slot 90 Couldn't find a replacement block for slot 113 Couldn't find a replacement block for slot 114 fixed slot 115 Couldn't find a replacement block for slot 116 Couldn't find a replacement block for slot 117 Couldn't find a replacement block for slot 118 fixed slot 127 Couldn't find a replacement block for slot 135 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 167 Couldn't find a replacement block for slot 179 Couldn't find a replacement block for slot 181 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 219 Couldn't find a replacement block for slot 221 Couldn't find a replacement block for slot 222 Couldn't find a replacement block for slot 235 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 242 Couldn't find a replacement block for slot 264 Couldn't find a replacement block for slot 265 Couldn't find a replacement block for slot 267 fixed slot 16 Couldn't find a replacement block for slot 23 Couldn't find a replacement block for slot 26 Couldn't find a replacement block for slot 41 fixed slot 44 Couldn't find a replacement block for slot 52 fixed slot 55 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 97 Couldn't find a replacement block for slot 98 fixed slot 100 Couldn't find a replacement block for slot 118 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 125 fixed slot 150 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 191 Couldn't find a replacement block for slot 198 Couldn't find a replacement block for slot 207 Couldn't find a replacement block for slot 208 fixed slot 211 fixed slot 213 Couldn't find a replacement block for slot 218 fixed slot 221 Couldn't find a replacement block for slot 231 fixed slot 248 Couldn't find a replacement block for slot 255 Couldn't find a replacement block for slot 256 fixed slot 173 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 5 fixed slot 28 Couldn't find a replacement block for slot 29 Couldn't find a replacement block for slot 30 fixed slot 31 fixed slot 39 Couldn't find a replacement block for slot 41 fixed slot 55 Couldn't find a replacement block for slot 78 Couldn't find a replacement block for slot 84 Couldn't find a replacement block for slot 98 Couldn't find a replacement block for slot 99 fixed slot 102 Couldn't find a replacement block for slot 114 Couldn't find a replacement block for slot 126 Couldn't find a replacement block for slot 128 Couldn't find a replacement block for slot 138 Couldn't find a replacement block for slot 139 Couldn't find a replacement block for slot 140 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 209 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 211 Couldn't find a replacement block for slot 213 Couldn't find a replacement block for slot 217 Couldn't find a replacement block for slot 222 Couldn't find a replacement block for slot 223 Couldn't find a replacement block for slot 224 Couldn't find a replacement block for slot 225 Couldn't find a replacement block for slot 226 Couldn't find a replacement block for slot 227 Couldn't find a replacement block for slot 229 Couldn't find a replacement block for slot 230 Couldn't find a replacement block for slot 231 Couldn't find a replacement block for slot 233 Couldn't find a replacement block for slot 252 Couldn't find a replacement block for slot 253 Couldn't find a replacement block for slot 254 fixed slot 256 Couldn't find a replacement block for slot 11 fixed slot 28 Couldn't find a replacement block for slot 33 fixed slot 39 Couldn't find a replacement block for slot 63 Couldn't find a replacement block for slot 72 fixed slot 119 Couldn't find a replacement block for slot 146 fixed slot 156 Couldn't find a replacement block for slot 179 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 193 Couldn't find a replacement block for slot 194 Couldn't find a replacement block for slot 197 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 250 Couldn't find a replacement block for slot 252 Couldn't find a replacement block for slot 5 Couldn't find a replacement block for slot 13 Couldn't find a replacement block for slot 18 Couldn't find a replacement block for slot 31 Couldn't find a replacement block for slot 53 Couldn't find a replacement block for slot 56 Couldn't find a replacement block for slot 57 Couldn't find a replacement block for slot 62 Couldn't find a replacement block for slot 84 Couldn't find a replacement block for slot 85 Couldn't find a replacement block for slot 86 Couldn't find a replacement block for slot 87 fixed slot 99 Couldn't find a replacement block for slot 100 Couldn't find a replacement block for slot 101 fixed slot 108 fixed slot 141 fixed slot 155 fixed slot 178 Couldn't find a replacement block for slot 181 Couldn't find a replacement block for slot 197 fixed slot 177 fixed slot 193 Couldn't find a replacement block for slot 194 Couldn't find a replacement block for slot 195 fixed slot 226 Couldn't find a replacement block for slot 273 fixed slot 302 Couldn't find a replacement block for slot 69 Couldn't find a replacement block for slot 70 Couldn't find a replacement block for slot 143 Couldn't find a replacement block for slot 144 fixed slot 204 Couldn't find a replacement block for slot 129 Couldn't find a replacement block for slot 130 Couldn't find a replacement block for slot 254 Couldn't find a replacement block for slot 255 Couldn't find a replacement block for slot 214 fixed slot 248 fixed slot 220 fixed slot 204 Couldn't find a replacement block for slot 205 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 207 Couldn't find a replacement block for slot 210 Couldn't find a replacement block for slot 211 Couldn't find a replacement block for slot 215 Couldn't find a replacement block for slot 220 fixed slot 234 Couldn't find a replacement block for slot 235 fixed slot 236 Couldn't find a replacement block for slot 247 Couldn't find a replacement block for slot 248 Couldn't find a replacement block for slot 267 fixed slot 276 fixed slot 288 Couldn't find a replacement block for slot 317 fixed slot 330 Couldn't find a replacement block for slot 337 Couldn't find a replacement block for slot 345 Couldn't find a replacement block for slot 346 fixed slot 361 Couldn't find a replacement block for slot 365 Couldn't find a replacement block for slot 366 Couldn't find a replacement block for slot 369 Couldn't find a replacement block for slot 370 Couldn't find a replacement block for slot 372 Couldn't find a replacement block for slot 373 fixed slot 379 Couldn't find a replacement block for slot 380 Couldn't find a replacement block for slot 381 Couldn't find a replacement block for slot 382 Couldn't find a replacement block for slot 387 Couldn't find a replacement block for slot 388 Couldn't find a replacement block for slot 389 Couldn't find a replacement block for slot 390 Couldn't find a replacement block for slot 394 fixed slot 395 Couldn't find a replacement block for slot 396 fixed slot 407 Couldn't find a replacement block for slot 408 Couldn't find a replacement block for slot 410 Couldn't find a replacement block for slot 411 Couldn't find a replacement block for slot 412 Couldn't find a replacement block for slot 425 Couldn't find a replacement block for slot 426 Couldn't find a replacement block for slot 427 Couldn't find a replacement block for slot 428 fixed slot 221 fixed slot 0 Couldn't find a replacement block for slot 1 Couldn't find a replacement block for slot 2 Couldn't find a replacement block for slot 8 Couldn't find a replacement block for slot 13 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 16 Couldn't find a replacement block for slot 17 Couldn't find a replacement block for slot 20 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 38 Couldn't find a replacement block for slot 48 Couldn't find a replacement block for slot 49 Couldn't find a replacement block for slot 50 Couldn't find a replacement block for slot 54 Couldn't find a replacement block for slot 55 Couldn't find a replacement block for slot 56 fixed slot 67 fixed slot 83 Couldn't find a replacement block for slot 84 Couldn't find a replacement block for slot 119 Couldn't find a replacement block for slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 126 fixed slot 206 Couldn't find a replacement block for slot 207 Couldn't find a replacement block for slot 220 fixed slot 224 fixed slot 143 fixed slot 158 Couldn't find a replacement block for slot 159 Couldn't find a replacement block for slot 160 Couldn't find a replacement block for slot 199 Couldn't find a replacement block for slot 204 Couldn't find a replacement block for slot 227 Couldn't find a replacement block for slot 228 fixed slot 229 fixed slot 225 fixed slot 95 fixed slot 104 Couldn't find a replacement block for slot 105 Couldn't find a replacement block for slot 125 Couldn't find a replacement block for slot 131 Couldn't find a replacement block for slot 141 Couldn't find a replacement block for slot 163 Couldn't find a replacement block for slot 165 fixed slot 171 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 184 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 203 fixed slot 205 Couldn't find a replacement block for slot 213 Couldn't find a replacement block for slot 215 fixed slot 221 Couldn't find a replacement block for slot 225 fixed slot 226 fixed slot 231 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 242 fixed slot 243 fixed slot 254 Couldn't find a replacement block for slot 255 Couldn't find a replacement block for slot 257 Couldn't find a replacement block for slot 258 Couldn't find a replacement block for slot 229 fixed slot 233 Couldn't find a replacement block for slot 147 Couldn't find a replacement block for slot 153 Couldn't find a replacement block for slot 154 fixed slot 155 Couldn't find a replacement block for slot 182 Couldn't find a replacement block for slot 183 Couldn't find a replacement block for slot 187 Couldn't find a replacement block for slot 188 Couldn't find a replacement block for slot 190 Couldn't find a replacement block for slot 191 Couldn't find a replacement block for slot 192 Couldn't find a replacement block for slot 193 Couldn't find a replacement block for slot 201 Couldn't find a replacement block for slot 228 Couldn't find a replacement block for slot 233 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 240 Couldn't find a replacement block for slot 242 Couldn't find a replacement block for slot 1 Couldn't find a replacement block for slot 2 Couldn't find a replacement block for slot 3 Couldn't find a replacement block for slot 4 Couldn't find a replacement block for slot 12 Couldn't find a replacement block for slot 13 Couldn't find a replacement block for slot 14 Couldn't find a replacement block for slot 19 Couldn't find a replacement block for slot 22 fixed slot 45 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 47 Couldn't find a replacement block for slot 48 Couldn't find a replacement block for slot 54 Couldn't find a replacement block for slot 59 Couldn't find a replacement block for slot 61 Couldn't find a replacement block for slot 76 Couldn't find a replacement block for slot 78 fixed slot 81 Couldn't find a replacement block for slot 85 Couldn't find a replacement block for slot 87 fixed slot 90 Couldn't find a replacement block for slot 91 Couldn't find a replacement block for slot 92 fixed slot 94 Couldn't find a replacement block for slot 104 fixed slot 170 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 208 fixed slot 210 fixed slot 218 Couldn't find a replacement block for slot 256 Couldn't find a replacement block for slot 271 fixed slot 235 fixed slot 14 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 16 Couldn't find a replacement block for slot 17 Couldn't find a replacement block for slot 24 Couldn't find a replacement block for slot 25 Couldn't find a replacement block for slot 233 fixed slot 245 Couldn't find a replacement block for slot 246 Couldn't find a replacement block for slot 247 Couldn't find a replacement block for slot 248 Couldn't find a replacement block for slot 249 Couldn't find a replacement block for slot 285 fixed slot 286 Couldn't find a replacement block for slot 287 fixed slot 49 Couldn't find a replacement block for slot 53 fixed slot 63 Couldn't find a replacement block for slot 67 Couldn't find a replacement block for slot 68 Couldn't find a replacement block for slot 70 fixed slot 78 Couldn't find a replacement block for slot 101 Couldn't find a replacement block for slot 102 fixed slot 105 Couldn't find a replacement block for slot 114 fixed slot 122 Couldn't find a replacement block for slot 123 Couldn't find a replacement block for slot 131 Couldn't find a replacement block for slot 133 Couldn't find a replacement block for slot 142 Couldn't find a replacement block for slot 144 Couldn't find a replacement block for slot 154 Couldn't find a replacement block for slot 155 fixed slot 179 fixed slot 196 fixed slot 199 Couldn't find a replacement block for slot 225 Couldn't find a replacement block for slot 259 Couldn't find a replacement block for slot 260 Couldn't find a replacement block for slot 261 fixed slot 29 Couldn't find a replacement block for slot 136 Couldn't find a replacement block for slot 206 Couldn't find a replacement block for slot 209 Couldn't find a replacement block for slot 429 fixed slot 289 Couldn't find a replacement block for slot 195 fixed slot 172 Couldn't find a replacement block for slot 173 Couldn't find a replacement block for slot 295 fixed slot 4 fixed slot 7 Couldn't find a replacement block for slot 11 Couldn't find a replacement block for slot 14 Couldn't find a replacement block for slot 15 fixed slot 16 Couldn't find a replacement block for slot 35 Couldn't find a replacement block for slot 36 Couldn't find a replacement block for slot 38 Couldn't find a replacement block for slot 40 Couldn't find a replacement block for slot 41 Couldn't find a replacement block for slot 46 Couldn't find a replacement block for slot 47 Couldn't find a replacement block for slot 48 fixed slot 49 Couldn't find a replacement block for slot 154 Couldn't find a replacement block for slot 253 fixed slot 7 Couldn't find a replacement block for slot 189 Couldn't find a replacement block for slot 191 Couldn't find a replacement block for slot 62 Couldn't find a replacement block for slot 63 Couldn't find a replacement block for slot 168 fixed slot 172 Couldn't find a replacement block for slot 180 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 186 Couldn't find a replacement block for slot 203 Couldn't find a replacement block for slot 214 fixed slot 267 fixed slot 268 Couldn't find a replacement block for slot 269 Couldn't find a replacement block for slot 270 Couldn't find a replacement block for slot 280 fixed slot 281 Couldn't find a replacement block for slot 282 fixed slot 13 Couldn't find a replacement block for slot 30 Couldn't find a replacement block for slot 31 Couldn't find a replacement block for slot 69 fixed slot 70 Couldn't find a replacement block for slot 90 Couldn't find a replacement block for slot 111 Couldn't find a replacement block for slot 130 Couldn't find a replacement block for slot 146 Couldn't find a replacement block for slot 147 Couldn't find a replacement block for slot 156 fixed slot 173 Couldn't find a replacement block for slot 174 Couldn't find a replacement block for slot 175 Couldn't find a replacement block for slot 185 Couldn't find a replacement block for slot 188 Couldn't find a replacement block for slot 209 Couldn't find a replacement block for slot 220 Couldn't find a replacement block for slot 221 fixed slot 229 Couldn't find a replacement block for slot 230 Couldn't find a replacement block for slot 231 Couldn't find a replacement block for slot 233 Couldn't find a replacement block for slot 237 Couldn't find a replacement block for slot 239 Couldn't find a replacement block for slot 241 Couldn't find a replacement block for slot 244 fixed slot 3 fixed slot 4 Couldn't find a replacement block for slot 5 Couldn't find a replacement block for slot 7 Couldn't find a replacement block for slot 8 Couldn't find a replacement block for slot 9 Couldn't find a replacement block for slot 13 Couldn't find a replacement block for slot 14 Couldn't find a replacement block for slot 15 Couldn't find a replacement block for slot 16 Couldn't find a replacement block for slot 17 Couldn't find a replacement block for slot 18 fixed slot 25 Couldn't find a replacement block for slot 28 Couldn't find a replacement block for slot 40 fixed slot 41 Couldn't find a replacement block for slot 42 Couldn't find a replacement block for slot 43 Couldn't find a replacement block for slot 44 Couldn't find a replacement block for slot 47 Couldn't find a replacement block for slot 55 fixed slot 60 Couldn't find a replacement block for slot 61 Couldn't find a replacement block for slot 63 Couldn't find a replacement block for slot 66 Couldn't find a replacement block for slot 73 Couldn't find a replacement block for slot 74 fixed slot 78 Couldn't find a replacement block for slot 80 Couldn't find a replacement block for slot 82 Couldn't find a replacement block for slot 92 Couldn't find a replacement block for slot 93 Couldn't find a replacement block for slot 99 fixed slot 103 Couldn't find a replacement block for slot 104 Couldn't find a replacement block for slot 106 fixed slot 109 Couldn't find a replacement block for slot 110 fixed slot 111 fixed slot 120 Couldn't find a replacement block for slot 121 fixed slot 125 Couldn't find a replacement block for slot 126 Couldn't find a replacement block for slot 127 Couldn't find a replacement block for slot 128 fixed slot 301 Couldn't find a replacement block for slot 19 fixed slot 22 fixed slot 28 Couldn't find a replacement block for slot 29 Couldn't find a replacement block for slot 30 fixed slot 31 Couldn't find a replacement block for slot 38 Couldn't find a replacement block for slot 39 fixed slot 48 Couldn't find a replacement block for slot 51 fixed slot 53 Couldn't find a replacement block for slot 55 Couldn't find a replacement block for slot 56 fixed slot 62 fixed slot 77 Couldn't find a replacement block for slot 78 Couldn't find a replacement bl (waiting for more output) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-07 22:09 ` Josef Bacik 2022-04-07 22:33 ` Marc MERLIN @ 2022-04-08 10:22 ` Marc MERLIN 2022-04-08 10:23 ` Josef Bacik 1 sibling, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-08 10:22 UTC (permalink / raw) To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote: > Just following up on this, I've got hungry kids, I'm about halfway > through the new shit. Depending on how much help kids need with > homework I may have this done later tonight, or it'll be tomorrow > morning. Thanks, It finished after more than 24H, but that didn't seem to be enough, quite. Hopefully we're getting closer, though :) Couldn't find a replacement block for slot 361 Couldn't find a replacement block for slot 363 fixed slot 372 Couldn't find a replacement block for slot 374 Couldn't find a replacement block for slot 375 Couldn't find a replacement block for slot 403 Couldn't find a replacement block for slot 404 Couldn't find a replacement block for slot 405 Couldn't find a replacement block for slot 406 Couldn't find a replacement block for slot 407 Couldn't find a replacement block for slot 408 Couldn't find a replacement block for slot 409 Couldn't find a replacement block for slot 413 Couldn't find a replacement block for slot 414 Couldn't find a replacement block for slot 415 Couldn't find a replacement block for slot 416 Couldn't find a replacement block for slot 417 Couldn't find a replacement block for slot 418 Couldn't find a replacement block for slot 419 Couldn't find a replacement block for slot 420 Couldn't find a replacement block for slot 421 Couldn't find a replacement block for slot 422 Couldn't find a replacement block for slot 423 Couldn't find a replacement block for slot 424 Couldn't find a replacement block for slot 427 Couldn't find a replacement block for slot 437 Couldn't find a replacement block for slot 438 Couldn't find a replacement block for slot 443 Couldn't find a replacement block for slot 444 Couldn't find a replacement block for slot 445 fixed slot 446 fixed slot 447 gargamel:/var/local/src/btrfs-progs-josefbacik# gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs check --init-extent-tree --repair /dev/mapper/dshelf1 enabling repair mode WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 parent transid verify failed on 22216704 wanted 1600938 found 1602177 FS_INFO IS 0x5641ceb95fd0 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 Couldn't find the last root for 8 parent transid verify failed on 15645251010560 wanted 1602089 found 1602297 FS_INFO AFTER IS 0x5641ceb95fd0 Checking filesystem on /dev/mapper/dshelf1 UUID: 96539b8c-ccc9-47bf-9e6c-29305890941e Creating a new extent tree checksum verify failed on 15645248897024 wanted 0xce96f609 found 0x2d1b5ea6 checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0 checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718 checksum verify failed on 15645365747712 wanted 0x3bdf1829 found 0xdbfcbd25 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 checksum verify failed on 15645365747712 wanted 0x4f4dcd5c found 0xa1d136c1 bad tree block 15645365747712, bad level, 240 > 8 Error reading tree block owner 7 error pinning down used bytes ERROR: commit_root already set when starting transaction extent buffer leak: start 13577814573056 len 16384 gargamel:/var/local/src/btrfs-progs-josefbacik# gargamel:/var/local/src/btrfs-progs-josefbacik# mount -o ro /dev/mapper/dshelf1 /mnt/mnt mount: /mnt/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/dshelf1, missing codepage or helper program, or other error. gargamel:/var/local/src/btrfs-progs-josefbacik# dmtail [106691.309851] BTRFS info (device dm-1): disk space caching is enabled [106691.363587] BTRFS info (device dm-1): has skinny extents [106691.394712] verify_parent_transid: 4 callbacks suppressed [106691.394716] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [106691.443339] BTRFS error (device dm-1): parent transid verify failed on 22216704 wanted 1600938 found 1602177 [106691.474616] BTRFS error (device dm-1): failed to read chunk tree: -5 [106691.501647] BTRFS error (device dm-1): open_ctree failed [106691.519613] systemd-udevd[31783]: btrfs-19: Process 'socket:/org/freedesktop/hal/udev_event' failed with exit code 1. -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-08 10:22 ` Marc MERLIN @ 2022-04-08 10:23 ` Josef Bacik 2022-04-08 20:09 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-08 10:23 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs On Fri, Apr 8, 2022 at 6:22 AM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote: > > Just following up on this, I've got hungry kids, I'm about halfway > > through the new shit. Depending on how much help kids need with > > homework I may have this done later tonight, or it'll be tomorrow > > morning. Thanks, > > It finished after more than 24H, but that didn't seem to be enough, quite. > Hopefully we're getting closer, though :) > Yup we're getting there, I'm about 90% done with the new stuff, and that'll delete block pointers that we can't find good matches for. Also it'll cache the results of the disk wide search, so it'll have to do it once per root instead of for every bad block we find, which will drastically cut down on the runtime. I'll let you know when it's ready, figure another hour or so. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-08 10:23 ` Josef Bacik @ 2022-04-08 20:09 ` Josef Bacik 2022-04-11 1:37 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-08 20:09 UTC (permalink / raw) To: Marc MERLIN; +Cc: Martin Steigerwald, linux-btrfs On Fri, Apr 8, 2022 at 6:23 AM Josef Bacik <josef@toxicpanda.com> wrote: > > On Fri, Apr 8, 2022 at 6:22 AM Marc MERLIN <marc@merlins.org> wrote: > > > > On Thu, Apr 07, 2022 at 06:09:53PM -0400, Josef Bacik wrote: > > > Just following up on this, I've got hungry kids, I'm about halfway > > > through the new shit. Depending on how much help kids need with > > > homework I may have this done later tonight, or it'll be tomorrow > > > morning. Thanks, > > > > It finished after more than 24H, but that didn't seem to be enough, quite. > > Hopefully we're getting closer, though :) > > > > Yup we're getting there, I'm about 90% done with the new stuff, and > that'll delete block pointers that we can't find good matches for. > > Also it'll cache the results of the disk wide search, so it'll have to > do it once per root instead of for every bad block we find, which will > drastically cut down on the runtime. I'll let you know when it's > ready, figure another hour or so. Thanks, > Course the last 10% takes the longest, but I corrupted a local file system and ran it to shake out all the stupid bugs. Go ahead and pull and run ./btrfs rescue tree-recover /dev/whatever and then *hopefully* you can just run btrfs check --repair, but it may fail out with a "btrfs unable to find ref byte", which is what I was seeing locally. I'm fixing that but it's tricky and may be a while. If you hit that then go ahead and use --init-extent-tree --init-csum-tree and let it ride. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-08 20:09 ` Josef Bacik @ 2022-04-11 1:37 ` Marc MERLIN 0 siblings, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-11 1:37 UTC (permalink / raw) To: Josef Bacik; +Cc: Martin Steigerwald, linux-btrfs On Fri, Apr 08, 2022 at 04:09:34PM -0400, Josef Bacik wrote: > Course the last 10% takes the longest, but I corrupted a local file > system and ran it to shake out all the stupid bugs. Go ahead and pull > and run > > ./btrfs rescue tree-recover /dev/whatever > > and then *hopefully* you can just run btrfs check --repair, but it may > fail out with a "btrfs unable to find ref byte", which is what I was > seeing locally. I'm fixing that but it's tricky and may be a while. > If you hit that then go ahead and use --init-extent-tree > --init-csum-tree and let it ride. Thanks, Sorry for the delay, too many things this weekend and I didn't do Email. gargamel:/var/local/src/btrfs-progs-josefbacik# git log | head -5 commit 580e1b4db52a0ec98661bec7310450d84ea1f441 Author: Josef Bacik <josef@toxicpanda.com> Date: Fri Apr 8 16:07:48 2022 -0400 handle fstree's differently gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 Ignoring transid failure FS_INFO IS 0x558174d9cbc0 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Couldn't find the last root for 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x558174d9cbc0 deleting slot 94 in block 21069824 FS_INFO IS 0x558174e1f430 Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Ignoring transid failure Couldn't find the last root for 8 ERROR: failed to read block groups: Input/output error FS_INFO AFTER IS 0x558174e1f430 Found completely clean tree for 3 in backup root, replacing ERROR: Still have update after a repair loop, bailing Tree recover failed Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 20:24 ` Josef Bacik 2022-04-05 20:37 ` Marc MERLIN @ 2022-04-05 23:51 ` Zygo Blaxell 2022-04-06 0:08 ` Marc MERLIN 1 sibling, 1 reply; 479+ messages in thread From: Zygo Blaxell @ 2022-04-05 23:51 UTC (permalink / raw) To: Josef Bacik; +Cc: Marc MERLIN, linux-btrfs On Tue, Apr 05, 2022 at 04:24:16PM -0400, Josef Bacik wrote: > On Tue, Apr 5, 2022 at 4:08 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 05, 2022 at 04:05:05PM -0400, Josef Bacik wrote: > > > Well it's still the same, and this thing is 20mib into your fs so IDK > > > how it would be screwing up now. Can you do > > > > > > ./btrfs inspect-internal dump-tree -b 21069824 > > > > > > and see what that spits out? IDK why it would suddenly start > > > complaining about your chunk root. Thanks, > > > > Thanks for your patience and sticking with me > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs inspect-internal dump-tree -b 21069824 /dev/mapper/dshelf1a >/dev/null > > Ok well that worked, which means it found the chunk tree fine, I'm There's two copies of the chunk tree in dup metadata. Maybe one of them is bad? It seems surprising to me if that's the case. I didn't think the userspace read code did anything like the current->pid % 2 dance, and even if it did, I'd expect it to show up before now. Other possibilites include irreproducible reads coming from the bcache layer if that's still active (e.g. if it was on mdadm raid1, and the raid1 mirrors were out of sync and didn't know it). Hopefully not, because it would make this...challenging. > going to chalk this up to it just fucking with me and ignore it for > now. I pushed some changes for the find root thing, can you re-run > > ./btrfs-find-root -o 1 /dev/whatever > > it should be less noisy and spit out one line at the end, "Found tree > root at blah blah". Thanks, > > Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-05 23:51 ` Zygo Blaxell @ 2022-04-06 0:08 ` Marc MERLIN 2022-04-06 1:40 ` Zygo Blaxell 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 0:08 UTC (permalink / raw) To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs On Tue, Apr 05, 2022 at 07:51:29PM -0400, Zygo Blaxell wrote: > Other possibilites include irreproducible reads coming from the bcache > layer if that's still active (e.g. if it was on mdadm raid1, and the bcache is still active, correct. mdadm5 array is running with 4 drives out of 5. I'm not sure if I full understand the tree root issue we've been having. We know from the dump that we have these backups backup_tree_root: 13577814573056 gen: 1602089 level: 1 backup_tree_root: 13577775284224 gen: 1602086 level: 1 backup_tree_root: 13577821667328 gen: 1602087 level: 1 backup_tree_root: 13577779511296 gen: 1602088 level: 1 but they didn't seem to work, or are we sure that none of them don't? Then we did a custom scan, but it found one of the 4 backup_tree_root we already had, so that doesn't help, correct? Found tree root at 13577814573056 gen 1602089 level 1 Was the expectation that there are more backup_tree_root hiding somewhere and we've been trying to find them? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 0:08 ` Marc MERLIN @ 2022-04-06 1:40 ` Zygo Blaxell 2022-04-06 4:09 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Zygo Blaxell @ 2022-04-06 1:40 UTC (permalink / raw) To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs On Tue, Apr 05, 2022 at 05:08:44PM -0700, Marc MERLIN wrote: > On Tue, Apr 05, 2022 at 07:51:29PM -0400, Zygo Blaxell wrote: > > Other possibilites include irreproducible reads coming from the bcache > > layer if that's still active (e.g. if it was on mdadm raid1, and the > > bcache is still active, correct. > mdadm5 array is running with 4 drives out of 5. > > I'm not sure if I full understand the tree root issue we've been having. Based on the history, I'd expect the filesystem is missing some number of tree nodes, from a few dozen to thousands, depending on how many writes were dropped after the 2nd drive failure before it was detected. Since the array was also degraded at that time, with 4 drives in raid5, there's 3 data drives, and if one of them was offline then we'd have a 2/3 success rate reading metadata blocks and 1/3 garbage. That's definitely in the "we need to write new software to recover from this" territory. Normally, I'd expect that once we dig through a few layers of simple dropped write blocks, we'll start hitting metadata pages with bad csums and trashed contents, since the parity blocks will be garbage in the raid5 stripes where the writes were lost. One important data point against this theory is that we have not seen a csum failure yet, so maybe this is a different (possibly better) scenario. Possibly some of the lost writes on the raid5 are still stored in the bcache, so there's few or no garbage blocks (though reading the array through the cache might evict the last copy of usable data and make some damage permanent--you might want to make a backup copy of the cache device). Backup roots only work if writes are dropped only in the most recent transaction, maybe two, because only these trees are guaranteed to be intact on disk. After that, previously occupied pages are fair game for new write allocations, and old metadata will be lost. Unlike other filesystems, btrfs never writes metadata in the same place twice, so when a write is dropped, there isn't an old copy of the data still available at the location of the dropped write--that location contains some completely unrelated piece of the metadata tree whose current version now lives at some other location. Later tree updates will overwrite old copies of the updated page, destroying the data in the affected page forever. Essentially there will be a set of metadata pages where you have two versions of different ages, and another set of metadata pages where you have zero versions, and (hopefully) most of the other pages are intact. > We know from the dump that we have these backups > backup_tree_root: 13577814573056 gen: 1602089 level: 1 > backup_tree_root: 13577775284224 gen: 1602086 level: 1 > backup_tree_root: 13577821667328 gen: 1602087 level: 1 > backup_tree_root: 13577779511296 gen: 1602088 level: 1 > > but they didn't seem to work, or are we sure that none of them don't? The root pointers are just pointers--having a pointer to a tree root isn't going to help if the trees were never written at the referenced location. Recursively, if you have a tree node but you don't have the next-level nodes the previous-level node points to, then the root isn't useful because large parts of its tree are unreachable. If you're missing a leaf node then whatever data was in the leaf is gone. Once you've found a root node, you can try walking over the entire tree to see if it's all there (e.g. with btrfs ins dump-tree -t N), but if anything's missing in any part of the tree, then that tree root can't be used as-is. > Then we did a custom scan, but it found one of the 4 backup_tree_root we > already had, so that doesn't help, correct? > Found tree root at 13577814573056 gen 1602089 level 1 > > Was the expectation that there are more backup_tree_root hiding > somewhere and we've been trying to find them? If we have a superblock, the chunk tree, and a subvol tree, we can drop all the other trees and rebuild them (bonus points if the csum tree survived, then we can verify all the data was recovered correctly; otherwise, we can read all the files and make a new csum tree but it won't detect any data corruption that might have happened in degraded mode). This is roughly what 'btrfs check --init-extent-tree' does (though due to implementation details it has a few extra dependencies that might get in the way) and you can find subvol roots with btrfs-find-root. If we don't have any intact subvol trees (or the subvol trees we really want aren't intact), then we can't recover this way. Instead we'd have to scrape the disk looking for metadata leaf nodes, and try to reinsert those into a new tree structure. The trick here is that we'll have the duplicated and inconsitent nodes and we won't have some nodes at all, and we'll have to make sense of those (or pass it to the existing btrfs check and hope it can cope with them). I'm guessing that a simplified version of this is what Josef is building at this point, or will be building soon if we aren't extremely lucky and find an intact subvol tree. After building an intact subvol tree (even with a few garbage items in it as long as check can handle them) we can go back to the --init-extent-tree step and rebuild the rest of the filesystem. > Thanks, > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ > ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 1:40 ` Zygo Blaxell @ 2022-04-06 4:09 ` Marc MERLIN 2022-04-06 18:07 ` Zygo Blaxell 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 4:09 UTC (permalink / raw) To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs On Tue, Apr 05, 2022 at 09:40:51PM -0400, Zygo Blaxell wrote: > Based on the history, I'd expect the filesystem is missing some number > of tree nodes, from a few dozen to thousands, depending on how many > writes were dropped after the 2nd drive failure before it was detected. > Since the array was also degraded at that time, with 4 drives in raid5, > there's 3 data drives, and if one of them was offline then we'd have a 2/3 > success rate reading metadata blocks and 1/3 garbage. That's definitely > in the "we need to write new software to recover from this" territory. I think your conclusion is correct. I'm very dismayed that the filesystem didn't go read only right away. Hell, the mdadm block device should have done read only as soon as it lost more than one drive. Why were any writes allowed once more than one drive was missing? Let's look at this for a second: Mar 28 02:28:11 gargamel kernel: [1512988.446844] sd 6:1:8:0: Device offlined - not ready after error recovery Mar 28 02:28:11 gargamel kernel: [1512988.475270] sd 6:1:8:0: rejecting I/O to offline device Mar 28 02:28:11 gargamel kernel: [1512988.491531] blk_update_request: I/O error, dev sdi, sector 261928312 op 0x0:(READ) flags 0x84700 phys_seg 42 prio class 0 Mar 28 02:28:11 gargamel kernel: [1512988.525073] blk_update_request: I/O error, dev sdi, sector 261928824 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0 Mar 28 02:28:12 gargamel kernel: [1512988.579667] blk_update_request: I/O error, dev sdi, sector 261927936 op 0x0:(READ) flags 0x80700 phys_seg 47 prio class 0 (..) Mar 28 02:28:12 gargamel kernel: [1512988.615910] md: super_written gets error=10 Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Disk failure on sdi1, disabling device. Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Operation continuing on 4 devices. Mar 28 02:28:21 gargamel kernel: [1512998.170192] usb 2-1.6-port1: disabled by hub (EMI?), re-enabling... Mar 28 02:28:21 gargamel kernel: [1512998.240404] print_req_error: 134 callbacks suppressed Mar 28 02:28:21 gargamel kernel: [1512998.240406] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x807 00 phys_seg 1 prio class 0 Mar 28 02:28:21 gargamel kernel: [1512998.243415] ftdi_sio 2-1.6.1:1.0: device disconnected Mar 28 02:28:21 gargamel kernel: [1512998.341221] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x0 p (...) Mar 28 02:28:22 gargamel kernel: [1512998.716351] blk_update_request: I/O error, dev sdi, sector 2058 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Mar 28 02:28:22 gargamel kernel: [1512998.716362] blk_update_request: I/O error, dev sdi, sector 2059 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 Ok, one drive died, but raid5 continues degraded. md7 : active raid5 sdi1[5](F) sdo1[7] sdg1[6] sdj1[3] sdh1[1] 23441561600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_] bitmap: 0/44 pages [0KB], 65536KB chunk not sure what these were: Mar 29 00:00:08 gargamel kernel: [1590505.415665] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore Mar 29 00:00:09 gargamel kernel: [1590505.866094] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore 9H later a 2nd drive dies just when I'm replacing the failed one: Mar 29 09:30:12 gargamel kernel: [1624709.301830] sd 6:1:5:0: [sdh] tag#523 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s Mar 29 09:30:12 gargamel kernel: [1624709.331812] sd 6:1:5:0: [sdh] tag#523 CDB: Read(16) 88 00 00 00 00 00 00 26 3f d0 00 00 00 18 00 00 Mar 29 09:30:12 gargamel kernel: [1624709.359459] blk_update_request: I/O error, dev sdh, sector 2506704 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0 Mar 29 09:30:12 gargamel kernel: [1624709.359465] md/raid:md7: read error not correctable (sector 2504656 on sdh1). Mar 29 09:30:12 gargamel kernel: [1624709.359471] md/raid:md7: read error not correctable (sector 2504664 on sdh1). Mar 29 09:30:12 gargamel kernel: [1624709.359472] md/raid:md7: read error not correctable (sector 2504672 on sdh1). Mar 29 09:30:12 gargamel kernel: [1624709.359486] md/raid:md7: read error not correctable (sector 2504656 on sdh1). Mar 29 09:30:12 gargamel kernel: [1624709.455886] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.681637] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.695785] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.717624] md/raid:md7: read error not correctable (sector 2504664 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624709.739746] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.757206] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.770348] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.790298] md/raid:md7: read error not correctable (sector 2504672 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624709.812546] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.825856] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.839006] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.866656] md/raid:md7: read error not correctable (sector 24815552 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624709.898796] md/raid:md7: read error not correctable (sector 24815552 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624709.921135] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.934315] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.947500] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624709.988589] md/raid:md7: read error not correctable (sector 1763985936 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624710.036204] md/raid:md7: read error not correctable (sector 1763985936 on sdh1). Mar 29 09:30:13 gargamel kernel: [1624710.059121] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624710.088858] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624710.102026] md: super_written gets error=10 Mar 29 09:30:13 gargamel kernel: [1624710.158830] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.096055] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:36:37 gargamel kernel: [1625094.122910] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 1, flush 0, corrupt 0, gen 0 Mar 29 09:36:37 gargamel kernel: [1625094.153249] md/raid:md7: read error not correctable (sector 6562801616 on sdh1). Mar 29 09:36:37 gargamel kernel: [1625094.176011] md/raid:md7: read error not correctable (sector 6562801624 on sdh1). Mar 29 09:36:37 gargamel kernel: [1625094.223351] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.250628] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.263726] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.276989] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.290121] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.303267] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.325084] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.342083] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.355206] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.368394] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.383304] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.396423] md: super_written gets error=10 Mar 29 09:36:37 gargamel kernel: [1625094.409498] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:36:37 gargamel kernel: [1625094.436355] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 2, flush 0, corrupt 0, gen 0 Mar 29 09:36:37 gargamel kernel: [1625094.466729] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:36:37 gargamel kernel: [1625094.493600] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 3, flush 0, corrupt 0, gen 0 Mar 29 09:36:37 gargamel kernel: [1625094.523998] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:36:38 gargamel kernel: [1625094.550938] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.066422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.093309] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 5, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.124768] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.151651] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 6, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.182803] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.209677] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 7, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.239972] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.266862] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 8, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.297234] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.324094] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 9, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.354422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.381286] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 10, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.411926] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.438770] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 11, flush 0, corrupt 0, gen 0 Mar 29 09:37:34 gargamel kernel: [1625151.469361] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:34 gargamel kernel: [1625151.496269] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 12, flush 0, corrupt 0, gen 0 Mar 29 09:37:35 gargamel kernel: [1625151.527455] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:35 gargamel kernel: [1625151.554360] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 13, flush 0, corrupt 0, gen 0 Mar 29 09:37:35 gargamel kernel: [1625151.584963] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable Mar 29 09:37:35 gargamel kernel: [1625151.611842] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 14, flush 0, corrupt 0, gen 0 24 seconds are lost before btrfs notices anything, and then it seems to continue without going read only for another full minute before things stopped. Why would the raid5 not go read only immediately after 2 drives are lost? > Normally, I'd expect that once we dig through a few layers of simple > dropped write blocks, we'll start hitting metadata pages with bad csums > and trashed contents, since the parity blocks will be garbage in the raid5 > stripes where the writes were lost. One important data point against > this theory is that we have not seen a csum failure yet, so maybe this > is a different (possibly better) scenario. Possibly some of the lost > writes on the raid5 are still stored in the bcache, so there's few or no > garbage blocks (though reading the array through the cache might evict > the last copy of usable data and make some damage permanent--you might > want to make a backup copy of the cache device). Interesting, thanks. > Backup roots only work if writes are dropped only in the most recent > transaction, maybe two, because only these trees are guaranteed to be > intact on disk. After that, previously occupied pages are fair game > for new write allocations, and old metadata will be lost. Unlike other > filesystems, btrfs never writes metadata in the same place twice, so when > a write is dropped, there isn't an old copy of the data still available at > the location of the dropped write--that location contains some completely > unrelated piece of the metadata tree whose current version now lives > at some other location. Later tree updates will overwrite old copies > of the updated page, destroying the data in the affected page forever. > Essentially there will be a set of metadata pages where you have two > versions of different ages, and another set of metadata pages where you > have zero versions, and (hopefully) most of the other pages are intact. I see. It's definitely a lot more complex and much more likely to break when some amount of recent writes get lost/corrupted. > If we have a superblock, the chunk tree, and a subvol tree, we can > drop all the other trees and rebuild them (bonus points if the csum > tree survived, then we can verify all the data was recovered correctly; > otherwise, we can read all the files and make a new csum tree but it won't > detect any data corruption that might have happened in degraded mode). > This is roughly what 'btrfs check --init-extent-tree' does (though due > to implementation details it has a few extra dependencies that might > get in the way) and you can find subvol roots with btrfs-find-root. Got it, thanks. > If we don't have any intact subvol trees (or the subvol trees we really > want aren't intact), then we can't recover this way. Instead we'd have > to scrape the disk looking for metadata leaf nodes, and try to reinsert > those into a new tree structure. The trick here is that we'll have the > duplicated and inconsitent nodes and we won't have some nodes at all, > and we'll have to make sense of those (or pass it to the existing btrfs > check and hope it can cope with them). I'm guessing that a simplified > version of this is what Josef is building at this point, or will be > building soon if we aren't extremely lucky and find an intact subvol tree. > After building an intact subvol tree (even with a few garbage items in it > as long as check can handle them) we can go back to the --init-extent-tree > step and rebuild the rest of the filesystem. I see. In that case, I'm still happy to help, to help improve the tools, but if I'm looking at some amount of non trivial loss/corruption, at soe point I'll go back to backups, since they'll be more intact than this now damaged filesystem. Thanks for the detailed answers. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 4:09 ` Marc MERLIN @ 2022-04-06 18:07 ` Zygo Blaxell 2022-04-06 19:13 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Zygo Blaxell @ 2022-04-06 18:07 UTC (permalink / raw) To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs On Tue, Apr 05, 2022 at 09:09:13PM -0700, Marc MERLIN wrote: > On Tue, Apr 05, 2022 at 09:40:51PM -0400, Zygo Blaxell wrote: > > Based on the history, I'd expect the filesystem is missing some number > > of tree nodes, from a few dozen to thousands, depending on how many > > writes were dropped after the 2nd drive failure before it was detected. > > Since the array was also degraded at that time, with 4 drives in raid5, > > there's 3 data drives, and if one of them was offline then we'd have a 2/3 > > success rate reading metadata blocks and 1/3 garbage. That's definitely > > in the "we need to write new software to recover from this" territory. > > I think your conclusion is correct. I'm very dismayed that the > filesystem didn't go read only right away. > Hell, the mdadm block device should have done read only as soon as it > lost more than one drive. > Why were any writes allowed once more than one drive was missing? > Let's look at this for a second: > Mar 28 02:28:11 gargamel kernel: [1512988.446844] sd 6:1:8:0: Device offlined - not ready after error recovery > Mar 28 02:28:11 gargamel kernel: [1512988.475270] sd 6:1:8:0: rejecting I/O to offline device > Mar 28 02:28:11 gargamel kernel: [1512988.491531] blk_update_request: I/O error, dev sdi, sector 261928312 op 0x0:(READ) flags 0x84700 > phys_seg 42 prio class 0 > Mar 28 02:28:11 gargamel kernel: [1512988.525073] blk_update_request: I/O error, dev sdi, sector 261928824 op 0x0:(READ) flags 0x80700 > phys_seg 5 prio class 0 > Mar 28 02:28:12 gargamel kernel: [1512988.579667] blk_update_request: I/O error, dev sdi, sector 261927936 op 0x0:(READ) flags 0x80700 > phys_seg 47 prio class 0 > (..) > Mar 28 02:28:12 gargamel kernel: [1512988.615910] md: super_written gets error=10 > Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Disk failure on sdi1, disabling device. > Mar 28 02:28:12 gargamel kernel: [1512988.619241] md/raid:md7: Operation continuing on 4 devices. > Mar 28 02:28:21 gargamel kernel: [1512998.170192] usb 2-1.6-port1: disabled by hub (EMI?), re-enabling... > Mar 28 02:28:21 gargamel kernel: [1512998.240404] print_req_error: 134 callbacks suppressed > Mar 28 02:28:21 gargamel kernel: [1512998.240406] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x807 > 00 phys_seg 1 prio class 0 > Mar 28 02:28:21 gargamel kernel: [1512998.243415] ftdi_sio 2-1.6.1:1.0: device disconnected > Mar 28 02:28:21 gargamel kernel: [1512998.341221] blk_update_request: I/O error, dev sdi, sector 11721044992 op 0x0:(READ) flags 0x0 p > (...) > Mar 28 02:28:22 gargamel kernel: [1512998.716351] blk_update_request: I/O error, dev sdi, sector 2058 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > Mar 28 02:28:22 gargamel kernel: [1512998.716362] blk_update_request: I/O error, dev sdi, sector 2059 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > Ok, one drive died, but raid5 continues degraded. > md7 : active raid5 sdi1[5](F) sdo1[7] sdg1[6] sdj1[3] sdh1[1] > 23441561600 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_] > bitmap: 0/44 pages [0KB], 65536KB chunk > > not sure what these were: > Mar 29 00:00:08 gargamel kernel: [1590505.415665] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore > Mar 29 00:00:09 gargamel kernel: [1590505.866094] bcache: bch_count_backing_io_errors() md7: Read-ahead I/O failed on backing device, ignore Readahead can fail for a number of reasons that aren't real problems, e.g. if it's necessary to swap or evict pages to do the readahead, the kernel doesn't bother, and fails the readahead operation instead. Another common case is lower layers of block IO stack, where a readahead request can't be satisfied because of some geometry constraint specific to one device in a multi-device LV. e.g. a drive is no longer present so it can't receive block IO requests. Normally readahead errors are ignored. If there's a real problem, the application will issue a normal read later on and the error will be reported then. Or the readahead might have been entirely speculative, but the app never tries to read the data at all. > 9H later a 2nd drive dies just when I'm replacing the failed one: > Mar 29 09:30:12 gargamel kernel: [1624709.301830] sd 6:1:5:0: [sdh] tag#523 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=2s > Mar 29 09:30:12 gargamel kernel: [1624709.331812] sd 6:1:5:0: [sdh] tag#523 CDB: Read(16) 88 00 00 00 00 00 00 26 3f d0 00 00 00 18 00 00 > Mar 29 09:30:12 gargamel kernel: [1624709.359459] blk_update_request: I/O error, dev sdh, sector 2506704 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0 > Mar 29 09:30:12 gargamel kernel: [1624709.359465] md/raid:md7: read error not correctable (sector 2504656 on sdh1). > Mar 29 09:30:12 gargamel kernel: [1624709.359471] md/raid:md7: read error not correctable (sector 2504664 on sdh1). > Mar 29 09:30:12 gargamel kernel: [1624709.359472] md/raid:md7: read error not correctable (sector 2504672 on sdh1). > Mar 29 09:30:12 gargamel kernel: [1624709.359486] md/raid:md7: read error not correctable (sector 2504656 on sdh1). > Mar 29 09:30:12 gargamel kernel: [1624709.455886] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.681637] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.695785] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.717624] md/raid:md7: read error not correctable (sector 2504664 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624709.739746] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.757206] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.770348] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.790298] md/raid:md7: read error not correctable (sector 2504672 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624709.812546] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.825856] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.839006] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.866656] md/raid:md7: read error not correctable (sector 24815552 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624709.898796] md/raid:md7: read error not correctable (sector 24815552 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624709.921135] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.934315] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.947500] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624709.988589] md/raid:md7: read error not correctable (sector 1763985936 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624710.036204] md/raid:md7: read error not correctable (sector 1763985936 on sdh1). > Mar 29 09:30:13 gargamel kernel: [1624710.059121] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624710.088858] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624710.102026] md: super_written gets error=10 > Mar 29 09:30:13 gargamel kernel: [1624710.158830] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.096055] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:36:37 gargamel kernel: [1625094.122910] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 1, flush 0, corrupt 0, gen 0 > Mar 29 09:36:37 gargamel kernel: [1625094.153249] md/raid:md7: read error not correctable (sector 6562801616 on sdh1). > Mar 29 09:36:37 gargamel kernel: [1625094.176011] md/raid:md7: read error not correctable (sector 6562801624 on sdh1). > Mar 29 09:36:37 gargamel kernel: [1625094.223351] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.250628] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.263726] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.276989] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.290121] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.303267] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.325084] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.342083] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.355206] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.368394] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.383304] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.396423] md: super_written gets error=10 > Mar 29 09:36:37 gargamel kernel: [1625094.409498] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:36:37 gargamel kernel: [1625094.436355] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 2, flush 0, corrupt 0, gen 0 > Mar 29 09:36:37 gargamel kernel: [1625094.466729] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:36:37 gargamel kernel: [1625094.493600] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 3, flush 0, corrupt 0, gen 0 > Mar 29 09:36:37 gargamel kernel: [1625094.523998] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:36:38 gargamel kernel: [1625094.550938] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 4, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.066422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.093309] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 5, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.124768] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.151651] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 6, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.182803] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.209677] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 7, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.239972] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.266862] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 8, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.297234] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.324094] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 9, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.354422] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.381286] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 10, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.411926] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.438770] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 11, flush 0, corrupt 0, gen 0 > Mar 29 09:37:34 gargamel kernel: [1625151.469361] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:34 gargamel kernel: [1625151.496269] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 12, flush 0, corrupt 0, gen 0 > Mar 29 09:37:35 gargamel kernel: [1625151.527455] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:35 gargamel kernel: [1625151.554360] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 13, flush 0, corrupt 0, gen 0 > Mar 29 09:37:35 gargamel kernel: [1625151.584963] bcache: bch_count_backing_io_errors() md7: IO error on backing device, unrecoverable > Mar 29 09:37:35 gargamel kernel: [1625151.611842] BTRFS error (device dm-17): bdev /dev/mapper/dshelf1a errs: wr 0, rd 14, flush 0, corrupt 0, gen 0 > > 24 seconds are lost before btrfs notices anything, and then it seems to > continue without going read only for another full minute before things > stopped. > > Why would the raid5 not go read only immediately after 2 drives are lost? Once a degraded array has an IO failure, mdadm assumes you're in a data-scraping recovery mode and simply passes errors through. bcache does something similar when the backing store fails. btrfs is the agent that has to stop attempting writes, and make sure any transaction in progress doesn't get committed. ext4 has similar responsibility and implements its own force-read-only feature. I note that btrfs is reporting only read errors here. There's a possibility that the drive dropped the writes during the bus reset at the start of the second drive failure. If write caching was enabled in the drive, and the drive has a firmware bug that drops the write cache contents (or it's just failing hardware, i.e. the CPU running the drive firmware is getting killed by electrical problems on the controller board, causing both the bus drop and the loss of write cache contents), then writes in the drive's cache could be lost _after_ mdadm, bcache and btrfs had been told by the drive that they were completed. If the failing drive also reorders cached writes across flush commands, then we go directly to parent transid verify failed in btrfs. Parent transid verification is designed to detect this exact firmware failure mode, and it usually works as intended. It's a pretty direct and reliable signal that write ordering is broken in a lower level of the storage stack, and must be fixed or disabled before trying again. Even if a single drive doesn't reorder writes, multiple drives in raid effectively reorder writes between drives as each drive has its own distinct write queue. A dropped write that would be harmless in a single-device filesystem could be harmful in a multi-device array as the non-failing drives will have writes that occurred after the lost writes on the failing drive. Normally mdadm enforces flush ordering across all component devices so that this isn't a problem, but if you have optimistic firmware and a drive failure after the flush command returns success, the inter-drive ordering enforcement fails and the result is the same as if an individual drive had a write/flush reordering bug. Firmware bugs are fairly rare across drive models, but failing drive models are vastly overrepresented in the field. I've tested hundreds of firmware revisions and only about 4% fail with bad write reordering. The few firmwares that do fail are found in popular low-end consumer drives and cheap enterprise drives (they're the same drive underneath, just a different label and price), so there are a lot of bad drives in the field. Once we identify a bad firmware revision, we avoid buying any more of it, disable write caching in the firmware on the drives we already have, and try to find a workload for the drive that is less sensitive to failure (or performance--fsync latency goes up when the drive can't use its cache for writes). Some years ago we did a fleetwide audit of bad firmware drives and found about a third of our drives were bad. We disabled write cache on all affected drives and prioritized replacement of the affected models. This stopped the worst btrfs problems we had been having up to that point. It also stopped a lot of strange one-off application data corruptions we had been chasing for years before on ext4/mdadm arrays and in postgresql databases. We'd assumed those were some kind of application bug, but we never found an app bug, and the problem disappeared once we turned off the write cache. > > If we don't have any intact subvol trees (or the subvol trees we really > > want aren't intact), then we can't recover this way. Instead we'd have > > to scrape the disk looking for metadata leaf nodes, and try to reinsert > > those into a new tree structure. The trick here is that we'll have the > > duplicated and inconsitent nodes and we won't have some nodes at all, > > and we'll have to make sense of those (or pass it to the existing btrfs > > check and hope it can cope with them). I'm guessing that a simplified > > version of this is what Josef is building at this point, or will be > > building soon if we aren't extremely lucky and find an intact subvol tree. > > After building an intact subvol tree (even with a few garbage items in it > > as long as check can handle them) we can go back to the --init-extent-tree > > step and rebuild the rest of the filesystem. > > I see. In that case, I'm still happy to help, to help improve the tools, but if > I'm looking at some amount of non trivial loss/corruption, at soe point > I'll go back to backups, since they'll be more intact than this now > damaged filesystem. Yeah, experimental recovery code is fun, but robust backups and a working disaster recovery plan is usually better. Even if the filesystem is up and running again, I'd want to compare all the files against backups because I'd trust nothing on a filesystem after fsck touched it. On the other hand, if you get lucky and the filesystem isn't too badly damaged, then comparing the data with backups will be more convenient than starting over with a new filesystem. > Thanks for the detailed answers. > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 18:07 ` Zygo Blaxell @ 2022-04-06 19:13 ` Marc MERLIN 2022-04-06 19:45 ` Zygo Blaxell ` (2 more replies) 0 siblings, 3 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 19:13 UTC (permalink / raw) To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote: > Readahead can fail for a number of reasons that aren't real problems, thanks. > Once a degraded array has an IO failure, mdadm assumes you're in > a data-scraping recovery mode and simply passes errors through. > bcache does something similar when the backing store fails. shouldn't it go read only also? I haven't found a setting to tell it to do that if it's not the default. > btrfs is the agent that has to stop attempting writes, and make sure > any transaction in progress doesn't get committed. ext4 has similar > responsibility and implements its own force-read-only feature. Agreed, but I like the defense in multiple layers approach mdadm knows that any data going to be written is going to be incomplete due to the 2nd missing drive, and there are few to no scenarios where continuing to write is a good thing. > I note that btrfs is reporting only read errors here. Yeah, me too. That seems very bad and likely the cause or the corruption I got. A double raid failure (I've had them before) should be almost a non event. The first write after that should fail and that should be the end of it. This underlying problem makes me very uncomfortable. As much as I'm super thankful for Josef's efforts, and ultimately it will make btrfs recovery better, which is a win for all, my situation shouldn't have happened from what I can tell. > There's a possibility that the drive dropped the writes during the > bus reset at the start of the second drive failure. If write caching > was enabled in the drive, and the drive has a firmware bug that drops > the write cache contents (or it's just failing hardware, i.e. the CPU > running the drive firmware is getting killed by electrical problems on the > controller board, causing both the bus drop and the loss of write cache > contents), then writes in the drive's cache could be lost _after_ mdadm, > bcache and btrfs had been told by the drive that they were completed. That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only? Write caching is on all those drives though: gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh /dev/sdh: multcount = 16 (on) readonly = 0 (off) readahead = 256 (on) geometry = 729601/255/63, sectors = 11721045168, start = 0 write-caching = 1 (on) I haven't heard that these drives have broken caching, but maybe they do? Device Model: ST6000VN0041-2EL11C Serial Number: ZA18TVFZ LU WWN Device Id: 5 000c50 0a4d9b49c Firmware Version: SC61 User Capacity: 6,001,175,126,016 bytes [6.00 TB] > If the failing drive also reorders cached writes across flush > commands, then we go directly to parent transid verify failed in btrfs. > Parent transid verification is designed to detect this exact firmware > failure mode, and it usually works as intended. It's a pretty direct > and reliable signal that write ordering is broken in a lower level of > the storage stack, and must be fixed or disabled before trying again. Agreed, I do want to fix the underlying problem here. What I can do is 1) disable write caching on the drives 2) disable bcache by removing the caching device 3) change the cache mode gargamel:/sys/block/md7/bcache# cat cache_mode [writethrough] writeback writearound none writethrough should be safe, but I could use writearound instead 4) if I end up wiping my device, I can just remove the bcache layer altogether. > Even if a single drive doesn't reorder writes, multiple drives in > raid effectively reorder writes between drives as each drive has its > own distinct write queue. A dropped write that would be harmless in a very good point. > single-device filesystem could be harmful in a multi-device array as the > non-failing drives will have writes that occurred after the lost writes > on the failing drive. Normally mdadm enforces flush ordering across all > component devices so that this isn't a problem, but if you have optimistic > firmware and a drive failure after the flush command returns success, > the inter-drive ordering enforcement fails and the result is the same > as if an individual drive had a write/flush reordering bug. That's all fair, but it feels like FS kept writing way longer than it was supposed to and that is what worries me the most. > Some years ago we did a fleetwide audit of bad firmware drives and > found about a third of our drives were bad. We disabled write cache on Looks like I just want to disable write caching then. Correctness beats speed for sure. > Yeah, experimental recovery code is fun, but robust backups and a working > disaster recovery plan is usually better. Even if the filesystem is > up and running again, I'd want to compare all the files against backups > because I'd trust nothing on a filesystem after fsck touched it. That is totally my plan. Given the output that I'm seeing, I'll definitely do at least a diff of all files between the backup and the array being recovered. I might even do an rsync, forcing md5 checksums, but if I do that, it will take days and longer than restoring the backup. > On the other hand, if you get lucky and the filesystem isn't too badly > damaged, then comparing the data with backups will be more convenient > than starting over with a new filesystem. Agreed. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 19:13 ` Marc MERLIN @ 2022-04-06 19:45 ` Zygo Blaxell 2022-04-06 20:38 ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN 2022-04-22 18:48 ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN 2 siblings, 0 replies; 479+ messages in thread From: Zygo Blaxell @ 2022-04-06 19:45 UTC (permalink / raw) To: Marc MERLIN; +Cc: Josef Bacik, linux-btrfs On Wed, Apr 06, 2022 at 12:13:17PM -0700, Marc MERLIN wrote: > On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote: > > Readahead can fail for a number of reasons that aren't real problems, > > thanks. > > > Once a degraded array has an IO failure, mdadm assumes you're in > > a data-scraping recovery mode and simply passes errors through. > > bcache does something similar when the backing store fails. > > shouldn't it go read only also? > I haven't found a setting to tell it to do that if it's not the default. bcache in writethrough mode could leave cached blocks dirty as long as the SSD completes the write. It should be reporting the write errors back to upper layers. Whether it actually does...I don't know, I haven't run that kind of stress test on bcache. I usually pair bcache SSD 1:1 with HDD, so if either the SSD or HDD fails, I scrub or replace them both as a single logical btrfs device, which makes this kind of question mostly irrelevant. Unfortunately this approach doesn't work with raid5--the striping cuts IO requests into tiny fragments, so the cache will try to cache everything. > > btrfs is the agent that has to stop attempting writes, and make sure > > any transaction in progress doesn't get committed. ext4 has similar > > responsibility and implements its own force-read-only feature. > > Agreed, but I like the defense in multiple layers approach > mdadm knows that any data going to be written is going to be incomplete > due to the 2nd missing drive, and there are few to no scenarios where > continuing to write is a good thing. True, there's no way the parity would make any sense with two drives in a stripe missing during an update. So maybe there weren't any writes going through mdadm at that point, and everything failed at the drive-offline event. Or writes completed on the working drives before the failing drive's failure was detected, messing up the data there. A typical drive bus timeout is a long time, and a lot of raid stripes could be modified on other drives while that happens. > > There's a possibility that the drive dropped the writes during the > > bus reset at the start of the second drive failure. If write caching > > was enabled in the drive, and the drive has a firmware bug that drops > > the write cache contents (or it's just failing hardware, i.e. the CPU > > running the drive firmware is getting killed by electrical problems on the > > controller board, causing both the bus drop and the loss of write cache > > contents), then writes in the drive's cache could be lost _after_ mdadm, > > bcache and btrfs had been told by the drive that they were completed. > > That's true, but I've seen btfrs remount read only before, and it didn't > there. Shouldn't hard IO errors immediately cause btrfs to go read only? No, only hard write IO errors on all metadata mirror drives, and cases where btrfs needs to CoW a page or read free space tree, and can't find an intact mirror. Anything less is correctable (write failure on some mirrors) or can be retried (any read failure) if the raid profile has redundancy. > Write caching is on all those drives though: > gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh > /dev/sdh: > multcount = 16 (on) > readonly = 0 (off) > readahead = 256 (on) > geometry = 729601/255/63, sectors = 11721045168, start = 0 > write-caching = 1 (on) > > I haven't heard that these drives have broken caching, but maybe they do? > Device Model: ST6000VN0041-2EL11C > Serial Number: ZA18TVFZ > LU WWN Device Id: 5 000c50 0a4d9b49c > Firmware Version: SC61 > User Capacity: 6,001,175,126,016 bytes [6.00 TB] I've heard of problems with SC60, but IIRC SC61 was supposed to be the fix for them. I had some SC60 drives for a while without encountering issues, but I didn't have any bus timeouts with them. When they failed, they had a burst of 10k+ UNC sectors and then stopped spinning up, with no bus timeouts. > > If the failing drive also reorders cached writes across flush > > commands, then we go directly to parent transid verify failed in btrfs. > > Parent transid verification is designed to detect this exact firmware > > failure mode, and it usually works as intended. It's a pretty direct > > and reliable signal that write ordering is broken in a lower level of > > the storage stack, and must be fixed or disabled before trying again. > > Agreed, I do want to fix the underlying problem here. > What I can do is > > 1) disable write caching on the drives > > 2) disable bcache by removing the caching device > > 3) change the cache mode > gargamel:/sys/block/md7/bcache# cat cache_mode > [writethrough] writeback writearound none > writethrough should be safe, but I could use writearound instead > > 4) if I end up wiping my device, I can just remove the bcache layer altogether. > > > Even if a single drive doesn't reorder writes, multiple drives in > > raid effectively reorder writes between drives as each drive has its > > own distinct write queue. A dropped write that would be harmless in a > > very good point. > > > single-device filesystem could be harmful in a multi-device array as the > > non-failing drives will have writes that occurred after the lost writes > > on the failing drive. Normally mdadm enforces flush ordering across all > > component devices so that this isn't a problem, but if you have optimistic > > firmware and a drive failure after the flush command returns success, > > the inter-drive ordering enforcement fails and the result is the same > > as if an individual drive had a write/flush reordering bug. > > That's all fair, but it feels like FS kept writing way longer than it > was supposed to and that is what worries me the most. The log excerpt I saw didn't show any write errors from btrfs, only from mdadm. That suggests to me that the failure happened earlier when it was already too late to respond. Also we can't rule out that bcache is doing something stupid. > > Some years ago we did a fleetwide audit of bad firmware drives and > > found about a third of our drives were bad. We disabled write cache on > > Looks like I just want to disable write caching then. Correctness beats > speed for sure. > > > Yeah, experimental recovery code is fun, but robust backups and a working > > disaster recovery plan is usually better. Even if the filesystem is > > up and running again, I'd want to compare all the files against backups > > because I'd trust nothing on a filesystem after fsck touched it. > > That is totally my plan. Given the output that I'm seeing, I'll definitely do > at least a diff of all files between the backup and the array being recovered. > I might even do an rsync, forcing md5 checksums, but if I do that, it > will take days and longer than restoring the backup. > > > On the other hand, if you get lucky and the filesystem isn't too badly > > damaged, then comparing the data with backups will be more convenient > > than starting over with a new filesystem. > > Agreed. > > Thanks, > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ > ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption 2022-04-06 19:13 ` Marc MERLIN 2022-04-06 19:45 ` Zygo Blaxell @ 2022-04-06 20:38 ` Marc MERLIN 2022-04-06 20:51 ` Josef Bacik 2022-04-22 18:48 ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN 2 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 20:38 UTC (permalink / raw) To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs This is an interesting discussion, so let's make a new thread out of it. TL;DR: I think btrfs may have failed to go read only earlier, causing more damage than needed to be, or some block layers just held enough data in flight that partial data got written, causing more damage than expected. Figuring out the underlying problem would be good to avoid this again. On Wed, Apr 06, 2022 at 02:07:47PM -0400, Zygo Blaxell wrote: > Readahead can fail for a number of reasons that aren't real problems, thanks. > Once a degraded array has an IO failure, mdadm assumes you're in > a data-scraping recovery mode and simply passes errors through. > bcache does something similar when the backing store fails. shouldn't it go read only also? I haven't found a setting to tell it to do that if it's not the default. > btrfs is the agent that has to stop attempting writes, and make sure > any transaction in progress doesn't get committed. ext4 has similar > responsibility and implements its own force-read-only feature. Agreed, but I like the defense in multiple layers approach mdadm knows that any data going to be written is going to be incomplete due to the 2nd missing drive, and there are few to no scenarios where continuing to write is a good thing. > I note that btrfs is reporting only read errors here. Yeah, me too. That seems very bad and likely the cause or the corruption I got. A double raid failure (I've had them before) should be almost a non event. The first write after that should fail and that should be the end of it. This underlying problem makes me very uncomfortable. As much as I'm super thankful for Josef's efforts, and ultimately it will make btrfs recovery better, which is a win for all, my situation shouldn't have happened from what I can tell. > There's a possibility that the drive dropped the writes during the > bus reset at the start of the second drive failure. If write caching > was enabled in the drive, and the drive has a firmware bug that drops > the write cache contents (or it's just failing hardware, i.e. the CPU > running the drive firmware is getting killed by electrical problems on the > controller board, causing both the bus drop and the loss of write cache > contents), then writes in the drive's cache could be lost _after_ mdadm, > bcache and btrfs had been told by the drive that they were completed. That's true, but I've seen btfrs remount read only before, and it didn't there. Shouldn't hard IO errors immediately cause btrfs to go read only? Write caching is on all those drives though: gargamel:/var/local/src/btrfs-progs-josefbacik# hdparm -v -W /dev/sdh /dev/sdh: multcount = 16 (on) readonly = 0 (off) readahead = 256 (on) geometry = 729601/255/63, sectors = 11721045168, start = 0 write-caching = 1 (on) I haven't heard that these drives have broken caching, but maybe they do? Device Model: ST6000VN0041-2EL11C Serial Number: ZA18TVFZ LU WWN Device Id: 5 000c50 0a4d9b49c Firmware Version: SC61 User Capacity: 6,001,175,126,016 bytes [6.00 TB] > If the failing drive also reorders cached writes across flush > commands, then we go directly to parent transid verify failed in btrfs. > Parent transid verification is designed to detect this exact firmware > failure mode, and it usually works as intended. It's a pretty direct > and reliable signal that write ordering is broken in a lower level of > the storage stack, and must be fixed or disabled before trying again. Agreed, I do want to fix the underlying problem here. What I can do is 1) disable write caching on the drives 2) disable bcache by removing the caching device 3) change the cache mode gargamel:/sys/block/md7/bcache# cat cache_mode [writethrough] writeback writearound none writethrough should be safe, but I could use writearound instead 4) if I end up wiping my device, I can just remove the bcache layer altogether. > Even if a single drive doesn't reorder writes, multiple drives in > raid effectively reorder writes between drives as each drive has its > own distinct write queue. A dropped write that would be harmless in a very good point. > single-device filesystem could be harmful in a multi-device array as the > non-failing drives will have writes that occurred after the lost writes > on the failing drive. Normally mdadm enforces flush ordering across all > component devices so that this isn't a problem, but if you have optimistic > firmware and a drive failure after the flush command returns success, > the inter-drive ordering enforcement fails and the result is the same > as if an individual drive had a write/flush reordering bug. That's all fair, but it feels like FS kept writing way longer than it was supposed to and that is what worries me the most. > Some years ago we did a fleetwide audit of bad firmware drives and > found about a third of our drives were bad. We disabled write cache on Looks like I just want to disable write caching then. Correctness beats speed for sure. > Yeah, experimental recovery code is fun, but robust backups and a working > disaster recovery plan is usually better. Even if the filesystem is > up and running again, I'd want to compare all the files against backups > because I'd trust nothing on a filesystem after fsck touched it. That is totally my plan. Given the output that I'm seeing, I'll definitely do at least a diff of all files between the backup and the array being recovered. I might even do an rsync, forcing md5 checksums, but if I do that, it will take days and longer than restoring the backup. > On the other hand, if you get lucky and the filesystem isn't too badly > damaged, then comparing the data with backups will be more convenient > than starting over with a new filesystem. Agreed. Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption 2022-04-06 20:38 ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN @ 2022-04-06 20:51 ` Josef Bacik 2022-04-06 21:14 ` Marc MERLIN 2022-04-07 12:27 ` Zygo Blaxell 0 siblings, 2 replies; 479+ messages in thread From: Josef Bacik @ 2022-04-06 20:51 UTC (permalink / raw) To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs On Wed, Apr 6, 2022 at 4:38 PM Marc MERLIN <marc@merlins.org> wrote: > > This is an interesting discussion, so let's make a new thread out of it. > TL;DR: I think btrfs may have failed to go read only earlier, causing > more damage than needed to be, or some block layers just held enough > data in flight that partial data got written, causing more damage than > expected. > Figuring out the underlying problem would be good to avoid this again. > There's a lot here, and there's a lot of good information, but in general btrfs should never fail like this. We have a lot of power fail testing and error handling testing, but this is very limited. I found a bug last year where we would write an updated super block that pointed to garbage if we happened to be fsync'ing at the same time we were committing the transaction and got an error during the transaction commit phase. We can't do anything about the disks lying to us. If a disk has a wonky FUA/FLUSH implementation then we're just sort of screwed. Unfortunately because our metadata moves around a lot we're waaaaay more susceptible to this failure case than ext4 or xfs, their metadata is relatively static they can put humpty dumpty back together again relatively simply. Btrfs needs to 1. Go whole hog on error injection testing. I only barely scratched the surface with my bpf error injection stuff. This is on our roadmap and I plan on devoting developer time to this, but clearly that doesn't help you right now. 2. Put a lot more effort into disaster recovery. What I've written for you is an idea I've had in my head for a while. Some of this failures aren't catastrophic, we can generally pretty easily put back together a file system that resembles something sane by simply stitching together blocks that we find that are close enough to what we wanted. Unfortunately this gets back-burnered because in reality this doesn't happen that often. 3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of these catastrophic failures it generally involves btrfs+<something else>. This is likely just because it's a timing thing, you put more layers you get a wider window in per-io races, you're more likely to be sad in the event of a failure. However it would be good to make sure these layers are doing the correct thing themselves. We need to be better about this scenario, both in making sure we don't have bugs that contribute to the problem, but also that we have the tools necessary to recover when things go wrong. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption 2022-04-06 20:51 ` Josef Bacik @ 2022-04-06 21:14 ` Marc MERLIN 2022-04-07 12:27 ` Zygo Blaxell 1 sibling, 0 replies; 479+ messages in thread From: Marc MERLIN @ 2022-04-06 21:14 UTC (permalink / raw) To: Josef Bacik, Zygo Blaxell; +Cc: linux-btrfs (Zygo, answer for you below) On Wed, Apr 06, 2022 at 04:51:14PM -0400, Josef Bacik wrote: > We can't do anything about the disks lying to us. If a disk has a > wonky FUA/FLUSH implementation then we're just sort of screwed. > Unfortunately because our metadata moves around a lot we're waaaaay > more susceptible to this failure case than ext4 or xfs, their metadata > is relatively static they can put humpty dumpty back together again > relatively simply. Yeah, that's clearly been my experience. The average end user (which kind of includes me), totally understands that last data written, includig potentially directories, will get damaged, potentially beyond repair, but losing the entire filesystem when 99.99%+ never got touched by the last operations, is unexpected and undesired. That said, yes, I understand that btrfs does things that are both cool and complex, and they are less tolerant of corruption. > 2. Put a lot more effort into disaster recovery. What I've written > for you is an idea I've had in my head for a while. Some of this I truly appreciate the time you've been spending with me on this. In the end, I do have a backup, but I could not have one, or it could fail when I'm reading back from it, so being able to recover the filesystem to some workable state, is definitely a big plus. Honestly I don't even really care if I lose 0.1%, 1%, or 5% of the data (even if probably only <0.001% of the data got touched in the minute+ that btrfs was still apparently trying to use the filesystem after the double failure), but a recovery that takes at most hours (for 24TB) and not days, or never (never is a multiple day restore over the network for me), will definitely be a great improvement. > 3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of > these catastrophic failures it generally involves btrfs+<something > else>. This is likely just because it's a timing thing, you put more > layers you get a wider window in per-io races, you're more likely to > be sad in the event of a failure. However it would be good to make > sure these layers are doing the correct thing themselves. Totally agreed, I'm actually the worst case scenario, I have disk hw write cache + swraid5 + bcache + dmcrypt + btrfs that's clearly a lot of layers. This is now my 4 or 5th pretty catastrophic btrfs failure over 2 arrays with those amount of layers over the last 8 years, but the last one was 2-3 years ago, so there is that :) I think it's pretty clear that having all those layers is significantly increasing my risk factor. > We need to be better about this scenario, both in making sure we don't > have bugs that contribute to the problem, but also that we have the > tools necessary to recover when things go wrong. Thanks, agreed to both, thanks again for your time on this On Wed, Apr 06, 2022 at 03:45:57PM -0400, Zygo Blaxell wrote: > > shouldn't it go read only also? > > I haven't found a setting to tell it to do that if it's not the default. > > bcache in writethrough mode could leave cached blocks dirty as long as > the SSD completes the write. It should be reporting the write errors > back to upper layers. > Whether it actually does...I don't know, I haven't run that kind of agreed. > > That's true, but I've seen btfrs remount read only before, and it didn't > > there. Shouldn't hard IO errors immediately cause btrfs to go read only? > > No, only hard write IO errors on all metadata mirror drives, and cases > where btrfs needs to CoW a page or read free space tree, and can't find > an intact mirror. Anything less is correctable (write failure on some > mirrors) or can be retried (any read failure) if the raid profile has > redundancy. Fair enough. And as you said, my errors were probably already staged in multiple layers and flushing before I got the first error reported back, so even by then it mau have been too late to stop it. > > I haven't heard that these drives have broken caching, but maybe they do? > > Device Model: ST6000VN0041-2EL11C > > Serial Number: ZA18TVFZ > > LU WWN Device Id: 5 000c50 0a4d9b49c > > Firmware Version: SC61 > > User Capacity: 6,001,175,126,016 bytes [6.00 TB] > > I've heard of problems with SC60, but IIRC SC61 was supposed to be the Well, there you go, just found out that I have a mix of SC60 and SC61. Ok, I'll just turn off write caching on them all. > The log excerpt I saw didn't show any write errors from btrfs, only > from mdadm. That suggests to me that the failure happened earlier when > it was already too late to respond. Also we can't rule out that bcache > is doing something stupid. Yeah, at this point I think the logs agree with that statement. Also, I wasn't actually using that array, stuff doesn't really get written to it unless I manually start copies. Thanks for looking this over and your opinion/sanity check. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: figuring out why transient double raid failure caused a fair amount of btrfs corruption 2022-04-06 20:51 ` Josef Bacik 2022-04-06 21:14 ` Marc MERLIN @ 2022-04-07 12:27 ` Zygo Blaxell 1 sibling, 0 replies; 479+ messages in thread From: Zygo Blaxell @ 2022-04-07 12:27 UTC (permalink / raw) To: Josef Bacik; +Cc: Marc MERLIN, linux-btrfs On Wed, Apr 06, 2022 at 04:51:14PM -0400, Josef Bacik wrote: > On Wed, Apr 6, 2022 at 4:38 PM Marc MERLIN <marc@merlins.org> wrote: > > > > This is an interesting discussion, so let's make a new thread out of it. > > TL;DR: I think btrfs may have failed to go read only earlier, causing > > more damage than needed to be, or some block layers just held enough > > data in flight that partial data got written, causing more damage than > > expected. > > Figuring out the underlying problem would be good to avoid this again. > We can't do anything about the disks lying to us. If a disk has a > wonky FUA/FLUSH implementation then we're just sort of screwed. > Unfortunately because our metadata moves around a lot we're waaaaay > more susceptible to this failure case than ext4 or xfs, their metadata > is relatively static they can put humpty dumpty back together again > relatively simply. Disks are pretty good at FUA/FLUSH bugs these days, though you can still buy new old drives that have them. SSDs have a whole lot of bugs, especially if they get old. have a few bits fail, and try to recover themselves, but blow up the whole filesystem by losing a few pages. These ones get through firmware qualification testing because they only misbehave when they're a year or two old. > Btrfs needs to > > 1. Go whole hog on error injection testing. I only barely scratched > the surface with my bpf error injection stuff. This is on our roadmap > and I plan on devoting developer time to this, but clearly that > doesn't help you right now. > 2. Put a lot more effort into disaster recovery. What I've written > for you is an idea I've had in my head for a while. Some of this > failures aren't catastrophic, we can generally pretty easily put back > together a file system that resembles something sane by simply > stitching together blocks that we find that are close enough to what > we wanted. Unfortunately this gets back-burnered because in reality > this doesn't happen that often. This does seem to be an important missing piece at the moment. In practice, when we see "parent transid verify failed," we go directly to mkfs + restore backups, as none of the existing tools will touch it. They all want intact interior nodes in some tree to work, and that's the one thing that FUA/FLUSH bugs destroy. It doesn't solve the underlying problem--the drive will still trash the filesystem every other month until its write ordering gets fixed or worked around--but at least it lets users fix the filesystem in place without mkfs + restoring a full set of backups. > 3. Test these btrfs+dmcrypt+mdraid setups. Every time I notice one of > these catastrophic failures it generally involves btrfs+<something > else>. This is likely just because it's a timing thing, you put more > layers you get a wider window in per-io races, you're more likely to > be sad in the event of a failure. However it would be good to make > sure these layers are doing the correct thing themselves. bcache and dmcache layers in series with the underlying storage multiply the failure rates (including software regressions) of the individual components, and mdadm blends firmware bugs from all the drives together. Statistically we're always going to see more problems if there are more moving parts in the system, especially if they aggregate risks and break fault isolation. FWIW I'm not seeing a difference in failure rate between "btrfs + mdadm" and "btrfs alone", but I'm intentionally avoiding the many mdadm configurations that introduce new failure modes by design (e.g. raid5 without ppl or journal, raid1 without component device integrity, or single/raid1 SSD cache in writeback mode running over multiple drives). There isn't a fix for those except to not use them. In most cases there's a better way to do the same thing, though there are some gaps (e.g. there's no working solution for writeback SSD caching on raid5). Thought experiment: if you have 2 drives from vendor A and 2 drives from vendor B, and you want to build two filesystems that replicate to each other, where do you put the drives if you know one vendor has a firmware bug but you don't know which one? With mdadm, you build filesystems with AA and BB drives, because that way a firmware bug corrupts one filesystems and leaves the other intact. With btrfs, you build two filesystems with AB and AB drives, because that way a firmware bug gets autocorrected by btrfs and both filesystems remain intact. If you put btrfs on mdadm on AB drives, you combine the risk from both vendors, lose both filesystems, and skew the failure statistics. > We need to be better about this scenario, both in making sure we don't > have bugs that contribute to the problem, but also that we have the > tools necessary to recover when things go wrong. Thanks, > > Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-06 19:13 ` Marc MERLIN 2022-04-06 19:45 ` Zygo Blaxell 2022-04-06 20:38 ` figuring out why transient double raid failure caused a fair amount of btrfs corruption Marc MERLIN @ 2022-04-22 18:48 ` Marc MERLIN 2022-04-22 19:46 ` Josef Bacik 2 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-22 18:48 UTC (permalink / raw) To: Zygo Blaxell; +Cc: Josef Bacik, linux-btrfs Back on list, Josef made a lot of changes to btrfs-progs for me (thanks Josef) gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x55c6d219bbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55c6d219bbc0 Checking root 2 Checking root 4 Checking root 5 Checking root 7 Checking root 9 Checking root 11221 Checking root 11222 Checking root 11223 After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been running more for than 48H now Starting repair. Opening filesystem to check... checksum verify failed on 15645878108160 found 00000027 wanted 0000001B checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4 (...) checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 Failed to find [13576823668736, 168, 16384] btrfs unable to find ref byte nr 13577801809920 parent 0 root 1 owner 0 offset 0 path->slots[0]: 134 path->nodes[0]: leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE leaf 49709056 flags 0x0() backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single (...) Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. (...) Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk. ref mismatch on [12582912 4096] extent item 0, found 1 data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0 backpointer mismatch on [12582912 4096] adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1 Repaired extent references for 12582912 ref mismatch on [12587008 4096] extent item 0, found 1 data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00 backpointer mismatch on [12587008 4096] adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1 Repaired extent references for 12587008 ref mismatch on [12591104 4096] extent item 0, found 1 (...) incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80 backpointer mismatch on [20963328 4096] adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1 Repaired extent references for 20963328 ref mismatch on [20967424 4096] extent item 0, found 1 data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0 backpointer mismatch on [20967424 4096] adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1 Repaired extent references for 20967424 ref mismatch on [20971520 16384] extent item 0, found 1 tree backref 20971520 root 3 not found in extent tree backpointer mismatch on [20971520 16384] adding new tree backref on start 20971520 len 16384 parent 0 root 3 Repaired extent references for 20971520 ref mismatch on [20987904 16384] extent item 0, found 1 tree backref 20987904 root 3 not found in extent tree backpointer mismatch on [20987904 16384] adding new tree backref on start 20987904 len 16384 parent 0 root 3 Repaired extent references for 20987904 ref mismatch on [21004288 16384] extent item 0, found 1 tree backref 21004288 root 3 not found in extent tree (..) adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320 Repaired extent references for 44531712 ref mismatch on [44548096 16384] extent item 0, found 1 tree backref 44548096 parent 47398912 not found in extent tree backpointer mismatch on [44548096 16384] adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912 Repaired extent references for 44548096 ref mismatch on [44564480 16384] extent item 0, found 1 tree backref 44564480 root 7 not found in extent tree backpointer mismatch on [44564480 16384] adding new tree backref on start 44564480 len 16384 parent 0 root 7 Repaired extent references for 44564480 ref mismatch on [44580864 16384] extent item 0, found 1 tree backref 44580864 root 7 not found in extent tree backpointer mismatch on [44580864 16384] adding new tree backref on start 44580864 len 16384 parent 0 root 7 Repaired extent references for 44580864 ref mismatch on [44597248 16384] extent item 0, found 1 tree backref 44597248 root 7 not found in extent tree backpointer mismatch on [44597248 16384] adding new tree backref on start 44597248 len 16384 parent 0 root 7 Repaired extent references for 44597248 ref mismatch on [44613632 16384] extent item 0, found 1 tree backref 44613632 parent 47398912 not found in extent tree backpointer mismatch on [44613632 16384] adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912 Repaired extent references for 44613632 ref mismatch on [44630016 16384] extent item 0, found 1 tree backref 44630016 parent 47398912 not found in extent tree backpointer mismatch on [44630016 16384] adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912 (...) gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1 9026 It's been running for close to 3 days, and I'm a bit confused that it's repairing so many things when it wast just a minute's worth of potential corruption. Do I keep running for many more days to see where this goes, or at this point 73,860 lines of output is not a bad sign and I should look at restoring from backup? Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-22 18:48 ` Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) Marc MERLIN @ 2022-04-22 19:46 ` Josef Bacik 2022-04-22 20:01 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-22 19:46 UTC (permalink / raw) To: Marc MERLIN; +Cc: Zygo Blaxell, linux-btrfs On Fri, Apr 22, 2022 at 2:48 PM Marc MERLIN <marc@merlins.org> wrote: > > Back on list, Josef made a lot of changes to btrfs-progs for me (thanks > Josef) > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 > FS_INFO IS 0x55c6d219bbc0 > Couldn't find the last root for 8 > FS_INFO AFTER IS 0x55c6d219bbc0 > Checking root 2 > Checking root 4 > Checking root 5 > Checking root 7 > Checking root 9 > Checking root 11221 > Checking root 11222 > Checking root 11223 > > After that ./btrfs check --init-extent-tree /dev/mapper/dshelf1 has been > running more for than 48H now > > Starting repair. > Opening filesystem to check... > checksum verify failed on 15645878108160 found 00000027 wanted 0000001B > checksum verify failed on 11821979287552 found 000000A9 wanted FFFFFFFB > checksum verify failed on 11822142046208 found 000000AC wanted FFFFFFC4 > (...) > checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 > checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 > Failed to find [13576823668736, 168, 16384] > btrfs unable to find ref byte nr 13577801809920 parent 0 root 1 owner 0 offset 0 > path->slots[0]: 134 path->nodes[0]: > leaf 49709056 items 169 free space 7975 generation 1602090 owner EXTENT_TREE > leaf 49709056 flags 0x0() backref revision 1 > fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e > chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a > item 0 key (13400696422400 BLOCK_GROUP_ITEM 1073741824) itemoff 16259 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 1 key (13401770164224 BLOCK_GROUP_ITEM 1073741824) itemoff 16235 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > (...) > Chunk[256, 228, 15365845286912] stripe[1, 14785462665216] is not found in dev extent > Chunk[256, 228, 15366919028736] stripe[1, 14786536407040] is not found in dev extent > Device extent[1, 11503033909248, 1073741824] didn't find the relative chunk. > Device extent[1, 11595375706112, 1073741824] didn't find the relative chunk. > Device extent[1, 11596449447936, 1073741824] didn't find the relative chunk. > Device extent[1, 11597523189760, 1073741824] didn't find the relative chunk. > Device extent[1, 11598596931584, 1073741824] didn't find the relative chunk. > (...) > Device extent[1, 11422503272448, 1073741824] didn't find the relative chunk. > Device extent[1, 10616123162624, 1073741824] didn't find the relative chunk. > ref mismatch on [12582912 4096] extent item 0, found 1 > data backref 12582912 root 11223 owner 260 offset 131072 num_refs 0 not found in extent tree > incorrect local backref count on 12582912 root 11223 owner 260 offset 131072 found 1 wanted 0 back 0x5604b40697a0 > backpointer mismatch on [12582912 4096] > adding new data backref on 12582912 root 11223 owner 260 offset 131072 found 1 > Repaired extent references for 12582912 > ref mismatch on [12587008 4096] extent item 0, found 1 > data backref 12587008 root 11223 owner 261 offset 20480 num_refs 0 not found in extent tree > incorrect local backref count on 12587008 root 11223 owner 261 offset 20480 found 1 wanted 0 back 0x5604b4069a00 > backpointer mismatch on [12587008 4096] > adding new data backref on 12587008 root 11223 owner 261 offset 20480 found 1 > Repaired extent references for 12587008 > ref mismatch on [12591104 4096] extent item 0, found 1 > (...) > incorrect local backref count on 20963328 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6d80 > backpointer mismatch on [20963328 4096] > adding new data backref on 20963328 parent 33718272 owner 0 offset 0 found 1 > Repaired extent references for 20963328 > ref mismatch on [20967424 4096] extent item 0, found 1 > data backref 20967424 parent 33718272 owner 0 offset 0 num_refs 0 not found in extent tree > incorrect local backref count on 20967424 parent 33718272 owner 0 offset 0 found 1 wanted 0 back 0x56043bdf6fe0 > backpointer mismatch on [20967424 4096] > adding new data backref on 20967424 parent 33718272 owner 0 offset 0 found 1 > Repaired extent references for 20967424 > ref mismatch on [20971520 16384] extent item 0, found 1 > tree backref 20971520 root 3 not found in extent tree > backpointer mismatch on [20971520 16384] > adding new tree backref on start 20971520 len 16384 parent 0 root 3 > Repaired extent references for 20971520 > ref mismatch on [20987904 16384] extent item 0, found 1 > tree backref 20987904 root 3 not found in extent tree > backpointer mismatch on [20987904 16384] > adding new tree backref on start 20987904 len 16384 parent 0 root 3 > Repaired extent references for 20987904 > ref mismatch on [21004288 16384] extent item 0, found 1 > tree backref 21004288 root 3 not found in extent tree > (..) > adding new tree backref on start 44531712 len 16384 parent 50872320 root 50872320 > Repaired extent references for 44531712 > ref mismatch on [44548096 16384] extent item 0, found 1 > tree backref 44548096 parent 47398912 not found in extent tree > backpointer mismatch on [44548096 16384] > adding new tree backref on start 44548096 len 16384 parent 47398912 root 47398912 > Repaired extent references for 44548096 > ref mismatch on [44564480 16384] extent item 0, found 1 > tree backref 44564480 root 7 not found in extent tree > backpointer mismatch on [44564480 16384] > adding new tree backref on start 44564480 len 16384 parent 0 root 7 > Repaired extent references for 44564480 > ref mismatch on [44580864 16384] extent item 0, found 1 > tree backref 44580864 root 7 not found in extent tree > backpointer mismatch on [44580864 16384] > adding new tree backref on start 44580864 len 16384 parent 0 root 7 > Repaired extent references for 44580864 > ref mismatch on [44597248 16384] extent item 0, found 1 > tree backref 44597248 root 7 not found in extent tree > backpointer mismatch on [44597248 16384] > adding new tree backref on start 44597248 len 16384 parent 0 root 7 > Repaired extent references for 44597248 > ref mismatch on [44613632 16384] extent item 0, found 1 > tree backref 44613632 parent 47398912 not found in extent tree > backpointer mismatch on [44613632 16384] > adding new tree backref on start 44613632 len 16384 parent 47398912 root 47398912 > Repaired extent references for 44613632 > ref mismatch on [44630016 16384] extent item 0, found 1 > tree backref 44630016 parent 47398912 not found in extent tree > backpointer mismatch on [44630016 16384] > adding new tree backref on start 44630016 len 16384 parent 47398912 root 47398912 > (...) > > gargamel:/var/local/src# grep -c 'adding new tree backref on start' checkrepair1 > 9026 > > It's been running for close to 3 days, and I'm a bit confused that it's repairing so > many things when it wast just a minute's worth of potential corruption. > > Do I keep running for many more days to see where this goes, or at this point > 73,860 lines of output is not a bad sign and I should look at restoring from backup? > We're doing the --init-extent-tree thing, which deletes the whole extent tree and rebuilds it. This isn't the fast path, 3 days is super shitty tho. I don't want to stop it and try and make it faster because once it's done you should just be golden. It looks scary, but that's because it just clears the extent tree and then lets the normal repair thing do its thing, so it yells loudly about all the messed up references and then fixes it. Now if we get to Monday and it's still running I can take a crack at making it faster. I was hoping it would only take a day or two, but we're balancing me trying to make it better and possibly fucking it up with letting it take the rest of our lives but be correct. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-22 19:46 ` Josef Bacik @ 2022-04-22 20:01 ` Marc MERLIN 2022-04-23 20:12 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-22 20:01 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Fri, Apr 22, 2022 at 03:46:08PM -0400, Josef Bacik wrote: > We're doing the --init-extent-tree thing, which deletes the whole > extent tree and rebuilds it. This isn't the fast path, 3 days is > super shitty tho. I don't want to stop it and try and make it faster > because once it's done you should just be golden. It looks scary, but > that's because it just clears the extent tree and then lets the normal > repair thing do its thing, so it yells loudly about all the messed up > references and then fixes it. Got it. > Now if we get to Monday and it's still running I can take a crack at > making it faster. I was hoping it would only take a day or two, but > we're balancing me trying to make it better and possibly fucking it up > with letting it take the rest of our lives but be correct. Thanks, Makes sense. I don't need faster, and it may not be able to go faster anyway, it's a lot of data. Just wanted to make sure the output and relative slow results were expected. And I'll go with slower but likely to work vs faster but very experimental :) That said, as Ted T'so said once, "anything with a progress bar is faster". So if it's possible in the future to add some completion percent, even if approximate, that will help when it can take multiple days (or week+) to run. Happy to wait in the meantime, thanks. Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-22 20:01 ` Marc MERLIN @ 2022-04-23 20:12 ` Marc MERLIN 2022-04-23 20:53 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-23 20:12 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote: > > Now if we get to Monday and it's still running I can take a crack at > > making it faster. I was hoping it would only take a day or two, but > > we're balancing me trying to make it better and possibly fucking it up > > with letting it take the rest of our lives but be correct. Thanks, > > Makes sense. I don't need faster, and it may not be able to go faster > anyway, it's a lot of data. Just wanted to make sure the output and > relative slow results were expected. Looking at the output, is there any way I can figure out if it's at 5% or 80% completion? tree backref 238026752 parent 236814336 not found in extent tree backpointer mismatch on [238026752 16384] adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 Repaired extent references for 238026752 ref mismatch on [238043136 16384] extent item 0, found 1 tree backref 238043136 parent 236814336 not found in extent tree backpointer mismatch on [238043136 16384] adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 Repaired extent references for 238043136 ref mismatch on [238059520 16384] extent item 0, found 1 tree backref 238059520 parent 236814336 not found in extent tree backpointer mismatch on [238059520 16384] adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336 Thanks, Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-23 20:12 ` Marc MERLIN @ 2022-04-23 20:53 ` Josef Bacik 2022-04-24 16:20 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-23 20:53 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote: > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote: > > > Now if we get to Monday and it's still running I can take a crack at > > > making it faster. I was hoping it would only take a day or two, but > > > we're balancing me trying to make it better and possibly fucking it up > > > with letting it take the rest of our lives but be correct. Thanks, > > > > Makes sense. I don't need faster, and it may not be able to go faster > > anyway, it's a lot of data. Just wanted to make sure the output and > > relative slow results were expected. > > Looking at the output, is there any way I can figure out if it's at 5% > or 80% completion? > > tree backref 238026752 parent 236814336 not found in extent tree > backpointer mismatch on [238026752 16384] > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 > Repaired extent references for 238026752 > ref mismatch on [238043136 16384] extent item 0, found 1 > tree backref 238043136 parent 236814336 not found in extent tree > backpointer mismatch on [238043136 16384] > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 > Repaired extent references for 238043136 > ref mismatch on [238059520 16384] extent item 0, found 1 > tree backref 238059520 parent 236814336 not found in extent tree > backpointer mismatch on [238059520 16384] > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336 > Hmm I don't know, that's byte 227, but you're in fs trees now, so hopefully soon? I've got some free time, let me rewrite this to be less stupid and see if I can get it done before your thing finishes, and I'll add some sort of progress indicator. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-23 20:53 ` Josef Bacik @ 2022-04-24 16:20 ` Josef Bacik 2022-04-24 16:24 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 16:20 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote: > > > > Now if we get to Monday and it's still running I can take a crack at > > > > making it faster. I was hoping it would only take a day or two, but > > > > we're balancing me trying to make it better and possibly fucking it up > > > > with letting it take the rest of our lives but be correct. Thanks, > > > > > > Makes sense. I don't need faster, and it may not be able to go faster > > > anyway, it's a lot of data. Just wanted to make sure the output and > > > relative slow results were expected. > > > > Looking at the output, is there any way I can figure out if it's at 5% > > or 80% completion? > > > > tree backref 238026752 parent 236814336 not found in extent tree > > backpointer mismatch on [238026752 16384] > > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 > > Repaired extent references for 238026752 > > ref mismatch on [238043136 16384] extent item 0, found 1 > > tree backref 238043136 parent 236814336 not found in extent tree > > backpointer mismatch on [238043136 16384] > > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 > > Repaired extent references for 238043136 > > ref mismatch on [238059520 16384] extent item 0, found 1 > > tree backref 238059520 parent 236814336 not found in extent tree > > backpointer mismatch on [238059520 16384] > > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336 > > > > Hmm I don't know, that's byte 227, but you're in fs trees now, so > hopefully soon? I've got some free time, let me rewrite this to be > less stupid and see if I can get it done before your thing finishes, > and I'll add some sort of progress indicator. Thanks, > Alright you can kill the command, pull my tree, then run btrfs rescue init-extent-tree <blah> This do the straight re-init of the extent tree without all the looping and extra stuff that fsck does, so hopefully will be faster? Also it keeps track of how many bytes we've processed, as well as how many bytes we may have to process, so you'll get a better feel for the progress. I have a scratch fs here that has like 30gib of metadata with 1tib of data and it took around 30 seconds to run. It has no snapshots, I ran it on a fsstress generated fs with like 5k snapshots and it took about a minute, but with significantly less metadata/data. Once you run this you'll still have to run --repair, but it shouldn't have to touch the extent tree. If you see it messing with the extent tree stop it and let me know what it's complaining about and I'll fix it up. I'm pretty sure my thing is going to work, but your fs is more complex than I can generate on the fly. Also, the indicator may show "processed" as less than "possible" bytes when it moves onto the next root, this is normal for snapshots, if we've already processed a subtree we won't walk down it, so it's fine if you see that. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 16:20 ` Josef Bacik @ 2022-04-24 16:24 ` Marc MERLIN 2022-04-24 17:09 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 16:24 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 12:20:42PM -0400, Josef Bacik wrote: > On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote: > > > > > Now if we get to Monday and it's still running I can take a crack at > > > > > making it faster. I was hoping it would only take a day or two, but > > > > > we're balancing me trying to make it better and possibly fucking it up > > > > > with letting it take the rest of our lives but be correct. Thanks, > > > > > > > > Makes sense. I don't need faster, and it may not be able to go faster > > > > anyway, it's a lot of data. Just wanted to make sure the output and > > > > relative slow results were expected. > > > > > > Looking at the output, is there any way I can figure out if it's at 5% > > > or 80% completion? > > > > > > tree backref 238026752 parent 236814336 not found in extent tree > > > backpointer mismatch on [238026752 16384] > > > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 > > > Repaired extent references for 238026752 > > > ref mismatch on [238043136 16384] extent item 0, found 1 > > > tree backref 238043136 parent 236814336 not found in extent tree > > > backpointer mismatch on [238043136 16384] > > > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 > > > Repaired extent references for 238043136 > > > ref mismatch on [238059520 16384] extent item 0, found 1 > > > tree backref 238059520 parent 236814336 not found in extent tree > > > backpointer mismatch on [238059520 16384] > > > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336 > > > > > > > Hmm I don't know, that's byte 227, but you're in fs trees now, so > > hopefully soon? I've got some free time, let me rewrite this to be > > less stupid and see if I can get it done before your thing finishes, > > and I'll add some sort of progress indicator. Thanks, > > > > Alright you can kill the command, pull my tree, then run > > btrfs rescue init-extent-tree <blah> Thanks. Tried the new version gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55bf50e98bc0 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55bf50e98bc0 Walking all our trees and pinning down the currently accessible blocks checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 (...) checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 Clearing the extent root and re-init'ing the block groups processed 1556480 of 0 possible bytes parent transid verify failed on 58736640 wanted 1619007 found 1619009 processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58753024 wanted 1619007 found 1619009 parent transid verify failed on 58753024 wanted 1619007 found 1619009 processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 parent transid verify failed on 58736640 wanted 1619007 found 1619009 corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad key order, prev (169143697408 192 1073741824) current (12582912 192 8388608) kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err` triggered, value -1 ./btrfs(+0x29683)[0x55bf4fea1683] ./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061] ./btrfs(+0x8ba18)[0x55bf4ff03a18] ./btrfs(+0x8bb27)[0x55bf4ff03b27] ./btrfs(+0x8bd8f)[0x55bf4ff03d8f] ./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092] ./btrfs(+0x83931)[0x55bf4fefb931] ./btrfs(handle_command_group+0x49)[0x55bf4fe9017b] ./btrfs(main+0x94)[0x55bf4fe90275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd] ./btrfs(_start+0x2a)[0x55bf4fe8fe1a] Aborted Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 16:24 ` Marc MERLIN @ 2022-04-24 17:09 ` Josef Bacik 2022-04-24 18:43 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 17:09 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 12:24 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 12:20:42PM -0400, Josef Bacik wrote: > > On Sat, Apr 23, 2022 at 4:53 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > > > On Sat, Apr 23, 2022 at 4:12 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > On Fri, Apr 22, 2022 at 01:01:15PM -0700, Marc MERLIN wrote: > > > > > > Now if we get to Monday and it's still running I can take a crack at > > > > > > making it faster. I was hoping it would only take a day or two, but > > > > > > we're balancing me trying to make it better and possibly fucking it up > > > > > > with letting it take the rest of our lives but be correct. Thanks, > > > > > > > > > > Makes sense. I don't need faster, and it may not be able to go faster > > > > > anyway, it's a lot of data. Just wanted to make sure the output and > > > > > relative slow results were expected. > > > > > > > > Looking at the output, is there any way I can figure out if it's at 5% > > > > or 80% completion? > > > > > > > > tree backref 238026752 parent 236814336 not found in extent tree > > > > backpointer mismatch on [238026752 16384] > > > > adding new tree backref on start 238026752 len 16384 parent 236814336 root 236814336 > > > > Repaired extent references for 238026752 > > > > ref mismatch on [238043136 16384] extent item 0, found 1 > > > > tree backref 238043136 parent 236814336 not found in extent tree > > > > backpointer mismatch on [238043136 16384] > > > > adding new tree backref on start 238043136 len 16384 parent 236814336 root 236814336 > > > > Repaired extent references for 238043136 > > > > ref mismatch on [238059520 16384] extent item 0, found 1 > > > > tree backref 238059520 parent 236814336 not found in extent tree > > > > backpointer mismatch on [238059520 16384] > > > > adding new tree backref on start 238059520 len 16384 parent 236814336 root 236814336 > > > > > > > > > > Hmm I don't know, that's byte 227, but you're in fs trees now, so > > > hopefully soon? I've got some free time, let me rewrite this to be > > > less stupid and see if I can get it done before your thing finishes, > > > and I'll add some sort of progress indicator. Thanks, > > > > > > > Alright you can kill the command, pull my tree, then run > > > > btrfs rescue init-extent-tree <blah> > > Thanks. Tried the new version > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 > FS_INFO IS 0x55bf50e98bc0 > checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 > Couldn't find the last root for 8 > FS_INFO AFTER IS 0x55bf50e98bc0 > Walking all our trees and pinning down the currently accessible blocks > checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d > checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 > checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 > checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 > (...) > checksum verify failed on 13577821011968 wanted 0x9a29aff5 found > 0x2cdff391 > checksum verify failed on 15645781196800 wanted 0xef669b11 found > 0x46985a93 > Clearing the extent root and re-init'ing the block groups > processed 1556480 of 0 possible bytes > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > processed 16384 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > processed 32768 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58753024 wanted 1619007 found 1619009 > parent transid verify failed on 58753024 wanted 1619007 found 1619009 > processed 49152 of 0 possible bytesparent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > parent transid verify failed on 58736640 wanted 1619007 found 1619009 > corrupt leaf: root=2 block=58736640 physical=1140867072 slot=107, bad > key order, prev (169143697408 192 1073741824) current (12582912 192 > 8388608) > kernel-shared/extent-tree.c:1287: btrfs_inc_extent_ref: BUG_ON `err` > triggered, value -1 > ./btrfs(+0x29683)[0x55bf4fea1683] > ./btrfs(btrfs_inc_extent_ref+0x522)[0x55bf4fea3061] > ./btrfs(+0x8ba18)[0x55bf4ff03a18] > ./btrfs(+0x8bb27)[0x55bf4ff03b27] > ./btrfs(+0x8bd8f)[0x55bf4ff03d8f] > ./btrfs(btrfs_init_extent_tree+0x226)[0x55bf4ff04092] > ./btrfs(+0x83931)[0x55bf4fefb931] > ./btrfs(handle_command_group+0x49)[0x55bf4fe9017b] > ./btrfs(main+0x94)[0x55bf4fe90275] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7faebe73a7fd] > ./btrfs(_start+0x2a)[0x55bf4fe8fe1a] > Aborted Argh you still have bogus stuff for some reason, maybe it's because of cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a couple of times to make sure it's stopped complaining. If it does complain after the first run please let me know, it really should fix everything the first go around. If it's not then I need to try and reproduce that locally and figure out wtf it's doing. Once tree-recover runs cleanly you can run btrfs rescue init-extent-tree, then hopefully you'll be good to go. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 17:09 ` Josef Bacik @ 2022-04-24 18:43 ` Marc MERLIN 2022-04-24 19:17 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 18:43 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 01:09:45PM -0400, Josef Bacik wrote: > Argh you still have bogus stuff for some reason, maybe it's because of > cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a > couple of times to make sure it's stopped complaining. If it does > complain after the first run please let me know, it really should fix > everything the first go around. If it's not then I need to try and > reproduce that locally and figure out wtf it's doing. Once > tree-recover runs cleanly you can run btrfs rescue init-extent-tree, > then hopefully you'll be good to go. Thanks, gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 FS_INFO IS 0x555a360efbc0 Couldn't find the last root for 8 FS_INFO AFTER IS 0x555a360efbc0 Checking root 2 Checking root 4 Checking root 5 Checking root 7 Checking root 9 Checking root 11221 Checking root 11222 Checking root 11223 Checking root 11224 Checking root 159785 Checking root 159787 Checking root 160494 Checking root 160496 Checking root 161197 Checking root 161199 Checking root 162628 Checking root 162632 Checking root 162645 Checking root 163298 Checking root 163302 Checking root 163303 Checking root 163316 Checking root 163318 Checking root 163916 Checking root 163920 Checking root 163921 Checking root 164620 Checking root 164624 Checking root 164633 Checking root 165098 Checking root 165100 Checking root 165198 Checking root 165200 Checking root 165294 Checking root 165298 Checking root 165299 Checking root 18446744073709551607 Tree recovery finished, you can run check now gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x56080e544bc0 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Couldn't find the last root for 8 FS_INFO AFTER IS 0x56080e544bc0 Walking all our trees and pinning down the currently accessible blocks checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed (...) checksum verify failed on 13577810395136 wanted 0x4171ba0a found 0xda4963ab checksum verify failed on 46576713728 wanted 0xbb1ee48e found 0x8809f6a6 checksum verify failed on 13577126330368 wanted 0xf7264639 found 0x8287ec94 checksum verify failed on 15645432397824 wanted 0xc706f8b6 found 0xf0425483 checksum verify failed on 15645954080768 wanted 0x3ff10ddf found 0x77ab2f14 checksum verify failed on 15645954129920 wanted 0x085fc24a found 0x941acdde checksum verify failed on 15645954195456 wanted 0x6c1c72a4 found 0xb06d5dd9 checksum verify failed on 15645262348288 wanted 0x433325c8 found 0x81ea056f checksum verify failed on 15645608116224 wanted 0x31706547 found 0x314fad1d checksum verify failed on 15645262413824 wanted 0x3affbd35 found 0x96d53d25 checksum verify failed on 15646053974016 wanted 0x98bd62bc found 0xdaa7dd44 checksum verify failed on 15646006722560 wanted 0x760a06a0 found 0xb44fcd95 checksum verify failed on 15645980491776 wanted 0xe1e674a7 found 0x0fdfda2c checksum verify failed on 15645526884352 wanted 0xc2d409c1 found 0xb3eee8d2 checksum verify failed on 12511656394752 wanted 0x607d7b9e found 0xec111bf5 checksum verify failed on 13577013936128 wanted 0x4ba00b03 found 0x64614751 checksum verify failed on 13577199878144 wanted 0x6e9e8bc6 found 0x61063457 checksum verify failed on 13577399156736 wanted 0x2869b8c7 found 0xbb1119e1 checksum verify failed on 12512437698560 wanted 0xca43b3f8 found 0xd7f6db69 checksum verify failed on 13577503686656 wanted 0xd81b7702 found 0x95a3c9a6 checksum verify failed on 15645781344256 wanted 0xb81e3df4 found 0xb5c70846 checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16 checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 Clearing the extent root and re-init'ing the block groups processed 1556480 of 0 possible bytes processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144] Segmentation fault Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559add0 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 #1 0x000055555559afd4 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640) at common/extent-cache.c:179 #2 0x0000555555584b67 in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584d9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x000055555558577d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976 #5 0x000055555557bbaf in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2222 #6 0x000055555557ed74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, iref=0xffffffffffffffe4, path=0x555555b82ed0, root=<optimized out>) at kernel-shared/extent-tree.c:1084 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434045, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b82ed0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1196 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261 #9 0x00005555555dfa18 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcb20, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499 #10 0x00005555555dfb27 in process_eb (trans=trans@entry=0x555558a89680, root=root@entry=0x55555564cde0, eb=0x55555566b790, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574 #11 0x00005555555dfd8f in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e0092 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:779 #13 0x00005555555d7931 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 18:43 ` Marc MERLIN @ 2022-04-24 19:17 ` Josef Bacik 2022-04-24 19:44 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 19:17 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 2:43 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 01:09:45PM -0400, Josef Bacik wrote: > > Argh you still have bogus stuff for some reason, maybe it's because of > > cancelling the rescue. Re-run btrfs rescue tree-recover, and do it a > > couple of times to make sure it's stopped complaining. If it does > > complain after the first run please let me know, it really should fix > > everything the first go around. If it's not then I need to try and > > reproduce that locally and figure out wtf it's doing. Once > > tree-recover runs cleanly you can run btrfs rescue init-extent-tree, > > then hopefully you'll be good to go. Thanks, > > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue tree-recover /dev/mapper/dshelf1 > FS_INFO IS 0x555a360efbc0 > Couldn't find the last root for 8 > FS_INFO AFTER IS 0x555a360efbc0 > Checking root 2 > Checking root 4 > Checking root 5 > Checking root 7 > Checking root 9 > Checking root 11221 > Checking root 11222 > Checking root 11223 > Checking root 11224 > Checking root 159785 > Checking root 159787 > Checking root 160494 > Checking root 160496 > Checking root 161197 > Checking root 161199 > Checking root 162628 > Checking root 162632 > Checking root 162645 > Checking root 163298 > Checking root 163302 > Checking root 163303 > Checking root 163316 > Checking root 163318 > Checking root 163916 > Checking root 163920 > Checking root 163921 > Checking root 164620 > Checking root 164624 > Checking root 164633 > Checking root 165098 > Checking root 165100 > Checking root 165198 > Checking root 165200 > Checking root 165294 > Checking root 165298 > Checking root 165299 > Checking root 18446744073709551607 > Tree recovery finished, you can run check now > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs rescue init-extent-tree /dev/mapper/dshelf1 > FS_INFO IS 0x56080e544bc0 > checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Ok this is sort of maddening, IDK how we'll trip over this broken block here, but not with tree-recover. Can you repull, build, and then run init-extent-tree. As soon as you see the JOSEF message you can kill it and send me the output, I need to figure out wtf is going on here. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 19:17 ` Josef Bacik @ 2022-04-24 19:44 ` Marc MERLIN 2022-04-24 20:01 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 19:44 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 03:17:33PM -0400, Josef Bacik wrote: > Ok this is sort of maddening, IDK how we'll trip over this broken > block here, but not with tree-recover. Can you repull, build, and > then run init-extent-tree. As soon as you see the JOSEF message you > can kill it and send me the output, I need to figure out wtf is going > on here. Thanks, Mmmh, not sure I got that far. gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF * Binary file btrfs matches Binary file btrfsck matches Binary file btrfs-convert matches Binary file btrfs-corrupt-block matches Binary file btrfs-find-root matches Binary file btrfs-image matches Binary file btrfs-map-logical matches Binary file btrfs-select-super matches Binary file btrfstune matches kernel-shared/ctree.c: printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid); Binary file kernel-shared/ctree.o matches kernel-shared/disk-io.c: printf("JOSEF: root %llu\n", root->root_key.objectid); Binary file kernel-shared/disk-io.o matches Binary file mkfs.btrfs matches (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x89f5cb13 found 0x04196d22 FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 (...) checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 Clearing the extent root and re-init'ing the block groups processed 1556480 of 0 possible bytes processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 #1 0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640) at common/extent-cache.c:179 #2 0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976 #5 0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224 #6 0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, iref=0xffffffffffffffe4, path=0x555555b83090, root=<optimized out>) at kernel-shared/extent-tree.c:1084 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b83090, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1196 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261 #9 0x00005555555dfaad in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcfa0, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499 #10 0x00005555555dfbbc in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574 #11 0x00005555555dfe24 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e0127 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:779 #13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 19:44 ` Marc MERLIN @ 2022-04-24 20:01 ` Josef Bacik 2022-04-24 20:31 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 20:01 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 3:44 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 03:17:33PM -0400, Josef Bacik wrote: > > Ok this is sort of maddening, IDK how we'll trip over this broken > > block here, but not with tree-recover. Can you repull, build, and > > then run init-extent-tree. As soon as you see the JOSEF message you > > can kill it and send me the output, I need to figure out wtf is going > > on here. Thanks, > > Mmmh, not sure I got that far. > > gargamel:/var/local/src/btrfs-progs-josefbacik# grep -r JOSEF * > Binary file btrfs matches > Binary file btrfsck matches > Binary file btrfs-convert matches > Binary file btrfs-corrupt-block matches > Binary file btrfs-find-root matches > Binary file btrfs-image matches > Binary file btrfs-map-logical matches > Binary file btrfs-select-super matches > Binary file btrfstune matches > kernel-shared/ctree.c: printf("JOSEF: read node slot on root %llu\n", root->root_key.objectid); > Binary file kernel-shared/ctree.o matches > kernel-shared/disk-io.c: printf("JOSEF: root %llu\n", root->root_key.objectid); > Binary file kernel-shared/disk-io.o matches > Binary file mkfs.btrfs matches > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > FS_INFO IS 0x55555564cbc0 > JOSEF: root 9 Huh ok, it's the UUID tree, weird. I pushed, can you re-run tree-recover, you can stop it after it does root 9, I just want to see what bytenr it thinks the root node is at. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 20:01 ` Josef Bacik @ 2022-04-24 20:31 ` Marc MERLIN 2022-04-24 20:32 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 20:31 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote: > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > FS_INFO IS 0x55555564cbc0 > > JOSEF: root 9 > > Huh ok, it's the UUID tree, weird. I pushed, can you re-run > tree-recover, you can stop it after it does root 9, I just want to see > what bytenr it thinks the root node is at. Thanks, (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 checksum verify failed on 364970688512 wanted 0x33a82891 found 0x154e33ed checksum verify failed on 15645178052608 wanted 0x4bb259dd found 0x4668121c checksum verify failed on 15645178363904 wanted 0xba219c04 found 0x86a9d7f0 checksum verify failed on 15645917708288 wanted 0x4dab2011 found 0xb6299718 checksum verify failed on 11970896232448 wanted 0x8b800530 found 0x3790ebce checksum verify failed on 8733701898240 wanted 0x80eed7c5 found 0xd6af2972 (..) checksum verify failed on 13577178939392 wanted 0x2cb83118 found 0x63f0b6bf checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16 checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 Clearing the extent root and re-init'ing the block groups processed 1556480 of 0 possible bytes processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd7f0, comp=comp@entry=0x55555559ae58 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd808) at common/rbtree-utils.c:48 #1 0x000055555559b05c in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99696640) at common/extent-cache.c:179 #2 0x0000555555584bea in set_extent_bits (tree=0x10000000060, start=99696640, end=99696639, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e1d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585800 in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976 #5 0x000055555557bbf3 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224 #6 0x000055555557edf7 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, iref=0xffffffffffffffe4, path=0x555555b928c0, root=<optimized out>) at kernel-shared/extent-tree.c:1084 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434176, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b928c0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1196 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261 #9 0x00005555555dfb31 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfcbe0, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:499 #10 0x00005555555dfc40 in process_eb (trans=trans@entry=0x555558a896e0, root=root@entry=0x55555564cde0, eb=0x55555566b790, current=current@entry=0x7fffffffdb88) at cmds/rescue-init-extent-tree.c:574 #11 0x00005555555dfea8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:645 #12 0x00005555555e01ab in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1d0 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:779 #13 0x00005555555d79c6 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 20:31 ` Marc MERLIN @ 2022-04-24 20:32 ` Josef Bacik 2022-04-24 20:54 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 20:32 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote: > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > > [Thread debugging using libthread_db enabled] > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > > FS_INFO IS 0x55555564cbc0 > > > JOSEF: root 9 > > > > Huh ok, it's the UUID tree, weird. I pushed, can you re-run > > tree-recover, you can stop it after it does root 9, I just want to see > > what bytenr it thinks the root node is at. Thanks, > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 Sorry, I need tree-recover, not init-extent-tree. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 20:32 ` Josef Bacik @ 2022-04-24 20:54 ` Marc MERLIN 2022-04-24 21:01 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 20:54 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 04:32:23PM -0400, Josef Bacik wrote: > On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote: > > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > > > [Thread debugging using libthread_db enabled] > > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > > > FS_INFO IS 0x55555564cbc0 > > > > JOSEF: root 9 > > > > > > Huh ok, it's the UUID tree, weird. I pushed, can you re-run > > > tree-recover, you can stop it after it does root 9, I just want to see > > > what bytenr it thinks the root node is at. Thanks, > > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > Sorry, I need tree-recover, not init-extent-tree. Thanks, gdb) run rescue tree-recover /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55555564cbc0 Checking root 2 bytenr 67387392 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 13577819963392 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576862547968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 6147) exited normally] (gdb) -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 20:54 ` Marc MERLIN @ 2022-04-24 21:01 ` Josef Bacik [not found] ` <20220424210732.GC29107@merlins.org> 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 21:01 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 4:54 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 04:32:23PM -0400, Josef Bacik wrote: > > On Sun, Apr 24, 2022 at 4:31 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Sun, Apr 24, 2022 at 04:01:34PM -0400, Josef Bacik wrote: > > > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > > > > [Thread debugging using libthread_db enabled] > > > > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > > > > FS_INFO IS 0x55555564cbc0 > > > > > JOSEF: root 9 > > > > > > > > Huh ok, it's the UUID tree, weird. I pushed, can you re-run > > > > tree-recover, you can stop it after it does root 9, I just want to see > > > > what bytenr it thinks the root node is at. Thanks, > > > > > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > > > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > > > > Sorry, I need tree-recover, not init-extent-tree. Thanks, > > gdb) run rescue tree-recover /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue > tree-recover /dev/mapper/dshelf1 Wtf, it's reading the right bytenr, but it's not failing here but failing when we do the init-extent-tree. I've pushed something again to force reads, maybe that's the problem, can you run tree-recover and then init-extent-tree again? Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
[parent not found: <20220424210732.GC29107@merlins.org>]
[parent not found: <CAEzrpqcMV+paWShgAnF8d9WaSQ1Fd5R_DZPRQp-+VNsJGDoASg@mail.gmail.com>]
[parent not found: <20220424212058.GD29107@merlins.org>]
[parent not found: <CAEzrpqcBvh0MC6WeXQ+-80igZhg6t68OcgZnKi6xu+r=njifeA@mail.gmail.com>]
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) [not found] ` <CAEzrpqcBvh0MC6WeXQ+-80igZhg6t68OcgZnKi6xu+r=njifeA@mail.gmail.com> @ 2022-04-24 22:38 ` Marc MERLIN 2022-04-24 22:56 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 22:38 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 05:26:11PM -0400, Josef Bacik wrote: > On Sun, Apr 24, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Sun, Apr 24, 2022 at 05:13:22PM -0400, Josef Bacik wrote: > > > On Sun, Apr 24, 2022 at 5:07 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > On Sun, Apr 24, 2022 at 05:01:38PM -0400, Josef Bacik wrote: > > > > > Wtf, it's reading the right bytenr, but it's not failing here but > > > > > failing when we do the init-extent-tree. I've pushed something again > > > > > to force reads, maybe that's the problem, can you run tree-recover and > > > > > then init-extent-tree again? Thanks, > > > > > > > > off-list: > > > > Seeing this, that kind of worried me so I stopped it. Is it ok to > > > > continue? > > > > > > Sorry, my printf was wrong, pushed. Thanks, > > > > Better? > > Do I worry about those new "had to be read from a different mirror" ? > > > > Ooooh ok, the other mirror is bad, but it finds the right thing. Ok > that makes sense. I pushed another patch just to make sure. Once > this is all done you'll need to run a scrub to fix the mirrors but > this is fine. Thanks, No more crash, but: ERROR: Error adding block group ERROR: commit_root already set when starting transaction (gdb) run rescue tree-recover /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue tree-recover /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 block 15645878108160 had to be read from a different mirror, ret 0 Couldn't find the last root for 8 block 58720256 had to be read from a different mirror, ret 0 block 58720256 had to be read from a different mirror, ret 0 FS_INFO AFTER IS 0x55555564cbc0 block 13577220816896 had to be read from a different mirror, ret 0 block 13577870934016 had to be read from a different mirror, ret 0 (...) block 13577821011968 had to be read from a different mirror, ret 0 block 15645781196800 had to be read from a different mirror, ret 0 Checking root 2 bytenr 67387392 block 58720256 had to be read from a different mirror, ret 0 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 block 11970891186176 had to be read from a different mirror, ret 0 block 12511729680384 had to be read from a different mirror, ret 0 (...) block 13577726541824 had to be read from a different mirror, ret 0 block 13577776611328 had to be read from a different mirror, ret 0 block 364637798400 had to be read from a different mirror, ret 0 block 364637962240 had to be read from a different mirror, ret 0 Checking root 7 bytenr 13577819963392 block 210337792 had to be read from a different mirror, ret 0 block 595607552 had to be read from a different mirror, ret 0 block 949698560 had to be read from a different mirror, ret 0 (...) Checking root 165299 bytenr 13577191505920 block 364679708672 had to be read from a different mirror, ret 0 block 364713902080 had to be read from a different mirror, ret 0 (...) block 13577821011968 had to be read from a different mirror, ret 0 block 15645781196800 had to be read from a different mirror, ret 0 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 5587) exited normally] (gdb) quit Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 block 15645878108160 had to be read from a different mirror, ret 0 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 (...) block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks checksum verify failed on 11651824091136 wanted 0x6d411825 found 0x3cf07c9d block 11651824091136 had to be read from a different mirror, ret 0 checksum verify failed on 606126080 wanted 0x8e0fb704 found 0xfc183188 block 606126080 had to be read from a different mirror, ret 0 checksum verify failed on 15645807640576 wanted 0xe97841cd found 0x4fa14858 block 15645807640576 had to be read from a different mirror, ret 0 checksum verify failed on 364863324160 wanted 0x741855d8 found 0x5aec3f82 block 364863324160 had to be read from a different mirror, ret 0 (...) checksum verify failed on 15645419667456 wanted 0xde0dab28 found 0x3ceddd16 block 15645419667456 had to be read from a different mirror, ret 0 checksum verify failed on 13577821011968 wanted 0x9a29aff5 found 0x2cdff391 block 13577821011968 had to be read from a different mirror, ret 0 checksum verify failed on 15645781196800 wanted 0xef669b11 found 0x46985a93 block 15645781196800 had to be read from a different mirror, ret 0 Clearing the extent root and re-init'ing the block groups parent transid verify failed on 29442048 wanted 1619017 found 1619015 parent transid verify failed on 29442048 wanted 1619017 found 1619015 parent transid verify failed on 29442048 wanted 1619017 found 1619015 parent transid verify failed on 29556736 wanted 1619017 found 1619013 parent transid verify failed on 29556736 wanted 1619017 found 1619013 parent transid verify failed on 29556736 wanted 1619017 found 1619013 parent transid verify failed on 29589504 wanted 1619017 found 1619013 parent transid verify failed on 29589504 wanted 1619017 found 1619013 parent transid verify failed on 29589504 wanted 1619017 found 1619013 ERROR: Error adding block group ERROR: commit_root already set when starting transaction WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 extent buffer leak: start 67469312 len 16384 extent buffer leak: start 29540352 len 16384 extent buffer leak: start 29540352 len 16384 WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 extent buffer leak: start 29655040 len 16384 WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 extent buffer leak: start 29589504 len 16384 extent buffer leak: start 29589504 len 16384 WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 extent buffer leak: start 34504704 len 16384 WARNING: dirty eb leak (aborted trans): start 34504704 len 16384 Init extent tree failed [Inferior 1 (process 11217) exited with code 0357] -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 22:38 ` Marc MERLIN @ 2022-04-24 22:56 ` Josef Bacik 2022-04-24 23:14 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 22:56 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 6:38 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 05:26:11PM -0400, Josef Bacik wrote: > > On Sun, Apr 24, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Sun, Apr 24, 2022 at 05:13:22PM -0400, Josef Bacik wrote: > > > > On Sun, Apr 24, 2022 at 5:07 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > > > On Sun, Apr 24, 2022 at 05:01:38PM -0400, Josef Bacik wrote: > > > > > > Wtf, it's reading the right bytenr, but it's not failing here but > > > > > > failing when we do the init-extent-tree. I've pushed something again > > > > > > to force reads, maybe that's the problem, can you run tree-recover and > > > > > > then init-extent-tree again? Thanks, > > > > > > > > > > off-list: > > > > > Seeing this, that kind of worried me so I stopped it. Is it ok to > > > > > continue? > > > > > > > > Sorry, my printf was wrong, pushed. Thanks, > > > > > > Better? > > > Do I worry about those new "had to be read from a different mirror" ? > > > > > > > Ooooh ok, the other mirror is bad, but it finds the right thing. Ok > > that makes sense. I pushed another patch just to make sure. Once > > this is all done you'll need to run a scrub to fix the mirrors but > > this is fine. Thanks, > > No more crash, but: > ERROR: Error adding block group > ERROR: commit_root already set when starting transaction > I feel like this thing is purposefully changing itself between each run so I can't get a grasp on wtf is going on. I pushed some stuff, lets see how that goes. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 22:56 ` Josef Bacik @ 2022-04-24 23:14 ` Marc MERLIN 2022-04-24 23:27 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-24 23:14 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote: > I feel like this thing is purposefully changing itself between each > run so I can't get a grasp on wtf is going on. I pushed some stuff, > lets see how that goes. Thanks, After all the tests we did, is it possible that some damaged the FS further? Either way: Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 block 15645878108160 had to be read from a different mirror, ret 0 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 block 58720256 had to be read from a different mirror, ret 0 (...) FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks block 11651824091136 had to be read from a different mirror, ret 0 block 606126080 had to be read from a different mirror, ret 0 block 15645807640576 had to be read from a different mirror, ret 0 block 364863324160 had to be read from a different mirror, ret 0 block 364970688512 had to be read from a different mirror, ret 0 block 15645178052608 had to be read from a different mirror, ret 0 (...) inserting block group 352753549312 inserting block group 353827291136 Ignoring transid failure Ignoring transid failure ERROR: Error adding block group -17 ERROR: commit_root already set when starting transaction WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 extent buffer leak: start 67469312 len 16384 extent buffer leak: start 29540352 len 16384 WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 extent buffer leak: start 29589504 len 16384 WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 extent buffer leak: start 29655040 len 16384 WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 Init extent tree failed [Inferior 1 (process 1264) exited with code 0357] -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 23:14 ` Marc MERLIN @ 2022-04-24 23:27 ` Josef Bacik 2022-04-25 0:24 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-24 23:27 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote: > > I feel like this thing is purposefully changing itself between each > > run so I can't get a grasp on wtf is going on. I pushed some stuff, > > lets see how that goes. Thanks, > > After all the tests we did, is it possible that some damaged the FS > further? > That's the thing, we're literally deleting the entire tree and starting over, it should do the same thing every time. I pushed another fix, I think I've been messing up the buffers and that's why we're getting random values. Lets try this again, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-24 23:27 ` Josef Bacik @ 2022-04-25 0:24 ` Marc MERLIN 2022-04-25 0:36 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-25 0:24 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 07:27:02PM -0400, Josef Bacik wrote: > On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote: > > > I feel like this thing is purposefully changing itself between each > > > run so I can't get a grasp on wtf is going on. I pushed some stuff, > > > lets see how that goes. Thanks, > > > > After all the tests we did, is it possible that some damaged the FS > > further? > > > > That's the thing, we're literally deleting the entire tree and > starting over, it should do the same thing every time. I pushed > another fix, I think I've been messing up the buffers and that's why > we're getting random values. Lets try this again, Gotcha. (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 (...) checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups inserting block group 12582912 inserting block group 20971520 inserting block group 29360128 inserting block group 1103101952 (...) inserting block group 345237356544 inserting block group 346311098368 inserting block group 347384840192 inserting block group 348458582016 inserting block group 349532323840 inserting block group 350606065664 inserting block group 351679807488 inserting block group 352753549312 inserting block group 353827291136 Ignoring transid failure Ignoring transid failure ERROR: Error adding block group -17 ERROR: commit_root already set when starting transaction WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 extent buffer leak: start 67469312 len 16384 extent buffer leak: start 29540352 len 16384 WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 extent buffer leak: start 29589504 len 16384 WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 extent buffer leak: start 29655040 len 16384 WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 Init extent tree failed [Inferior 1 (process 6259) exited with code 0357] (gdb) I ran it a second time and got the same output Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-25 0:24 ` Marc MERLIN @ 2022-04-25 0:36 ` Josef Bacik 2022-04-26 0:28 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-25 0:36 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 8:24 PM Marc MERLIN <marc@merlins.org> wrote: > > On Sun, Apr 24, 2022 at 07:27:02PM -0400, Josef Bacik wrote: > > On Sun, Apr 24, 2022 at 7:14 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Sun, Apr 24, 2022 at 06:56:01PM -0400, Josef Bacik wrote: > > > > I feel like this thing is purposefully changing itself between each > > > > run so I can't get a grasp on wtf is going on. I pushed some stuff, > > > > lets see how that goes. Thanks, > > > > > > After all the tests we did, is it possible that some damaged the FS > > > further? > > > > > > > That's the thing, we're literally deleting the entire tree and > > starting over, it should do the same thing every time. I pushed > > another fix, I think I've been messing up the buffers and that's why > > we're getting random values. Lets try this again, > > Gotcha. > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > FS_INFO IS 0x55555564cbc0 > JOSEF: root 9 > checksum verify failed on 15645878108160 wanted 0x1beaa67b found 0x27edb2c4 > Couldn't find the last root for 8 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > (...) > checksum verify failed on 58720256 wanted 0x39dcdd2a found 0x79b62995 > FS_INFO AFTER IS 0x55555564cbc0 > Walking all our trees and pinning down the currently accessible blocks > Clearing the extent root and re-init'ing the block groups > inserting block group 12582912 > inserting block group 20971520 > inserting block group 29360128 > inserting block group 1103101952 > (...) > inserting block group 345237356544 > inserting block group 346311098368 > inserting block group 347384840192 > inserting block group 348458582016 > inserting block group 349532323840 > inserting block group 350606065664 > inserting block group 351679807488 > inserting block group 352753549312 > inserting block group 353827291136 > Ignoring transid failure > Ignoring transid failure > ERROR: Error adding block group -17 > ERROR: commit_root already set when starting transaction > WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 > extent buffer leak: start 67469312 len 16384 > extent buffer leak: start 29540352 len 16384 > WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 > extent buffer leak: start 29589504 len 16384 > WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 > extent buffer leak: start 29655040 len 16384 > WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 > Init extent tree failed > [Inferior 1 (process 6259) exited with code 0357] > (gdb) > > > I ran it a second time and got the same output Well thats something at least, I've rigged it up to dump the leaf so I can see wtf is going on here. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-25 0:36 ` Josef Bacik @ 2022-04-26 0:28 ` Marc MERLIN 2022-04-26 20:43 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-26 0:28 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Sun, Apr 24, 2022 at 08:36:42PM -0400, Josef Bacik wrote: > > I ran it a second time and got the same output > > Well thats something at least, I've rigged it up to dump the leaf so I > can see wtf is going on here. Thanks, Sorry for the delay, took a while to run: (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 inserting block group 253969301504 item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single ERROR: Error adding block group -17 ERROR: commit_root already set when starting transaction WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 extent buffer leak: start 67469312 len 16384 extent buffer leak: start 29540352 len 16384 WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 extent buffer leak: start 29589504 len 16384 WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 extent buffer leak: start 29655040 len 16384 WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 Init extent tree failed [Inferior 1 (process 24213) exited with code 0357] -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-26 0:28 ` Marc MERLIN @ 2022-04-26 20:43 ` Marc MERLIN 2022-04-26 21:20 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-26 20:43 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs Generally would you say we're still on the right path and helping your recovery tools getting better, or is it getting close or to the time where I should restore from backups? On Mon, Apr 25, 2022 at 05:28:04PM -0700, Marc MERLIN wrote: > On Sun, Apr 24, 2022 at 08:36:42PM -0400, Josef Bacik wrote: > > > I ran it a second time and got the same output > > > > Well thats something at least, I've rigged it up to dump the leaf so I > > can see wtf is going on here. Thanks, > > Sorry for the delay, took a while to run: > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > FS_INFO IS 0x55555564cbc0 > JOSEF: root 9 > Couldn't find the last root for 8 > inserting block group 253969301504 > item 33 key (328057487360 BLOCK_GROUP_ITEM 1073741824) itemoff 15467 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 173 key (477844471808 BLOCK_GROUP_ITEM 1073741824) itemoff 12107 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 174 key (478918213632 BLOCK_GROUP_ITEM 1073741824) itemoff 12083 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 175 key (479991955456 BLOCK_GROUP_ITEM 1073741824) itemoff 12059 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 176 key (481065697280 BLOCK_GROUP_ITEM 1073741824) itemoff 12035 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 177 key (482139439104 BLOCK_GROUP_ITEM 1073741824) itemoff 12011 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 178 key (483213180928 BLOCK_GROUP_ITEM 1073741824) itemoff 11987 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 179 key (484286922752 BLOCK_GROUP_ITEM 1073741824) itemoff 11963 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 180 key (485360664576 BLOCK_GROUP_ITEM 1073741824) itemoff 11939 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 181 key (486434406400 BLOCK_GROUP_ITEM 1073741824) itemoff 11915 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 182 key (487508148224 BLOCK_GROUP_ITEM 1073741824) itemoff 11891 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 183 key (488581890048 BLOCK_GROUP_ITEM 1073741824) itemoff 11867 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 184 key (489655631872 BLOCK_GROUP_ITEM 1073741824) itemoff 11843 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 185 key (490729373696 BLOCK_GROUP_ITEM 1073741824) itemoff 11819 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 186 key (491803115520 BLOCK_GROUP_ITEM 1073741824) itemoff 11795 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 187 key (492876857344 BLOCK_GROUP_ITEM 1073741824) itemoff 11771 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 188 key (493950599168 BLOCK_GROUP_ITEM 1073741824) itemoff 11747 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 189 key (495024340992 BLOCK_GROUP_ITEM 1073741824) itemoff 11723 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 190 key (496098082816 BLOCK_GROUP_ITEM 1073741824) itemoff 11699 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 191 key (497171824640 BLOCK_GROUP_ITEM 1073741824) itemoff 11675 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 192 key (498245566464 BLOCK_GROUP_ITEM 1073741824) itemoff 11651 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 193 key (499319308288 BLOCK_GROUP_ITEM 1073741824) itemoff 11627 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 194 key (500393050112 BLOCK_GROUP_ITEM 1073741824) itemoff 11603 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 195 key (501466791936 BLOCK_GROUP_ITEM 1073741824) itemoff 11579 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 196 key (502540533760 BLOCK_GROUP_ITEM 1073741824) itemoff 11555 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 197 key (503614275584 BLOCK_GROUP_ITEM 1073741824) itemoff 11531 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 198 key (504688017408 BLOCK_GROUP_ITEM 1073741824) itemoff 11507 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 199 key (505761759232 BLOCK_GROUP_ITEM 1073741824) itemoff 11483 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 200 key (506835501056 BLOCK_GROUP_ITEM 1073741824) itemoff 11459 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 201 key (507909242880 BLOCK_GROUP_ITEM 1073741824) itemoff 11435 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 202 key (508982984704 BLOCK_GROUP_ITEM 1073741824) itemoff 11411 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 203 key (510056726528 BLOCK_GROUP_ITEM 1073741824) itemoff 11387 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 204 key (511130468352 BLOCK_GROUP_ITEM 1073741824) itemoff 11363 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 205 key (512204210176 BLOCK_GROUP_ITEM 1073741824) itemoff 11339 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 206 key (513277952000 BLOCK_GROUP_ITEM 1073741824) itemoff 11315 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 207 key (514351693824 BLOCK_GROUP_ITEM 1073741824) itemoff 11291 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 208 key (515425435648 BLOCK_GROUP_ITEM 1073741824) itemoff 11267 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 209 key (516499177472 BLOCK_GROUP_ITEM 1073741824) itemoff 11243 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 213 key (520794144768 BLOCK_GROUP_ITEM 1073741824) itemoff 11147 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 216 key (524015370240 BLOCK_GROUP_ITEM 1073741824) itemoff 11075 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > item 217 key (525089112064 BLOCK_GROUP_ITEM 1073741824) itemoff 11051 itemsize 24 > block group used 0 chunk_objectid 256 flags DATA|single > ERROR: Error adding block group -17 > ERROR: commit_root already set when starting transaction > WARNING: reserved space leaked, flag=0x4 bytes_reserved=81920 > extent buffer leak: start 67469312 len 16384 > extent buffer leak: start 29540352 len 16384 > WARNING: dirty eb leak (aborted trans): start 29540352 len 16384 > extent buffer leak: start 29589504 len 16384 > WARNING: dirty eb leak (aborted trans): start 29589504 len 16384 > extent buffer leak: start 29655040 len 16384 > WARNING: dirty eb leak (aborted trans): start 29655040 len 16384 > Init extent tree failed > [Inferior 1 (process 24213) exited with code 0357] > > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Home page: http://marc.merlins.org/ > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-26 20:43 ` Marc MERLIN @ 2022-04-26 21:20 ` Josef Bacik 2022-04-26 21:36 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-26 21:20 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > Generally would you say we're still on the right path and helping your > recovery tools getting better, or is it getting close or to the time > where I should restore from backups? > Yup sorry for the radio silence, loads of meetings today, but good news is I've reproduced your problem locally, so I'm trying to hammer it out. I hope to have something useful for you today. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-26 21:20 ` Josef Bacik @ 2022-04-26 21:36 ` Josef Bacik 2022-04-27 3:54 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-26 21:36 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > Generally would you say we're still on the right path and helping your > > recovery tools getting better, or is it getting close or to the time > > where I should restore from backups? > > > > Yup sorry for the radio silence, loads of meetings today, but good > news is I've reproduced your problem locally, so I'm trying to hammer > it out. I hope to have something useful for you today. Thanks, > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to make sure any stupidity I've caused is undone, and then run rescue init-extent-tree and then we can go from there? Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-26 21:36 ` Josef Bacik @ 2022-04-27 3:54 ` Marc MERLIN 2022-04-27 14:44 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 3:54 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote: > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > Generally would you say we're still on the right path and helping your > > > recovery tools getting better, or is it getting close or to the time > > > where I should restore from backups? > > > > > > > Yup sorry for the radio silence, loads of meetings today, but good > > news is I've reproduced your problem locally, so I'm trying to hammer > > it out. I hope to have something useful for you today. Thanks, > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to > make sure any stupidity I've caused is undone, and then run rescue > init-extent-tree and then we can go from there? Thanks, FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55555564cbc0 Checking root 2 bytenr 67387392 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 13577819963392 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576862547968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 28238) exited normally] (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55555564cbc0 Checking root 2 bytenr 67387392 Checking root 4 bytenr 15645196861440 Checking root 5 bytenr 13577660252160 Checking root 7 bytenr 13577819963392 Checking root 9 bytenr 15645878108160 Checking root 11221 bytenr 13577562996736 Checking root 11222 bytenr 15645261905920 Checking root 11223 bytenr 13576862547968 Checking root 11224 bytenr 13577126182912 Checking root 159785 bytenr 6781490577408 Checking root 159787 bytenr 15645908385792 Checking root 160494 bytenr 6781491265536 Checking root 160496 bytenr 11822309965824 Checking root 161197 bytenr 6781492101120 Checking root 161199 bytenr 13576850833408 Checking root 162628 bytenr 15645764812800 Checking root 162632 bytenr 6781492756480 Checking root 162645 bytenr 5809981095936 Checking root 163298 bytenr 15645124263936 Checking root 163302 bytenr 6781495197696 Checking root 163303 bytenr 15645365993472 Checking root 163316 bytenr 6781496393728 Checking root 163318 bytenr 15645980491776 Checking root 163916 bytenr 11822437826560 Checking root 163920 bytenr 11971021275136 Checking root 163921 bytenr 11971073802240 Checking root 164620 bytenr 15645434036224 Checking root 164624 bytenr 15645502210048 Checking root 164633 bytenr 15645526884352 Checking root 165098 bytenr 11970667446272 Checking root 165100 bytenr 11970733621248 Checking root 165198 bytenr 12511656394752 Checking root 165200 bytenr 12511677972480 Checking root 165294 bytenr 13576901328896 Checking root 165298 bytenr 13577133326336 Checking root 165299 bytenr 13577191505920 Checking root 18446744073709551607 bytenr 13576823685120 Tree recovery finished, you can run check now [Inferior 1 (process 28238) exited normally] (gdb) item 210 key (517572919296 BLOCK_GROUP_ITEM 1073741824) itemoff 11219 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 211 key (518646661120 BLOCK_GROUP_ITEM 1073741824) itemoff 11195 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 212 key (519720402944 BLOCK_GROUP_ITEM 1073741824) itemoff 11171 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 214 key (521867886592 BLOCK_GROUP_ITEM 1073741824) itemoff 11123 itemsize 24 block group used 0 chunk_objectid 256 flags DATA|single item 215 key (522941628416 BLOCK_GROUP_ITEM 1073741824) itemoff 11099 itemsize 24 inserting block group 15731991248896 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1228800 of 0 possible bytesFailed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559ae9e <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48 #1 0x000055555559b0a2 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=99713024) at common/extent-cache.c:179 #2 0x0000555555584c30 in set_extent_bits (tree=0x10000000060, start=99713024, end=99713023, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e63 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585846 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976 #5 0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee3d in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, iref=0xffffffffffffffe4, path=0x555555b92bd0, root=<optimized out>) at kernel-shared/extent-tree.c:1084 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434246, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b92bd0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1196 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1261 #9 0x00005555555dfb81 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559cfce80, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:502 #10 0x00005555555dfc90 in process_eb (trans=trans@entry=0x555558a89620, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:577 #11 0x00005555555dfef8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:648 #12 0x00005555555e0202 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:784 #13 0x00005555555d7a0c in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 3:54 ` Marc MERLIN @ 2022-04-27 14:44 ` Josef Bacik 2022-04-27 16:34 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 14:44 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote: > > On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote: > > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > Generally would you say we're still on the right path and helping your > > > > recovery tools getting better, or is it getting close or to the time > > > > where I should restore from backups? > > > > > > > > > > Yup sorry for the radio silence, loads of meetings today, but good > > > news is I've reproduced your problem locally, so I'm trying to hammer > > > it out. I hope to have something useful for you today. Thanks, > > > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to > > make sure any stupidity I've caused is undone, and then run rescue > > init-extent-tree and then we can go from there? Thanks, > Ok back to this problem again, I've added some debugging. Unfortunately the real bytenr is getting lost, so the new debugging will print the actual bytenr we're trying to add, and then I can add more targeted debugging to figure out whats going on. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 14:44 ` Josef Bacik @ 2022-04-27 16:34 ` Marc MERLIN 2022-04-27 17:49 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 16:34 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 10:44:26AM -0400, Josef Bacik wrote: > On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote: > > > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > > > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > > > Generally would you say we're still on the right path and helping your > > > > > recovery tools getting better, or is it getting close or to the time > > > > > where I should restore from backups? > > > > > > > > > > > > > Yup sorry for the radio silence, loads of meetings today, but good > > > > news is I've reproduced your problem locally, so I'm trying to hammer > > > > it out. I hope to have something useful for you today. Thanks, > > > > > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to > > > make sure any stupidity I've caused is undone, and then run rescue > > > init-extent-tree and then we can go from there? Thanks, > > Ok back to this problem again, I've added some debugging. > Unfortunately the real bytenr is getting lost, so the new debugging > will print the actual bytenr we're trying to add, and then I can add > more targeted debugging to figure out whats going on. Thanks, Small typo: 409 | prinf("doing insert of %llu\n", key->objectid); | ^~~~~ | printf Result: doing insert of 12233379401728 Failed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffdd00, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdd18) at common/rbtree-utils.c:48 #1 0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c3a in set_extent_bits (tree=0x10000000060, start=100564992, end=100564991, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585850 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976 #5 0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee47 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, iref=0xffffffffffffffe4, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824992434256, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfba4 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00cc0, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:503 #10 0x00005555555dfcb3 in process_eb (trans=trans@entry=0x555558a8d760, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:578 #11 0x00005555555dff1b in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:649 #12 0x00005555555e0225 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:785 #13 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 16:34 ` Marc MERLIN @ 2022-04-27 17:49 ` Josef Bacik 2022-04-27 18:24 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 17:49 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 12:34 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 10:44:26AM -0400, Josef Bacik wrote: > > On Tue, Apr 26, 2022 at 11:54 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Tue, Apr 26, 2022 at 05:36:28PM -0400, Josef Bacik wrote: > > > > On Tue, Apr 26, 2022 at 5:20 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > > > > > > > On Tue, Apr 26, 2022 at 4:43 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > > > > > Generally would you say we're still on the right path and helping your > > > > > > recovery tools getting better, or is it getting close or to the time > > > > > > where I should restore from backups? > > > > > > > > > > > > > > > > Yup sorry for the radio silence, loads of meetings today, but good > > > > > news is I've reproduced your problem locally, so I'm trying to hammer > > > > > it out. I hope to have something useful for you today. Thanks, > > > > > > > > Sigh I'm dumb as fuck, can you pull and re-run tree-recover just to > > > > make sure any stupidity I've caused is undone, and then run rescue > > > > init-extent-tree and then we can go from there? Thanks, > > > > Ok back to this problem again, I've added some debugging. > > Unfortunately the real bytenr is getting lost, so the new debugging > > will print the actual bytenr we're trying to add, and then I can add > > more targeted debugging to figure out whats going on. Thanks, > > Small typo: > 409 | prinf("doing insert of %llu\n", key->objectid); > | ^~~~~ > | printf > > Result: > doing insert of 12233379401728 > Failed to find [7750833868800, 168, 262144] Oh it's data, interesting. Pushed some more debugging. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 17:49 ` Josef Bacik @ 2022-04-27 18:24 ` Marc MERLIN 2022-04-27 20:21 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 18:24 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote: > > Result: > > doing insert of 12233379401728 > > Failed to find [7750833868800, 168, 262144] > > Oh it's data, interesting. Pushed some more debugging. Thanks, inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? Failed to find [7750833868800, 168, 262144] processed 1245184 of 0 possible bytesIgnoring transid failure kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fbf3)[0x555555573bf3] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a] kernel-shared/ctree.c:1940: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1d13f)[0x55555557113f] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_leaf_free_space+0xd3)[0x555555573b36] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x1fc38)[0x555555573c38] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2083b)[0x55555557483b] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_search_slot+0x11cd)[0x5555555767c9] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_insert_empty_items+0x76)[0x555555576e6b] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8b97d)[0x5555555df97d] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bc04)[0x5555555dfc04] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bcfb)[0x5555555dfcfb] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8bf59)[0x5555555dff59] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0x22d)[0x5555555e0263] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a16)[0x5555555d7a16] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x1000000000060, key=key@entry=0x7fffffffd8f0, comp=comp@entry=0x55555559aea8 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd908) at common/rbtree-utils.c:48 #1 0x000055555559b0ac in search_cache_extent (tree=tree@entry=0x1000000000060, start=start@entry=1) at common/extent-cache.c:179 #2 0x0000555555584c3a in set_extent_bits (tree=0x1000000000060, start=1, end=0, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e6d in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585850 in set_extent_buffer_dirty (eb=0x555555708580) at kernel-shared/extent_io.c:976 #5 0x000055555557bc1e in btrfs_mark_buffer_dirty (eb=<optimized out>) at kernel-shared/disk-io.c:2224 #6 0x00005555555740c3 in push_leaf_right (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, empty=empty@entry=0) at kernel-shared/ctree.c:2106 #7 0x000055555557483b in split_leaf (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, ins_key=ins_key@entry=0x7fffffffdf60, path=path@entry=0x7fffffffde40, data_size=data_size@entry=49, extend=extend@entry=0) at kernel-shared/ctree.c:2385 #8 0x00005555555767c9 in btrfs_search_slot (trans=0x555558a8d700, root=root@entry=0x5555559547d0, key=key@entry=0x7fffffffdf60, p=p@entry=0x7fffffffde40, ins_len=ins_len@entry=49, cow=cow@entry=1) at kernel-shared/ctree.c:1432 #9 0x0000555555576e6b in btrfs_insert_empty_items (trans=trans@entry=0x555558a8d700, root=root@entry=0x5555559547d0, path=path@entry=0x7fffffffde40, cpu_key=cpu_key@entry=0x7fffffffdf60, data_size=data_size@entry=0x7fffffffde3c, nr=nr@entry=1) at kernel-shared/ctree.c:2820 #10 0x00005555555df97d in btrfs_insert_empty_item (data_size=<optimized out>, key=0x7fffffffdf60, path=0x7fffffffde40, root=0x5555559547d0, trans=0x555558a8d700) at ./kernel-shared/ctree.h:2780 #11 insert_empty_extent (trans=trans@entry=0x555558a8d700, key=key@entry=0x7fffffffdf60, generation=generation@entry=1590260, flags=flags@entry=1) at cmds/rescue-init-extent-tree.c:435 #12 0x00005555555dfc04 in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d04fe0, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:508 #13 0x00005555555dfcfb in process_eb (trans=trans@entry=0x555558a8d700, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:594 #14 0x00005555555dff59 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:665 #15 0x00005555555e0263 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:801 #16 0x00005555555d7a16 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #17 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #18 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #19 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #20 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 18:24 ` Marc MERLIN @ 2022-04-27 20:21 ` Josef Bacik 2022-04-27 21:02 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 20:21 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote: > > > Result: > > > doing insert of 12233379401728 > > > Failed to find [7750833868800, 168, 262144] > > > > Oh it's data, interesting. Pushed some more debugging. Thanks, > > inserting block group 15840439173120 > inserting block group 15842586656768 > processed 1556480 of 0 possible bytes > processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? Well crap, you have bytenrs that overlap. I've adjusted init-extent-tree to dump the paths of the files that have overlapping extents, as well as the keys. Run it again so we know what files are fucked. From there you need to tell me which one you don't care about and are willing to delete, and then I'll give you the command you need to remove that bad extent and then we can go again. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 20:21 ` Josef Bacik @ 2022-04-27 21:02 ` Marc MERLIN 2022-04-27 21:11 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 21:02 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 04:21:31PM -0400, Josef Bacik wrote: > On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote: > > > > Result: > > > > doing insert of 12233379401728 > > > > Failed to find [7750833868800, 168, 262144] > > > > > > Oh it's data, interesting. Pushed some more debugging. Thanks, > > > > inserting block group 15840439173120 > > inserting block group 15842586656768 > > processed 1556480 of 0 possible bytes > > processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? > > Well crap, you have bytenrs that overlap. I've adjusted > init-extent-tree to dump the paths of the files that have overlapping > extents, as well as the keys. Run it again so we know what files are > fucked. From there you need to tell me which one you don't care about > and are willing to delete, and then I'll give you the command you need > to remove that bad extent and then we can go again. Thanks, I see. Not sure if that helped. Should I have run it differently? Author: Josef Bacik <josef@toxicpanda.com> Date: Wed Apr 27 16:19:46 2022 -0400 fix some things so dump path works (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 (...) inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] doing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths Failed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 #1 0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976 #5 0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, iref=0xffffffffffffffe3, path=0x555555b8cfd0, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998756304, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cfd0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfcf4 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:555 #10 0x00005555555dfe4d in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:630 #11 0x00005555555e00b8 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:701 #12 0x00005555555e03c2 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:837 #13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 21:02 ` Marc MERLIN @ 2022-04-27 21:11 ` Josef Bacik 2022-04-27 21:20 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 21:11 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 5:02 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 04:21:31PM -0400, Josef Bacik wrote: > > On Wed, Apr 27, 2022 at 2:24 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Wed, Apr 27, 2022 at 01:49:46PM -0400, Josef Bacik wrote: > > > > > Result: > > > > > doing insert of 12233379401728 > > > > > Failed to find [7750833868800, 168, 262144] > > > > > > > > Oh it's data, interesting. Pushed some more debugging. Thanks, > > > > > > inserting block group 15840439173120 > > > inserting block group 15842586656768 > > > processed 1556480 of 0 possible bytes > > > processed 1130496 of 0 possible bytesdoing an insert that overlaps our bytenr 7750833627136 262144 > > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? > > > > Well crap, you have bytenrs that overlap. I've adjusted > > init-extent-tree to dump the paths of the files that have overlapping > > extents, as well as the keys. Run it again so we know what files are > > fucked. From there you need to tell me which one you don't care about > > and are willing to delete, and then I'll give you the command you need > > to remove that bad extent and then we can go again. Thanks, > > I see. > > Not sure if that helped. Should I have run it differently? > > > Author: Josef Bacik <josef@toxicpanda.com> > Date: Wed Apr 27 16:19:46 2022 -0400 > fix some things so dump path works > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > (...) > inserting block group 15838291689472 > inserting block group 15839365431296 > inserting block group 15840439173120 > inserting block group 15842586656768 > processed 1556480 of 0 possible bytes > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] > doing an insert that overlaps our bytenr 7750833627136 262144 > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > Failed to find [7750833868800, 168, 262144] > Of course it doesn't work for you, I pushed some debug stuff. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 21:11 ` Josef Bacik @ 2022-04-27 21:20 ` Marc MERLIN 2022-04-27 21:27 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 21:20 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 05:11:30PM -0400, Josef Bacik wrote: > > inserting block group 15838291689472 > > inserting block group 15839365431296 > > inserting block group 15840439173120 > > inserting block group 15842586656768 > > processed 1556480 of 0 possible bytes > > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] > > doing an insert that overlaps our bytenr 7750833627136 262144 > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > Failed to find [7750833868800, 168, 262144] > > > > Of course it doesn't work for you, I pushed some debug stuff. Thanks, (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks (...) inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] elem_cnt 0 elem_missed 0 ret -2 doing an insert that overlaps our bytenr 7750833627136 262144 processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths elem_cnt 0 elem_missed 0 ret -2 Failed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffdce0, comp=comp@entry=0x55555559ae9a <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffdcf8) at common/rbtree-utils.c:48 #1 0x000055555559b09e in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=100564992) at common/extent-cache.c:179 #2 0x0000555555584c2c in set_extent_bits (tree=0x100000060, start=100564992, end=100564991, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e5f in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585842 in set_extent_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/extent_io.c:976 #5 0x000055555557bc10 in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555729660) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee39 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, iref=0xffffffffffffffe3, path=0x555555b8cda0, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998755744, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8cda0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfd17 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00f00, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:557 #10 0x00005555555dfe70 in process_eb (trans=trans@entry=0x555558a8d6a0, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffe098) at cmds/rescue-init-extent-tree.c:632 #11 0x00005555555e00db in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:703 #12 0x00005555555e03e5 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe6c5 "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:839 #13 0x00005555555d7a08 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffe3c8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffe3c8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffe3c0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffe3c0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 21:20 ` Marc MERLIN @ 2022-04-27 21:27 ` Josef Bacik 2022-04-27 22:59 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 21:27 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 5:20 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 05:11:30PM -0400, Josef Bacik wrote: > > > inserting block group 15838291689472 > > > inserting block group 15839365431296 > > > inserting block group 15840439173120 > > > inserting block group 15842586656768 > > > processed 1556480 of 0 possible bytes > > > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] > > > doing an insert that overlaps our bytenr 7750833627136 262144 > > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > > Failed to find [7750833868800, 168, 262144] > > > > > > > Of course it doesn't work for you, I pushed some debug stuff. Thanks, > > (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 > Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > FS_INFO IS 0x55555564cbc0 > JOSEF: root 9 > Couldn't find the last root for 8 > checksum verify failed on 58720256 wanted 0x0d38525a found 0xb3a707fa > FS_INFO AFTER IS 0x55555564cbc0 > Walking all our trees and pinning down the currently accessible blocks > (...) > inserting block group 15842586656768 > processed 1556480 of 0 possible bytes > processed 1130496 of 0 possible bytesadding a bytenr that overlaps our thing, dumping paths for [5064, 108, 0] > elem_cnt 0 elem_missed 0 ret -2 > doing an insert that overlaps our bytenr 7750833627136 262144 > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > elem_cnt 0 elem_missed 0 ret -2 > Failed to find [7750833868800, 168, 262144] Sigh, added another print_leaf. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 21:27 ` Josef Bacik @ 2022-04-27 22:59 ` Marc MERLIN 2022-04-27 23:02 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-27 22:59 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote: > Sigh, added another print_leaf. Thanks, doing an insert that overlaps our bytenr 7750833627136 262144 processed 1146880 of 0 possible bytes processed 1163264 of 0 possible bytes processed 1179648 of 0 possible bytes processed 1196032 of 0 possible bytes processed 1212416 of 0 possible bytes processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths inode ref info failed??? leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE leaf 15645023322112 flags 0x1(WRITTEN) backref revision 1 fs uuid 96539b8c-ccc9-47bf-9e6c-29305890941e chunk uuid 7257b24b-3702-41e5-8b61-6f6ea524255a item 0 key (5457 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1835008000 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7000 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.690807153 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 1 key (5457 EXTENT_DATA 0) itemoff 16070 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207705702400 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 2 key (5458 INODE_ITEM 0) itemoff 15910 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 46137344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.690807153 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 3 key (5458 EXTENT_DATA 0) itemoff 15857 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207791230976 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 4 key (5459 INODE_ITEM 0) itemoff 15697 itemsize 160 generation 1482901 transid 1482901 size 262144 nbytes 74448896 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 284 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169214.790245505 (2020-10-19 21:46:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 5 key (5459 EXTENT_DATA 0) itemoff 15644 itemsize 53 generation 1482901 type 1 (regular) extent data disk byte 7314895085568 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 6 key (5460 INODE_ITEM 0) itemoff 15484 itemsize 160 generation 1439787 transid 1439787 size 262144 nbytes 398721024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1521 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1591038874.596346316 (2020-06-01 12:14:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 7 key (5460 EXTENT_DATA 0) itemoff 15431 itemsize 53 generation 1439787 type 1 (regular) extent data disk byte 12470912008192 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 8 key (5461 INODE_ITEM 0) itemoff 15271 itemsize 160 generation 1482901 transid 1482901 size 262144 nbytes 25165824 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 96 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169214.790245505 (2020-10-19 21:46:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 9 key (5461 EXTENT_DATA 0) itemoff 15218 itemsize 53 generation 1482901 type 1 (regular) extent data disk byte 7482517606400 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 10 key (5462 INODE_ITEM 0) itemoff 15058 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1549795328 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5912 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.694807100 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 11 key (5462 EXTENT_DATA 0) itemoff 15005 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207794606080 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 12 key (5463 INODE_ITEM 0) itemoff 14845 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 746848256 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2849 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.694807100 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 13 key (5463 EXTENT_DATA 0) itemoff 14792 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207801716736 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 14 key (5464 INODE_ITEM 0) itemoff 14632 itemsize 160 generation 1482589 transid 1482589 size 262144 nbytes 686555136 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2619 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603031991.590888741 (2020-10-18 07:39:51) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 15 key (5464 EXTENT_DATA 0) itemoff 14579 itemsize 53 generation 1482589 type 1 (regular) extent data disk byte 952344715264 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 16 key (5465 INODE_ITEM 0) itemoff 14419 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1620574208 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6182 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.698807046 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 17 key (5465 EXTENT_DATA 0) itemoff 14366 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207808765952 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 18 key (5466 INODE_ITEM 0) itemoff 14206 itemsize 160 generation 1511028 transid 1511028 size 262144 nbytes 777256960 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2965 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1610956842.625245815 (2021-01-18 00:00:42) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 19 key (5466 EXTENT_DATA 0) itemoff 14153 itemsize 53 generation 1511028 type 1 (regular) extent data disk byte 12245962477568 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 20 key (5467 INODE_ITEM 0) itemoff 13993 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1741684736 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6644 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.702806994 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 21 key (5467 EXTENT_DATA 0) itemoff 13940 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207860899840 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 22 key (5468 INODE_ITEM 0) itemoff 13780 itemsize 160 generation 1525970 transid 1525970 size 262144 nbytes 95944704 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 366 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222802.713761123 (2021-03-08 09:00:02) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 23 key (5468 EXTENT_DATA 0) itemoff 13727 itemsize 53 generation 1525970 type 1 (regular) extent data disk byte 7437273346048 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 24 key (5469 INODE_ITEM 0) itemoff 13567 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 3301703680 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12595 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.702806994 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 25 key (5469 EXTENT_DATA 0) itemoff 13514 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12207862214656 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 26 key (5470 INODE_ITEM 0) itemoff 13354 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 2621440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.706806940 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 27 key (5470 EXTENT_DATA 0) itemoff 13301 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12208843333632 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 28 key (5471 INODE_ITEM 0) itemoff 13141 itemsize 160 generation 1482591 transid 1482591 size 262144 nbytes 992477184 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3786 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603032110.61310221 (2020-10-18 07:41:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 29 key (5471 EXTENT_DATA 0) itemoff 13088 itemsize 53 generation 1482591 type 1 (regular) extent data disk byte 3184741777408 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 30 key (5472 INODE_ITEM 0) itemoff 12928 itemsize 160 generation 1525968 transid 1525968 size 262144 nbytes 35389440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 135 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222730.746702497 (2021-03-08 08:58:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 31 key (5472 EXTENT_DATA 0) itemoff 12875 itemsize 53 generation 1525968 type 1 (regular) extent data disk byte 8818662686720 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 32 key (5473 INODE_ITEM 0) itemoff 12715 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 2621440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.706806940 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 33 key (5473 EXTENT_DATA 0) itemoff 12662 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12209607069696 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 34 key (5474 INODE_ITEM 0) itemoff 12502 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 521666560 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1990 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.710806887 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 35 key (5474 EXTENT_DATA 0) itemoff 12449 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12209981120512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 36 key (5475 INODE_ITEM 0) itemoff 12289 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1441792000 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5500 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.710806887 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 37 key (5475 EXTENT_DATA 0) itemoff 12236 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12210260660224 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 38 key (5476 INODE_ITEM 0) itemoff 12076 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1366556672 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5213 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.710806887 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 39 key (5476 EXTENT_DATA 0) itemoff 12023 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12210686017536 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 40 key (5477 INODE_ITEM 0) itemoff 11863 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 7077888 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 27 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.714806834 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 41 key (5477 EXTENT_DATA 0) itemoff 11810 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12210843799552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 42 key (5478 INODE_ITEM 0) itemoff 11650 itemsize 160 generation 1482901 transid 1482901 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169214.794245452 (2020-10-19 21:46:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 43 key (5478 EXTENT_DATA 0) itemoff 11597 itemsize 53 generation 1482901 type 1 (regular) extent data disk byte 7513535868928 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 44 key (5479 INODE_ITEM 0) itemoff 11437 itemsize 160 generation 1509233 transid 1509233 size 262144 nbytes 201850880 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 770 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1610527033.221370143 (2021-01-13 00:37:13) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 45 key (5479 EXTENT_DATA 0) itemoff 11384 itemsize 53 generation 1509233 type 1 (regular) extent data disk byte 11682327523328 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 46 key (5480 INODE_ITEM 0) itemoff 11224 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 968884224 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3696 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.714806834 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 47 key (5480 EXTENT_DATA 0) itemoff 11171 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12210946199552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 48 key (5481 INODE_ITEM 0) itemoff 11011 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1839464448 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 7017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.718806781 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 49 key (5481 EXTENT_DATA 0) itemoff 10958 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12211111063552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 50 key (5482 INODE_ITEM 0) itemoff 10798 itemsize 160 generation 1525968 transid 1525968 size 262144 nbytes 1832124416 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6989 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222730.746702497 (2021-03-08 08:58:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 51 key (5482 EXTENT_DATA 0) itemoff 10745 itemsize 53 generation 1525968 type 1 (regular) extent data disk byte 8818664521728 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 52 key (5483 INODE_ITEM 0) itemoff 10585 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 6291456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 24 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.718806781 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 53 key (5483 EXTENT_DATA 0) itemoff 10532 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12211529863168 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 54 key (5484 INODE_ITEM 0) itemoff 10372 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.722806727 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 55 key (5484 EXTENT_DATA 0) itemoff 10319 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12228941504512 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 56 key (5485 INODE_ITEM 0) itemoff 10159 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 370933760 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1415 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.722806727 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 57 key (5485 EXTENT_DATA 0) itemoff 10106 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12231089319936 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 58 key (5486 INODE_ITEM 0) itemoff 9946 itemsize 160 generation 1525967 transid 1525967 size 262144 nbytes 1185415168 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4522 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222694.723170138 (2021-03-08 08:58:14) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 59 key (5486 EXTENT_DATA 0) itemoff 9893 itemsize 53 generation 1525967 type 1 (regular) extent data disk byte 9425865584640 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 60 key (5487 INODE_ITEM 0) itemoff 9733 itemsize 160 generation 1501750 transid 1501750 size 262144 nbytes 1346895872 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5138 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1609833640.497822971 (2021-01-05 00:00:40) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 61 key (5487 EXTENT_DATA 0) itemoff 9680 itemsize 53 generation 1501750 type 1 (regular) extent data disk byte 7873777274880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 62 key (5488 INODE_ITEM 0) itemoff 9520 itemsize 160 generation 1530421 transid 1530421 size 262144 nbytes 1375207424 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5246 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1617139918.873813209 (2021-03-30 14:31:58) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 63 key (5488 EXTENT_DATA 0) itemoff 9467 itemsize 53 generation 1530421 type 1 (regular) extent data disk byte 7441055866880 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 64 key (5489 INODE_ITEM 0) itemoff 9307 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 2621440 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 10 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.726806674 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 65 key (5489 EXTENT_DATA 0) itemoff 9254 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12231929507840 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 66 key (5490 INODE_ITEM 0) itemoff 9094 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 3932160 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 15 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.730806621 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 67 key (5490 EXTENT_DATA 0) itemoff 9041 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12232045236224 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 68 key (5491 INODE_ITEM 0) itemoff 8881 itemsize 160 generation 1482911 transid 1482911 size 262144 nbytes 17301504 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 66 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169591.357269745 (2020-10-19 21:53:11) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 69 key (5491 EXTENT_DATA 0) itemoff 8828 itemsize 53 generation 1482911 type 1 (regular) extent data disk byte 10889918492672 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 70 key (5492 INODE_ITEM 0) itemoff 8668 itemsize 160 generation 1482901 transid 1482901 size 262144 nbytes 26214400 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 100 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169214.794245452 (2020-10-19 21:46:54) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 71 key (5492 EXTENT_DATA 0) itemoff 8615 itemsize 53 generation 1482901 type 1 (regular) extent data disk byte 7515684741120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 72 key (5493 INODE_ITEM 0) itemoff 8455 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 18087936 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 69 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.730806621 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 73 key (5493 EXTENT_DATA 0) itemoff 8402 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233142820864 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 74 key (5494 INODE_ITEM 0) itemoff 8242 itemsize 160 generation 1439787 transid 1439787 size 262144 nbytes 54263808 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 207 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1591038874.720343072 (2020-06-01 12:14:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 75 key (5494 EXTENT_DATA 0) itemoff 8189 itemsize 53 generation 1439787 type 1 (regular) extent data disk byte 12473955663872 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 76 key (5495 INODE_ITEM 0) itemoff 8029 itemsize 160 generation 1482908 transid 1482908 size 262144 nbytes 2883584 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 11 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603169482.790704298 (2020-10-19 21:51:22) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 77 key (5495 EXTENT_DATA 0) itemoff 7976 itemsize 53 generation 1482908 type 1 (regular) extent data disk byte 9382917087232 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 78 key (5496 INODE_ITEM 0) itemoff 7816 itemsize 160 generation 1482593 transid 1482593 size 262144 nbytes 873725952 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3333 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603032208.635996917 (2020-10-18 07:43:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 79 key (5496 EXTENT_DATA 0) itemoff 7763 itemsize 53 generation 1482593 type 1 (regular) extent data disk byte 4583295479808 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 80 key (5497 INODE_ITEM 0) itemoff 7603 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1531183104 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 5841 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.734806568 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 81 key (5497 EXTENT_DATA 0) itemoff 7550 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233145966592 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 82 key (5498 INODE_ITEM 0) itemoff 7390 itemsize 160 generation 1439787 transid 1439787 size 262144 nbytes 18612224 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 71 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1591038874.724342967 (2020-06-01 12:14:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 83 key (5498 EXTENT_DATA 0) itemoff 7337 itemsize 53 generation 1439787 type 1 (regular) extent data disk byte 12473955926016 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 84 key (5499 INODE_ITEM 0) itemoff 7177 itemsize 160 generation 1525968 transid 1525968 size 262144 nbytes 432013312 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1648 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222730.750702446 (2021-03-08 08:58:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 85 key (5499 EXTENT_DATA 0) itemoff 7124 itemsize 53 generation 1525968 type 1 (regular) extent data disk byte 8818666881024 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 86 key (5500 INODE_ITEM 0) itemoff 6964 itemsize 160 generation 1482593 transid 1482593 size 262144 nbytes 266600448 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1017 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1603032208.639996865 (2020-10-18 07:43:28) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 87 key (5500 EXTENT_DATA 0) itemoff 6911 itemsize 53 generation 1482593 type 1 (regular) extent data disk byte 4654161346560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 88 key (5501 INODE_ITEM 0) itemoff 6751 itemsize 160 generation 1525961 transid 1525961 size 262144 nbytes 1067450368 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4072 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222478.881970135 (2021-03-08 08:54:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 89 key (5501 EXTENT_DATA 0) itemoff 6698 itemsize 53 generation 1525961 type 1 (regular) extent data disk byte 7982222049280 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 90 key (5502 INODE_ITEM 0) itemoff 6538 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 562561024 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2146 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.734806568 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 91 key (5502 EXTENT_DATA 0) itemoff 6485 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233186123776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 92 key (5503 INODE_ITEM 0) itemoff 6325 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 3145728 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 12 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.734806568 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 93 key (5503 EXTENT_DATA 0) itemoff 6272 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233187991552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 94 key (5504 INODE_ITEM 0) itemoff 6112 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1200095232 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4578 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.738806514 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 95 key (5504 EXTENT_DATA 0) itemoff 6059 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233257922560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 96 key (5505 INODE_ITEM 0) itemoff 5899 itemsize 160 generation 1525961 transid 1525961 size 262144 nbytes 71827456 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 274 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222478.885970083 (2021-03-08 08:54:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 97 key (5505 EXTENT_DATA 0) itemoff 5846 itemsize 53 generation 1525961 type 1 (regular) extent data disk byte 8013364232192 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 98 key (5506 INODE_ITEM 0) itemoff 5686 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 4718592 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 18 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.738806514 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 99 key (5506 EXTENT_DATA 0) itemoff 5633 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233379401728 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 100 key (5507 INODE_ITEM 0) itemoff 5473 itemsize 160 generation 1525959 transid 1525959 size 262144 nbytes 294649856 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1124 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222403.234950654 (2021-03-08 08:53:23) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 101 key (5507 EXTENT_DATA 0) itemoff 5420 itemsize 53 generation 1525959 type 1 (regular) extent data disk byte 7750833868800 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 102 key (5508 INODE_ITEM 0) itemoff 5260 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 362807296 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1384 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.742806462 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 103 key (5508 EXTENT_DATA 0) itemoff 5207 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233610690560 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 104 key (5509 INODE_ITEM 0) itemoff 5047 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 766246912 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 2923 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.742806462 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 105 key (5509 EXTENT_DATA 0) itemoff 4994 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233544069120 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 106 key (5510 INODE_ITEM 0) itemoff 4834 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1768685568 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 6747 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.746806408 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 107 key (5510 EXTENT_DATA 0) itemoff 4781 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233581916160 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 108 key (5511 INODE_ITEM 0) itemoff 4621 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 958398464 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 3656 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.746806408 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 109 key (5511 EXTENT_DATA 0) itemoff 4568 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233725329408 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 110 key (5512 INODE_ITEM 0) itemoff 4408 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 408420352 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 1558 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.750806355 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 111 key (5512 EXTENT_DATA 0) itemoff 4355 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233793339392 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 112 key (5513 INODE_ITEM 0) itemoff 4195 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 1253834752 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4783 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.750806355 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 113 key (5513 EXTENT_DATA 0) itemoff 4142 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233785352192 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 114 key (5514 INODE_ITEM 0) itemoff 3982 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 8126464 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 31 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.754806301 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233813143552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160 generation 1439787 transid 1439787 size 262144 nbytes 44826624 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1591038874.816340561 (2020-06-01 12:14:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53 generation 1439787 type 1 (regular) extent data disk byte 12474011467776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160 generation 1546750 transid 1546750 size 262144 nbytes 2405433344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1621321238.520473625 (2021-05-18 00:00:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53 generation 1546750 type 1 (regular) extent data disk byte 12251618267136 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 40370176 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.758806249 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233872674816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160 generation 1525963 transid 1525963 size 262144 nbytes 57933824 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222550.521041172 (2021-03-08 08:55:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) elem_cnt 0 elem_missed 0 ret -2 Failed to find [7750833868800, 168, 262144] Segmentation fault -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 22:59 ` Marc MERLIN @ 2022-04-27 23:02 ` Josef Bacik 2022-04-27 23:21 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 23:02 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote: > > Sigh, added another print_leaf. Thanks, > > doing an insert that overlaps our bytenr 7750833627136 262144 > processed 1146880 of 0 possible bytes > processed 1163264 of 0 possible bytes > processed 1179648 of 0 possible bytes > processed 1196032 of 0 possible bytes > processed 1212416 of 0 possible bytes > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > inode ref info failed??? > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE Ooooh that explains it, it's the free space cache, that's perfect! I'll get something wired up and let you know when it's ready. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 23:02 ` Josef Bacik @ 2022-04-27 23:21 ` Josef Bacik 2022-04-28 0:18 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-27 23:21 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote: > > On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote: > > > Sigh, added another print_leaf. Thanks, > > > > doing an insert that overlaps our bytenr 7750833627136 262144 > > processed 1146880 of 0 possible bytes > > processed 1163264 of 0 possible bytes > > processed 1179648 of 0 possible bytes > > processed 1196032 of 0 possible bytes > > processed 1212416 of 0 possible bytes > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > inode ref info failed??? > > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE > > Ooooh that explains it, it's the free space cache, that's perfect! > I'll get something wired up and let you know when it's ready. Thanks, > Ok, lets hope for better results this time. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-27 23:21 ` Josef Bacik @ 2022-04-28 0:18 ` Marc MERLIN 2022-04-28 0:44 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 0:18 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 07:21:51PM -0400, Josef Bacik wrote: > On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote: > > > > Sigh, added another print_leaf. Thanks, > > > > > > doing an insert that overlaps our bytenr 7750833627136 262144 > > > processed 1146880 of 0 possible bytes > > > processed 1163264 of 0 possible bytes > > > processed 1179648 of 0 possible bytes > > > processed 1196032 of 0 possible bytes > > > processed 1212416 of 0 possible bytes > > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > > inode ref info failed??? > > > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE > > > > Ooooh that explains it, it's the free space cache, that's perfect! > > I'll get something wired up and let you know when it's ready. Thanks, > > > > Ok, lets hope for better results this time. Thanks, item 115 key (5514 EXTENT_DATA 0) itemoff 3929 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233813143552 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 116 key (5515 INODE_ITEM 0) itemoff 3769 itemsize 160 generation 1439787 transid 1439787 size 262144 nbytes 44826624 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 171 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1591038874.816340561 (2020-06-01 12:14:34) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 117 key (5515 EXTENT_DATA 0) itemoff 3716 itemsize 53 generation 1439787 type 1 (regular) extent data disk byte 12474011467776 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 118 key (5516 INODE_ITEM 0) itemoff 3556 itemsize 160 generation 1546750 transid 1546750 size 262144 nbytes 2405433344 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 9176 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1621321238.520473625 (2021-05-18 00:00:38) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 119 key (5516 EXTENT_DATA 0) itemoff 3503 itemsize 53 generation 1546750 type 1 (regular) extent data disk byte 12251618267136 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 120 key (5517 INODE_ITEM 0) itemoff 3343 itemsize 160 generation 1526158 transid 1526158 size 262144 nbytes 40370176 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 154 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615276843.758806249 (2021-03-09 00:00:43) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) item 121 key (5517 EXTENT_DATA 0) itemoff 3290 itemsize 53 generation 1526158 type 1 (regular) extent data disk byte 12233872674816 nr 262144 extent data offset 0 nr 262144 ram 262144 extent compression 0 (none) item 122 key (5518 INODE_ITEM 0) itemoff 3130 itemsize 160 generation 1525963 transid 1525963 size 262144 nbytes 57933824 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 221 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC) atime 0.0 (1969-12-31 16:00:00) ctime 1615222550.521041172 (2021-03-08 08:55:50) mtime 0.0 (1969-12-31 16:00:00) otime 0.0 (1969-12-31 16:00:00) elem_cnt 0 elem_missed 0 ret -2 Failed to find [7750833868800, 168, 262144] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd740, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd758) at common/rbtree-utils.c:48 #1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=13576826748928) at common/extent-cache.c:179 #2 0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=13576826748928, end=13576826748927, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/extent_io.c:976 #5 0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555555721210) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, iref=0xffffffffffffffe3, path=0x555555b8e5f0, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=0, owner=5507, root_objectid=93824998761968, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555555b8e5f0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=1, owner=5507, offset=0) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfd52 in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=eb@entry=0x555559d00d20, current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:615 #10 0x00005555555dfeab in process_eb (trans=trans@entry=0x555558a8d7c0, root=root@entry=0x55555564cde0, eb=0x555555944100, current=current@entry=0x7fffffffdaf8) at cmds/rescue-init-extent-tree.c:690 #11 0x00005555555e0116 in record_root (root=0x55555564cde0) at cmds/rescue-init-extent-tree.c:761 #12 0x00005555555e0436 in btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:897 #13 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #14 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #15 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #16 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #17 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 0:18 ` Marc MERLIN @ 2022-04-28 0:44 ` Josef Bacik 2022-04-28 3:00 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 0:44 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 8:18 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 07:21:51PM -0400, Josef Bacik wrote: > > On Wed, Apr 27, 2022 at 7:02 PM Josef Bacik <josef@toxicpanda.com> wrote: > > > > > > On Wed, Apr 27, 2022 at 6:59 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > > > On Wed, Apr 27, 2022 at 05:27:44PM -0400, Josef Bacik wrote: > > > > > Sigh, added another print_leaf. Thanks, > > > > > > > > doing an insert that overlaps our bytenr 7750833627136 262144 > > > > processed 1146880 of 0 possible bytes > > > > processed 1163264 of 0 possible bytes > > > > processed 1179648 of 0 possible bytes > > > > processed 1196032 of 0 possible bytes > > > > processed 1212416 of 0 possible bytes > > > > processed 1228800 of 0 possible bytesWTF???? we think we already inserted this bytenr?? [5507, 108, 0] dumping paths > > > > inode ref info failed??? > > > > leaf 15645023322112 items 123 free space 55 generation 1546750 owner ROOT_TREE > > > > > > Ooooh that explains it, it's the free space cache, that's perfect! > > > I'll get something wired up and let you know when it's ready. Thanks, > > > > > > > Ok, lets hope for better results this time. Thanks, > Ok it should work now. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 0:44 ` Josef Bacik @ 2022-04-28 3:00 ` Marc MERLIN 2022-04-28 3:08 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 3:00 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote: > Ok it should work now. Thanks, Mmmh, it got worse, unfortunately (gdb) run rescue init-extent-tree /dev/mapper/dshelf1 Starting program: /var/local/src/btrfs-progs-josefbacik/btrfs rescue init-extent-tree /dev/mapper/dshelf1 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". FS_INFO IS 0x55555564cbc0 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55555564cbc0 Walking all our trees and pinning down the currently accessible blocks Clearing the extent root and re-init'ing the block groups Program received signal SIGSEGV, Segmentation fault. 0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff78e1f46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000555555585950 in memmove (__len=<optimized out>, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:36 #2 memmove_extent_buffer (dst=dst@entry=0x555555b57f60, dst_offset=<optimized out>, src_offset=<optimized out>, len=<optimized out>) at kernel-shared/extent_io.c:1021 #3 0x000055555557753b in btrfs_del_items (trans=trans@entry=0x555558a8d460, root=root@entry=0x55555564cde0, path=path@entry=0x7fffffffdc50, slot=slot@entry=36, nr=nr@entry=1) at ./kernel-shared/ctree.h:2387 #4 0x00005555555e0627 in clear_space_cache (block_group=<optimized out>, trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:116 #5 reset_block_groups (trans=0x555558a8d460) at cmds/rescue-init-extent-tree.c:307 #6 reinit_extent_tree (fs_info=0x7fffffffdc50) at cmds/rescue-init-extent-tree.c:451 #7 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:922 #8 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #9 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #10 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #11 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #12 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08 ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 3:00 ` Marc MERLIN @ 2022-04-28 3:08 ` Josef Bacik 2022-04-28 3:11 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 3:08 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote: > > Ok it should work now. Thanks, > > Mmmh, it got worse, unfortunately > Well not worse, just failed earlier because that's where the new code is. I think I just made a simple mistake, but I added some print'fs just in case. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 3:08 ` Josef Bacik @ 2022-04-28 3:11 ` Marc MERLIN 2022-04-28 4:03 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 3:11 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 11:08:02PM -0400, Josef Bacik wrote: > On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote: > > > Ok it should work now. Thanks, > > > > Mmmh, it got worse, unfortunately > > > > Well not worse, just failed earlier because that's where the new code > is. I think I just made a simple mistake, but I added some print'fs > just in case. Thanks, That helped: inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 18446744073706422272 possible bytes Recording extents for root 4 processed 1032192 of 1064960 possible bytes Recording extents for root 5 processed 10960896 of 10977280 possible bytes Recording extents for root 7 processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON `owner < BTRFS_FIRST_FREE_OBJECTID` triggered, value 1 /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116] /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089] /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43] /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a] Program received signal SIGABRT, Aborted. 0x00007ffff78768a1 in raise () from /lib/x86_64-linux-gnu/libc.so.6 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 3:11 ` Marc MERLIN @ 2022-04-28 4:03 ` Josef Bacik 2022-04-28 4:12 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 4:03 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Wed, Apr 27, 2022 at 11:11 PM Marc MERLIN <marc@merlins.org> wrote: > > On Wed, Apr 27, 2022 at 11:08:02PM -0400, Josef Bacik wrote: > > On Wed, Apr 27, 2022 at 11:00 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Wed, Apr 27, 2022 at 08:44:01PM -0400, Josef Bacik wrote: > > > > Ok it should work now. Thanks, > > > > > > Mmmh, it got worse, unfortunately > > > > > > > Well not worse, just failed earlier because that's where the new code > > is. I think I just made a simple mistake, but I added some print'fs > > just in case. Thanks, > > That helped: > inserting block group 15839365431296 > inserting block group 15840439173120 > inserting block group 15842586656768 > processed 1556480 of 0 possible bytes > processed 1474560 of 18446744073706422272 possible bytes > Recording extents for root 4 > processed 1032192 of 1064960 possible bytes > Recording extents for root 5 > processed 10960896 of 10977280 possible bytes > Recording extents for root 7 > processed 1707278336 of 16545742848 possible byteskernel-shared/extent-tree.c:1193: insert_inline_extent_backref: BUG_ON `owner < BTRFS_FIRST_FREE_OBJECTID` triggered, value 1 > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x2975e)[0x55555557d75e] > /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_inc_extent_ref+0x138)[0x55555557ed77] > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8be01)[0x5555555dfe01] > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab] > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8beab)[0x5555555dfeab] > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x8c116)[0x5555555e0116] > /var/local/src/btrfs-progs-josefbacik/btrfs(btrfs_init_extent_tree+0xe96)[0x5555555e1089] > /var/local/src/btrfs-progs-josefbacik/btrfs(+0x83a43)[0x5555555d7a43] > /var/local/src/btrfs-progs-josefbacik/btrfs(handle_command_group+0x49)[0x55555556c17b] > /var/local/src/btrfs-progs-josefbacik/btrfs(main+0x94)[0x55555556c275] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7ffff78617fd] > /var/local/src/btrfs-progs-josefbacik/btrfs(_start+0x2a)[0x55555556be1a] > Cool, we're pointing at the same block in different places in the same tree. I should make tree-recover catch that and fix it, but since you're going to re-generate your csum tree anyway I've adjusted init-extent-tree to just clear the csum tree too, lets see how far we get with this. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 4:03 ` Josef Bacik @ 2022-04-28 4:12 ` Marc MERLIN 2022-04-28 15:30 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 4:12 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 12:03:29AM -0400, Josef Bacik wrote: > Cool, we're pointing at the same block in different places in the same > tree. I should make tree-recover catch that and fix it, but since > you're going to re-generate your csum tree anyway I've adjusted > init-extent-tree to just clear the csum tree too, lets see how far we > get with this. Thanks, inserting block group 15835070464000 inserting block group 15836144205824 inserting block group 15837217947648 inserting block group 15838291689472 inserting block group 15839365431296 inserting block group 15840439173120 inserting block group 15842586656768 processed 1556480 of 0 possible bytes processed 1474560 of 0 possible bytes Recording extents for root 4 processed 1032192 of 1064960 possible bytes Recording extents for root 5 processed 10960896 of 10977280 possible bytes Recording extents for root 7 processed 16384 of 16545742848 possible bytes Recording extents for root 9 processed 16384 of 16384 possible bytes Recording extents for root 11221 processed 16384 of 255983616 possible bytes Recording extents for root 11222 processed 49479680 of 49479680 possible bytes Recording extents for root 11223 processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x10000000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 #1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x10000000060, start=start@entry=15645286318080) at common/extent-cache.c:179 #2 0x0000555555584c67 in set_extent_bits (tree=0x10000000060, start=15645286318080, end=15645286318079, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/extent_io.c:976 #5 0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x555560e15ef0) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, parent=0, iref=0xffffffffffffffe4, path=0x555560dfe010, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825185865744, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x555560dfe010, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfea9 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x555560e0ddf0, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659 #10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=eb@entry=0x55555a1be9a0, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555558b5c9a0, root=root@entry=0x555555af1ed0, eb=0x555559ce9020, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #12 0x00005555555e026d in record_root (root=root@entry=0x555555af1ed0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 4:12 ` Marc MERLIN @ 2022-04-28 15:30 ` Josef Bacik 2022-04-28 16:27 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 15:30 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 12:12 AM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 28, 2022 at 12:03:29AM -0400, Josef Bacik wrote: > > Cool, we're pointing at the same block in different places in the same > > tree. I should make tree-recover catch that and fix it, but since > > you're going to re-generate your csum tree anyway I've adjusted > > init-extent-tree to just clear the csum tree too, lets see how far we > > get with this. Thanks, > > inserting block group 15835070464000 > inserting block group 15836144205824 > inserting block group 15837217947648 > inserting block group 15838291689472 > inserting block group 15839365431296 > inserting block group 15840439173120 > inserting block group 15842586656768 > processed 1556480 of 0 possible bytes > processed 1474560 of 0 possible bytes > Recording extents for root 4 > processed 1032192 of 1064960 possible bytes > Recording extents for root 5 > processed 10960896 of 10977280 possible bytes > Recording extents for root 7 > processed 16384 of 16545742848 possible bytes > Recording extents for root 9 > processed 16384 of 16384 possible bytes > Recording extents for root 11221 > processed 16384 of 255983616 possible bytes > Recording extents for root 11222 > processed 49479680 of 49479680 possible bytes > Recording extents for root 11223 > processed 1634992128 of 1635549184 possible bytesFailed to find [3700677820416, 168, 53248] > Hell yes we're in the fs tree's now, in the home stretch hopefully. I've pushed new debugging, you may have another overlapping extent. I'm going to have to wire up a tool for that, but hopefully we can just target delete a few things and get you up and running. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 15:30 ` Josef Bacik @ 2022-04-28 16:27 ` Marc MERLIN 2022-04-28 20:13 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 16:27 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote: > Hell yes we're in the fs tree's now, in the home stretch hopefully. > I've pushed new debugging, you may have another overlapping extent. > I'm going to have to wire up a tool for that, but hopefully we can > just target delete a few things and get you up and running. Thanks, Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ? item 186 key (1834097 EXTENT_DATA 56098816) itemoff 5452 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451680858112 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 187 key (1834097 EXTENT_DATA 56623104) itemoff 5399 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451683733504 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451684257792 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451684782080 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451685568512 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457553899520 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457554423808 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457561710592 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457563021312 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457564856320 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aed5 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 #1 0x000055555559b0d9 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157752832) at common/extent-cache.c:179 #2 0x0000555555584c67 in set_extent_bits (tree=0x100000060, start=15645157752832, end=15645157752831, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584e9a in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x000055555558587d in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/extent_io.c:976 #5 0x000055555557bc4b in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ba350) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee74 in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, parent=0, iref=0xffffffffffffffe3, path=0x55555d2822b0, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123492528, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d2822b0, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfea9 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x55555d291e50, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659 #10 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=eb@entry=0x5555564687c0, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #11 0x00005555555e0002 in process_eb (trans=trans@entry=0x555556dc2cd0, root=root@entry=0x555557a97ca0, eb=0x555555f13e20, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #12 0x00005555555e026d in record_root (root=root@entry=0x555557a97ca0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e10f9 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a43 in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 16:27 ` Marc MERLIN @ 2022-04-28 20:13 ` Josef Bacik 2022-04-28 20:22 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 20:13 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote: > > Hell yes we're in the fs tree's now, in the home stretch hopefully. > > I've pushed new debugging, you may have another overlapping extent. > > I'm going to have to wire up a tool for that, but hopefully we can > > just target delete a few things and get you up and running. Thanks, > > Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ? Cool, do ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device> Then you should be able to run the init-extent-tree and get past that part. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 20:13 ` Josef Bacik @ 2022-04-28 20:22 ` Marc MERLIN 2022-04-28 20:28 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 20:22 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 04:13:31PM -0400, Josef Bacik wrote: > On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote: > > > > On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote: > > > Hell yes we're in the fs tree's now, in the home stretch hopefully. > > > I've pushed new debugging, you may have another overlapping extent. > > > I'm going to have to wire up a tool for that, but hopefully we can > > > just target delete a few things and get you up and running. Thanks, > > > > Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ? > > Cool, do > > ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device> > > Then you should be able to run the init-extent-tree and get past that > part. Thanks, Sorry, I must be triggering every single unhandled path in the code :) gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55fd6e82d600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55fd6e82d600 kernel-shared/ctree.c:365: update_ref_for_cow: BUG_ON `ret` triggered, value -5 ./btrfs-corrupt-block(+0x1137d)[0x55fd6d53237d] ./btrfs-corrupt-block(__btrfs_cow_block+0x3a9)[0x55fd6d533a90] ./btrfs-corrupt-block(btrfs_cow_block+0x73)[0x55fd6d53414f] ./btrfs-corrupt-block(btrfs_search_slot+0x11a)[0x55fd6d536b51] ./btrfs-corrupt-block(main+0x141d)[0x55fd6d56f4a9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f526bee17fd] ./btrfs-corrupt-block(_start+0x2a)[0x55fd6d52d8ba] Aborted -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 20:22 ` Marc MERLIN @ 2022-04-28 20:28 ` Josef Bacik 2022-04-28 20:57 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 20:28 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 4:22 PM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 28, 2022 at 04:13:31PM -0400, Josef Bacik wrote: > > On Thu, Apr 28, 2022 at 12:27 PM Marc MERLIN <marc@merlins.org> wrote: > > > > > > On Thu, Apr 28, 2022 at 11:30:35AM -0400, Josef Bacik wrote: > > > > Hell yes we're in the fs tree's now, in the home stretch hopefully. > > > > I've pushed new debugging, you may have another overlapping extent. > > > > I'm going to have to wire up a tool for that, but hopefully we can > > > > just target delete a few things and get you up and running. Thanks, > > > > > > Delete Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules ? > > > > Cool, do > > > > ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 <device> > > > > Then you should be able to run the init-extent-tree and get past that > > part. Thanks, > > Sorry, I must be triggering every single unhandled path in the code :) > "Make recovery tools more resilient" is definitely muuuuuch higher on the team priority list that's for fucking sure. Try again please, hopefully it works this time. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 20:28 ` Josef Bacik @ 2022-04-28 20:57 ` Marc MERLIN 2022-04-28 20:58 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 20:57 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 04:28:53PM -0400, Josef Bacik wrote: > "Make recovery tools more resilient" is definitely muuuuuch higher on > the team priority list that's for fucking sure. Try again please, > hopefully it works this time. Thanks, Great to hear, and obviously not just hearing, thanks for your persistence in this mega thread. Here's the new one: gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55f248b64600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55f248b64600 Error searching to node -2 Marc -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 20:57 ` Marc MERLIN @ 2022-04-28 20:58 ` Josef Bacik 2022-04-28 21:42 ` Marc MERLIN 0 siblings, 1 reply; 479+ messages in thread From: Josef Bacik @ 2022-04-28 20:58 UTC (permalink / raw) To: Marc MERLIN; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 4:57 PM Marc MERLIN <marc@merlins.org> wrote: > > On Thu, Apr 28, 2022 at 04:28:53PM -0400, Josef Bacik wrote: > > "Make recovery tools more resilient" is definitely muuuuuch higher on > > the team priority list that's for fucking sure. Try again please, > > hopefully it works this time. Thanks, > > Great to hear, and obviously not just hearing, thanks for your > persistence in this mega thread. > > Here's the new one: > gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d > "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 > FS_INFO IS 0x55f248b64600 > JOSEF: root 9 > Couldn't find the last root for 8 > FS_INFO AFTER IS 0x55f248b64600 > Error searching to node -2 > -2 is enoent, it must have committed with the deleted block, which is sort of scary but at this point I'll take it. Go ahead and do the init-extent-tree. Thanks, Josef ^ permalink raw reply [flat|nested] 479+ messages in thread
* Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed) 2022-04-28 20:58 ` Josef Bacik @ 2022-04-28 21:42 ` Marc MERLIN 2022-04-28 21:54 ` Josef Bacik 0 siblings, 1 reply; 479+ messages in thread From: Marc MERLIN @ 2022-04-28 21:42 UTC (permalink / raw) To: Josef Bacik; +Cc: linux-btrfs On Thu, Apr 28, 2022 at 04:58:24PM -0400, Josef Bacik wrote: > -2 is enoent, it must have committed with the deleted block, which is > sort of scary but at this point I'll take it. Go ahead and do the > init-extent-tree. Thanks, Same output :( Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248] it's the one we just tried to clear, and got enoent on. Unless -r 11223 is not the correct value? (not sure where it came from) gargamel:/var/local/src/btrfs-progs-josefbacik# ./btrfs-corrupt-block -d "3700677820416,168,53248" -r 11223 /dev/mapper/dshelf1 FS_INFO IS 0x55d8bac70600 JOSEF: root 9 Couldn't find the last root for 8 FS_INFO AFTER IS 0x55d8bac70600 Error searching to node -2 New run still returns the same thing: item 188 key (1834097 EXTENT_DATA 57147392) itemoff 5346 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451684257792 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 189 key (1834097 EXTENT_DATA 57671680) itemoff 5293 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451684782080 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 190 key (1834097 EXTENT_DATA 58195968) itemoff 5240 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7451685568512 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 191 key (1834097 EXTENT_DATA 58720256) itemoff 5187 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457553899520 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 192 key (1834097 EXTENT_DATA 59244544) itemoff 5134 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457554423808 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 193 key (1834097 EXTENT_DATA 59768832) itemoff 5081 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457561710592 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 194 key (1834097 EXTENT_DATA 60293120) itemoff 5028 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457563021312 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) item 195 key (1834097 EXTENT_DATA 60817408) itemoff 4975 itemsize 53 generation 1590259 type 1 (regular) extent data disk byte 7457564856320 nr 524288 extent data offset 0 nr 524288 ram 524288 extent compression 0 (none) elem_cnt 1 elem_missed 0 ret 0 Xilinx_Unified_2020.1_0602_1208/tps/lnx64/jre9.0.4/lib/modules Failed to find [3700677820416, 168, 53248] Program received signal SIGSEGV, Segmentation fault. rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 48 struct rb_node *n = root->rb_node; (gdb) bt #0 rb_search (root=root@entry=0x100000060, key=key@entry=0x7fffffffd640, comp=comp@entry=0x55555559aee0 <cache_tree_comp_range>, next_ret=next_ret@entry=0x7fffffffd658) at common/rbtree-utils.c:48 #1 0x000055555559b0e4 in search_cache_extent (tree=tree@entry=0x100000060, start=start@entry=15645157801984) at common/extent-cache.c:179 #2 0x0000555555584c72 in set_extent_bits (tree=0x100000060, start=15645157801984, end=15645157801983, bits=bits@entry=1) at kernel-shared/extent_io.c:380 #3 0x0000555555584ea5 in set_extent_dirty (tree=<optimized out>, start=<optimized out>, end=<optimized out>) at kernel-shared/extent_io.c:486 #4 0x0000555555585888 in set_extent_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/extent_io.c:976 #5 0x000055555557bc56 in btrfs_mark_buffer_dirty (eb=eb@entry=0x55555d2ab870) at kernel-shared/disk-io.c:2224 #6 0x000055555557ee7f in setup_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, parent=0, iref=0xffffffffffffffe3, path=0x55555d273340, root=<optimized out>) at kernel-shared/extent-tree.c:1085 #7 insert_inline_extent_backref (refs_to_add=1, offset=1835008, owner=1834097, root_objectid=93825123431232, parent=0, num_bytes=<optimized out>, bytenr=<optimized out>, path=0x55555d273340, root=<optimized out>, trans=<optimized out>) at kernel-shared/extent-tree.c:1197 #8 btrfs_inc_extent_ref (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, bytenr=<optimized out>, num_bytes=<optimized out>, parent=parent@entry=0, root_objectid=root_objectid@entry=11223, owner=1834097, offset=1835008) at kernel-shared/extent-tree.c:1262 #9 0x00005555555dfeb4 in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x55555d283370, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:659 #10 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=eb@entry=0x555556461260, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #11 0x00005555555e000d in process_eb (trans=trans@entry=0x555556fcc400, root=root@entry=0x555556faefe0, eb=0x555555f0c680, current=current@entry=0x7fffffffdae8) at cmds/rescue-init-extent-tree.c:734 #12 0x00005555555e0278 in record_root (root=root@entry=0x555556faefe0) at cmds/rescue-init-extent-tree.c:805 #13 0x00005555555e1104 in record_roots (fs_info=0x55555564cbc0) at cmds/rescue-init-extent-tree.c:860 #14 btrfs_init_extent_tree (path=path@entry=0x7fffffffe1cd "/dev/mapper/dshelf1") at cmds/rescue-init-extent-tree.c:944 #15 0x00005555555d7a4e in cmd_rescue_init_extent_tree (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds/rescue.c:65 #16 0x000055555556c17b in cmd_execute (argv=0x7fffffffdeb8, argc=2, cmd=0x555555642d40 <cmd_struct_rescue_init_extent_tree>) at cmds/commands.h:125 #17 handle_command_group (cmd=<optimized out>, argc=2, argv=0x7fffffffdeb8) at btrfs.c:152 #18 0x000055555556c275 in cmd_execute (argv=0x7fffffffdeb0, argc=3, cmd=0x555555643cc0 <cmd_struct_rescue>) at cmds/commands.h:125 #19 main (argc=3, argv=0x7fffffffdeb0) at btrfs.c:405 -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Home page: http://marc.merlins.org/ ^ permalink raw reply [flat|nested] 479+ messages in thread